Open
Description
Hi ,
I am trying to crawl sitemap xml file which includes bulk urls - 100% completed (563 processed/563 total)
I am getting error when committing to Azure.
I have tried many time running the norconex -
command being used: collector-http.bat -a start -c collectorconfig.xml
PFB error details from logs -
Crawler : 2018-12-25 23:34:04 INFO - Azure Search REST API Http Client closed.
Crawler : 2018-12-25 23:34:04 INFO - Azure Search REST API Http Client closed.
Crawler : 2018-12-25 23:34:04 ERRO
57E9
R - Could not commit batched operations.
com.norconex.committer.core.CommitterException: Invalid HTTP response: "HTTP/1.1 413 Request Entity Too Large". Azure Response: The page was not displayed because the request entity is too large.
at com.norconex.committer.azuresearch.AzureSearchCommitter.handleResponse(AzureSearchCommitter.java:509)
at com.norconex.committer.azuresearch.AzureSearchCommitter.commitBatch(AzureSearchCommitter.java:478)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.azuresearch.AzureSearchCommitter.commit(AzureSearchCommitter.java:405)
at com.norconex.collector.core.crawler.AbstractCrawler.execute(AbstractCrawler.java:274)
at com.norconex.collector.core.crawler.AbstractCrawler.doExecute(AbstractCrawler.java:228)
at com.norconex.collector.core.crawler.AbstractCrawler.startExecution(AbstractCrawler.java:184)
at com.norconex.jef4.job.AbstractResumableJob.execute(AbstractResumableJob.java:49)
at com.norconex.jef4.suite.JobSuite.runJob(JobSuite.java:355)
at com.norconex.jef4.suite.JobSuite.doExecute(JobSuite.java:296)
at com.norconex.jef4.suite.JobSuite.execute(JobSuite.java:168)
at com.norconex.collector.core.AbstractCollector.start(AbstractCollector.java:132)
at com.norconex.collector.core.AbstractCollectorLauncher.launch(AbstractCollectorLauncher.java:95)
at com.norconex.collector.http.HttpCollector.main(HttpCollector.java:74)
Can you please advise what needs to be done for this.
Br,
Akash
Metadata
Metadata
Assignees
Labels
No labels