Not Scraping Every Page?
-
So I am trying to scrape about 3000 pages of a directory (ie, all the pages are in the same format). However, it seems to selectively scrape the pages. Sometimes it works, sometimes it doesn’t.
For example, I might send it to scrape a list of 100 pages, and it will scrape 80 of them successfully then freeze/stop working. Then Ill run the same script again, and this time, maybe 50 of them are scraped before it stops working.
Does anybody know what’s going on? Why its timing out (I think thats whats happening) at different points each time?
- The topic ‘Not Scraping Every Page?’ is closed to new replies.