Crawler Issue / question

  • Resolved scysys

    (@scysys)


    4 years, 1 month ago

    Crawler is working on each project. Overall the same ISP.
    But with one site i have some problems to get it working.

    Imgur: https://imgur.com/a/Dqyso9y

    In can see that the crawler is doing something. Update positions. But on other projects i see all green under hit. Next complete crawler is on 10/12. Stopped reason is always “stopped_maxtime”.

    Can i safely increase the run duration and the interval? Or should i just wait on those big projects?

    • This topic was modified 4 years, 1 month ago by scysys.
Viewing 3 replies - 1 through 3 (of 3 total)
  • Thread Starter scysys

    (@scysys)

    Wenn i visit the crawler site and click on “Watch Crawler Status”. The crawler seems to work again.

    Imgur: https://imgur.com/4kg3DiI

    But as i see the process is start from the beginning? Also it makes no sense when i manually must visit the site to wakeup the crawler ??

    The cron is definitely working as system cron every minute.

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    No , you don’t need to visit crawler status every now and then.

    when cron is running properly , it will initiate the crawler cron as well

    if you don’t have resources restriction and enough power to run crawlers , then yes, please consider increase run duration and decrease crawler interval.

    you would see all green AFTER crawler has finished all pages , but during the process it won’t show as all green.

    Best regards,

    Thread Starter scysys

    (@scysys)

    Thanks qtwark!

    After 2 Days now it looks like this: https://imgur.com/lOzkUES
    Really great.

    It′s solved. My failure. Next time i wait a little bit longer. ??

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Crawler Issue / question’ is closed to new replies.

Tags