Crawler schedule 00:00 and a lot of resource usage

  • Resolved nixeens

    (@nixeens)


    2 years, 12 months ago

    Hello,
    I have a couple of questions, but first about my site.
    Im hosting on sharedhosting with litespeed server. Provider enabled for me crawler option and it helped a lot of speed, but couple days ago i started to see that crawler is using a lot of cpu and slowing down site. Even losing connection to database.
    Crawler settings – https://snipboard.io/jUO2P4.jpg
    Resource from my hosting plesk panel – https://snipboard.io/QaKANe.jpg
    So my question.
    1. How can i set up schedule for crawler to start like 00:00 so that visitors dosnt get angry that site so slow if crawler started to run.
    2. What could be optimized crawler config for my case. ia have like 250 pages to crawl.Some times on watch crawler status i got some position with Status: stopped_maxtime

    But the 1st questions is more important, its not nice when it starts some time in middle of day, when there are most of visitors.

    Report number: FHZFGBZM

    Report date: 11/28/2021 19:45:56

    The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Support qtwrk

    (@qtwrk)

    Hi,

    you can’t set crawler at which time exact to run

    in that case, disbale the plugin’s crawler, and use crawler script then use system cron to start it at 00:00

    Best regards,

    Thread Starter nixeens

    (@nixeens)

    ok, then what about settings run duration interval between runs and delay for better performance. When crawler runs it feels like page is not usable. And cpu is high.
    Just some times i think its making bootneck.

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    these setting won’t be able to control the start time precisely , so you need to use crawler script

    the crawler itself doesn’t really take much resources, like 90% of resources usage will be from the page generation

    Best regards,

    Thread Starter nixeens

    (@nixeens)

    Hosting provider: Your tariff plan does not include CRON management.
    So i couldnt make cronjob for that with t hat crawler script.

    But with those setting can i control how agressive crawler is? If its not using my website resource then could i make it faster do that. With these settings it runs about 4h

    Plugin Support qtwrk

    (@qtwrk)

    Hi,

    the thing is not how “aggressive” that is , but you can not control when it will start

    for example today it could start at 00:00 , but tomorrow will be 02:30 , the next after will be 05:00 , after will be 07:00 …etc

    eventually it will end up in daylight time and compete the resources

    one thing you can try is to reduce thread from 3 to 1

    Best regards,

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Crawler schedule 00:00 and a lot of resource usage’ is closed to new replies.

Tags