Crawler Not Running

Hi,

you need to lower the crawler interval time

when you set to 302400 which is default , it will only run once every 3.5 days even you manually trigger it again , it won’t start

Best regards,

What do I need to lower it to so that it will start when I manually try to run it?

Crawler setting -> Crawl Interval

make it low value , depends on the resources you have

if it’s your own VPS/dedicated server , you can just make it very aggressive, set it like 1800 or 3600

if it’s shared hosting , make it higher , like 43200 or 86400 …etc

I suggest you fix it in the next update to make it clear to the user that selecting the crawler to manually run doesn’t run it when they press the button. It runs at the next scheduled interval.

The way the setting is makes it seem like the user is able to run the crawler as soon as the button is pressed. One the FAQ it is confusing. It states:

Use the Manually run button to start the crawler without waiting for the cron job.

https://docs.litespeedtech.com/lscache/lscwp/crawler/

It also states:

Crawl Interval?
302400

This value determines how long to wait before re-initiating the entire crawling process. To keep your site regularly-crawled, determine how long the crawler usually takes to run, and set this value to slightly longer than that.

https://docs.litespeedtech.com/lscache/lscwp/crawler/

How do you figure out how long it takes the crawler to run?

duly noted.

for how long , it depends on your server env , your resources accessibility , among other factor

if it’s your own VPS, you can just set it like 600 or something , so crawler will run very aggressive.

if it’s shared hosting , then I’d suggest to make it longer , like a day or two , at least

but it also depends how many pages you have

Best regards,

One suggestion is to add an indicator to let the person know how long it took the crawler to crawl the entire website. Another suggestion would be to have the crawler automatically adjust the crawl time based on the user’s server.

thanks for the suggestion

but I am afraid that is not really possible for the first one

e.g.

my site has 1000 pages , some page might take 1 second , some takes 2 seconds, some takes 10 seconds … etc , each page is different

and even same page, this time could be 1 second , next time it might takes 2 seconds …etc because the server performance would fluctuate.

so it’s literally impossible to calculate , or even estimate ??

however on each success run , it will show like last time crawled xxx seconds , crawled xxx pages

for second one , we will think about it.