Noticeable Drop In Performance Of The Crawler

Hi There! ??

Since updating to version 5.5 of the Litespeed Cache plugin, I seem to have noticed a drop in the performance of the Crawler.

I saw that the changelog introduced some new changes to the Crawler in this version, which I do welcome as they’re a decent improvement to it.

Though I’m wondering if any of these changes were intended to alter it’s performance – as previously I had mine configured to run successfully with the following values:

Server Load Limit: 3
Threads: 4
And it would successfully run through my entire sitemap of 21000 URLs on either 2/3 threads without ever hitting a “stopped_highload”.

However, since updating to version 5.5, with the same server load of 3 – I am only be able to run 1 thread & crawl between 20-200 URLs before hitting a “stopped_highload”

I’ve since bumped these figures up to:
Server Load Limit: 5
Threads: 4

And it once again is able to crawl through the entire sitemap without an issue on a mix between 3/4 threads.

I understand that these results could vary wildly based on resource consumption, but there hasn’t been any major changes in my environment since then & my CPU idle has consistently been sitting around 70-80% when no crawler is running. (4 cores, 8 threads)

Just thought I’d bring this up in case there is something else going on which perhaps Litespeed could pick up on that I’m not aware of?

• This topic was modified 1 year, 5 months ago by seanbremner.