Crawler Threads & Server Load Limit

  • Resolved webarkitekterne

    (@webarkitekterne)


    1 year, 12 months ago

    Hi,

    Our webserver is a very fast 4 core (AMD) machine with plenty or RAM, but when a crawler runs, the frontend becomes very unresponsive (from less than an sec serving a page to 30-50 sec).

    Settings for Treads are 4 and Server Load Limit is 3.

    We have thousands of pages and as you can see in this image, the runtime for all 8 crawlers with 4 treads goes up to 75513 sec (about 21 hours), meaning the frontend are quite slugish a lot of the time over a day. We have daily updates to many pages, so we need to run the crawlers daily.

    How do we setup litespeed better?

    See all crawler settings here.

    Thanks!

Viewing 15 replies - 1 through 15 (of 30 total)

  • Do you have access to command line interface? (SSH)

    Thread Starter webarkitekterne

    (@webarkitekterne)

    Yes, our techie has access. What would you have him do?

    Run “uptime” command while the crawler is running and post the result.

    • This reply was modified 1 year, 12 months ago by serpentdriver.
    Thread Starter webarkitekterne

    (@webarkitekterne)

    He will do it tomorrow.

    Plugin Support qtwrk

    (@qtwrk)

    what’s your env ?

    any control panel in place ?

    the default max php concurrency is 10 , this might be the bottleneck

    the default max php concurrency is 10 , this might be the bottleneck

    If @webarkitekterne complains that the frontend seems to have glitches, then I suspect that his server is not designed for the amount of articles. The specified values for the CPU cores and RAM then seem to be more vCores and vRam and he runs shared hosting.

    Thread Starter webarkitekterne

    (@webarkitekterne)

    Uptime result (while crawler 8 is running):
    192958 up 16 days, 23:59, 1 user, load average: 1.74, 1.77, 1.83

    Thread Starter webarkitekterne

    (@webarkitekterne)

    It’s a cloud solution with 4 vcores and 16bg vram. Once the crawlers are done, the site is lightning fast. A page (post) not crawled is about 10 sec to load, and up til 50 sec while any crawler is running (if that page is not in cached).

    I can’t say with absolute certainty because more information would be needed, but it looks like your server is chronically overloaded and you should consider switching to a more powerful server. In any case, you should reduce the number of threads to a maximum of 3.

    Thread Starter webarkitekterne

    (@webarkitekterne)

    Crawler Status:
    28 Nov 2022 19:38:15, Size: 5541, Crawler: #8, Position: 2238, Threads: 4, Status: crawling, updated position

    It’s a cloud solution with 4 vcores and 16bg vram.

    I had already assumed that these are virtual resources and that you have to share these resources with others. Given your high item count and the fact that WooCommerce is a performance killer, you should consider switching to a dedicated server.

    Thread Starter webarkitekterne

    (@webarkitekterne)

    you should reduce the number of threads to a maximum of 3.

    Means crawling will take more that 24 hours.

    We will switch to a dedicated 12 core server soon. Still would like to dial in the perfect crawler settings before this happens.

    Thread Starter webarkitekterne

    (@webarkitekterne)

    the fact that WooCommerce is a performance killer

    Not really using woo much, but are heavy on ACF posts.

    Just to give you an example. I have developed an alternative crawler that crawls around 90.000 URLs on a non-WP site in just under 5 hours. On a Shared Hosting.

    Not really using woo much, but are heavy on ACF posts.

    That doesn’t matter. WordPress + WooCommerce + Plugins kills performance.

Viewing 15 replies - 1 through 15 (of 30 total)
  • The topic ‘Crawler Threads & Server Load Limit’ is closed to new replies.