Optimize cache purging on high-traffic site

  • Resolved dfmcvn

    (@dfmcvn)


    3 months ago

    Hello,
    I’m currently using the plugin on a high-traffic site where most pages are static.
    I’ve set up page caching with rewrite rules to minimize server resources. The Maximum lifetime of cache objects is set to 3600 seconds, and the Garbage collection interval is set to 600 seconds.
    However, I’m experiencing a spike in CPU usage every time the cron job runs every 10 minutes. It renames all expired *.html files to *.html_old, which causes a spike in CPU usage as all pages need to be rebuilt at the same time.
    Is there a way to distribute the server loads more evenly? For example, can the expired HTML pages be cleared sequentially to reduce the CPU spikes?
    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @dfmcvn

    Thank you for reaching out and I am happy to help!

    You should avoid lowering the Garbage collection interval. The default of 1 hour is the optimal solution. There is no need for the cleanup every 10 mins if the content is not updated. What you can also do is use memory-based caching like Redis or Memcached for Page Caching instead of Disk-based caching to avoid this kind of behavior.

    I hope this helps!

    Thanks!

    Thread Starter dfmcvn

    (@dfmcvn)

    Hello @vmarko , thank you for the suggestion.
    We have decreased the garbage collection interval to refresh the comment section regularly while still benefiting from static HTML page caching. Currently, we are not using Redis or Memcached due to the server environment reasons.
    By running the garbage collection every 10 minutes, we will have a lower CPU spike compared to running it every hour (because of the accumulation of expired HTML files).
    Is there a way to distribute the server loads more evenly? For example, a setting to sequentially purge HTML files (instead of all at once), or a setting to serve stale HTML files while the server is creating an up-to-date version?
    Thank you.

    Plugin Contributor Marko Vasiljevic

    (@vmarko)

    Hello @dfmcvn

    Thank you for your feedback.

    Currently, there is no option in the W3TC to achieve this. We are working on a solution to schedule cache purge for specific pages/articles. This will be an advanced solution for these types of use cases.

    Can you please also share what kind of server you are running the website on and specification?

    Thansk!

Viewing 3 replies - 1 through 3 (of 3 total)
  • You must be logged in to reply to this topic.