W3 Total Cache – Page cache not preloading

How do you know for sure that it is not working? It starts working according to the cron interval.

It is easy to see the preload is not working because by site is new and traffic is almost zero. I browse the site and every page is not cached until a second visit. Also, the page cache debug is turned on, and says not cached until second visit. Also, I can see the (general lack of) pages in wp-content/w3tc/pgcache.

You are saying it works off cron? Where is that documented? I checked the cron log and there is no sign of a W3 related job triggering at any time.

I too was having trouble with cache preloading.
I checked the sitemap.xml location, and its well formedness, set the preloading interval to 30 seconds (to make it happen soon), then saved.
I checked the page cache for the next few minutes with an ftp program and no cached pages were created in \wp-content\w3tc\pgcache\
After research of wp_cron I found that it is triggered by any page load (which makes sense, everything needs a trigger).
So I called the home page in a browser.
Sure enough, 10 pages appeared in \wp-content\w3tc\pgcache\.
I imagine that by setting up a cron job in your hosting panel to hit the file, [ABSOLUTE PATH TO WEBROOT]\wp-cron.php, then the pagecache could be populated x number of pages () per cron run even without requiring site visitors to trigger it.

Huge thank you to Frederick for creating such an awesome plugin.
I struggle to think of any feature which would make it better.

I’m having the same issue. I have a “semi-static” home page where I have posts that are being pulled in random order. I also have a script where it cycles through a set of quotes.

I have cache preload setting “automatically prime the page cache” enabled, 900 seconds, pages per interval 10 pages.

I verified my sitemap: https://fearlessflyer.com/sitemap.xml to be valid.

I also enabled “HTTP compression” and “Expires header lifetime” via Browser Cache settings (per note in the page cache settings page).

Now, the reason that I can tell that it’s not working is after an hour or so – the random post / random quote / random testimonial – is still the same order.

Am I missing something? my website is: https://fearlessflyer.com/

Discard my last entry – I discovered that it is flushing the cached pages. It just took longer than I was expecting.

Thanks for an awesome plugin.

Unfortunately, there are no perfect settings to ensure that an entire blog is preloaded without risking that all server resources are exhausted to get the job done. That is, WP Cron is used to do this and as a result, a certain ratio of traffic to cached pages must occur in order for enough “attempts” to prime the cache to happen. There’s no ideal algorithm that works for all servers although the settings can be tuned. If someone has a better algorithm, I’m all ears, for now I don’t see a way to implement this without creating other liabilities. For that reason the site map was used with the ability to set priorities so that reasonable amounts of control are possible.

A number of separate issues that all affect w3 cache preload are being discussed in this thread.
1). Firing of wp-cron.
This can be controlled by hitting wp-cron with an http request from the hosting account’s crontab or by using an external spider application.
*/30 * * * * wget -O - https://domain.com/wp-cron.php > /dev/null 2>&1
will fire wp-cron every 30 minutes whether the site has traffic or not.
This has no liability. If wp-cron has tasks scheduled they will run in the same way as if a website visitor had triggered them. This would have to be setup independently of the w3tc plug-in.
2). Pages missing from cache because garbage collection has deleted them and w3tc is waiting for next cron run to rebuild them.
This could be eliminated if the w3tc plugin first built a path to, then deleted the cached page before its http request to rebuild it. This would have a very small resource liability. The resource liability could be furthur minimised by adding a sleep after each http request.
3). Garbage collection only deletes files whose html expires header lifetime has expired. (which is set up on a different page to cache preload but has a huge effect on it)

Adding sleep operation to scripts is actually non-performant. Lots of these points sound reasonable, but they do not scale.

I too pondered this a bit and it seemed like the ideal would be to do the priming outside PHP, so I ended up writing a Python script, Optimus Cache Prime, which does essentially the same as the W3 priming feature, except for a few things: It supports throttling (to my knowledge, PHP can’t really do this without blocking other requests), but most importantly it checks the state of your static file cache before making any requests, so only the pages that aren’t already in the cache get crawled.

After doing some testing, it seems that OCP can check around 10,000 pages per second with WordPress and W3 Total Cache, assuming that the cache is already mostly primed (if not, there would be a lot of requests).

You need a server with Python and the ability to run a script to use it. Hopefully someone will find a use for it ??

when trying to execute the script with python i get error line 113

urlmap[url.text] = prio.text if prio is not None else ‘0.0’

I have read about OCP and its just FANTASTIC, at our website we have more then 200.000 posts that need to be preloaded, with OCP our problem is solved, so please tell me why i get this error?

Thank you

Hi Avin!

Glad to hear it will be useful for you ??

Could you tell me the URL of your sitemap and your Python version (‘python –version’) — I will try to recreate the problem.

Feel free to email it to contact at pmylund dot com

Hello,

Thank you for the answer, my python version 2.4.3. and for site map we use xml Site map a wordpress plugin.
Thank you

Hi Avin,

I will test it, but if you have any possibility of upgrading to Python 2.5, 2.6 or 2.7 (https://python.org/download/), that would be an easy solution. Python 2.4 is very old. (It’s from 2004.)

Hello,
we use HG servers and we will ask if they can update our python, you think this is the problem i get the error ?
Thank you again for the answer.

Hi Avin,

I just tested with Python 2.4, and got the same error. I fixed the line mentioned, but the module for parsing the XML (ElementTree) didn’t exist in Python 2.4, so it won’t work without major alterations.

I am almost completely sure it would work if you tried with Python 2.5-2.7. If you are able to compile applications, you can compile the Python sources without root access.

Please let me know how it goes ??