Preload facility

Hello @bsthp

Thank you for your question and I am happy to help.
Yes, W3 Total Cache has a Cache Preload option in Performance>Page Cache>Cache Preload.
What the feature does is this:
– Check the last offset
– Check how many URLs are allowed to be processed per run
– Fetch all URLs from sitemap (even nested sitemaps work)
– Loop through a set of URLs to visit them`
Here are some general rules for Cache Preload:
Example:
You have 1000 pages.
Update interval 900s (the number of seconds to wait before creating another set of cached pages.)
Pages pr interval 10 (Limit the number of pages to create per batch. Fewer pages may be better for under-powered servers.)
This means that 10 pages will be cached every 15 minutes so your entire website will be cached in 25hrs, You can, of course, adjust these settings and reduce the update interval or increase the pages per interval. this is entirely up to you but take note that this may increase the usage of server resources.
The preload is based on wp-cron so it doesn’t work when you don’t have WordPress-handled requests. It would be a good idea to add define(‘DISABLE_WP_CRON’, true); and Once this is done, you should create a custom cron job that will call wp-cron.php at a set interval.
Make sure the sitemap file is set and contains URLs
If you don’t want to rely on wp-cron – use the wp-CLI command for preloading and Unix cron
wp w3-total-cache pgcache_prime
I hope this helps!
Thank you!

Marko, thanks for your detailed response. On activating the above, the site is now displaying the Warning: home/sites/xx/x/xxxxxxxx/public_html/wp-content/plugins/jet-data-importer/jet-data-importer.php on line 371 in the front-end. Any ideas what setting in Total Cache may be causing this?

Hello @bsthp

Thank you for the information.
The most likely problem is that the mentioned plugin performs with native wp_cron.
Try to remove define(‘DISABLE_WP_CRON’, true); and let me know if the problem persists.
Thanks!

Marko, thanks for your feedback. It seems to have resolved itself. However, I am having an issue with Preload. Checked the settings over and over and there is nothing obvious. Using nearly all default settings with Disk: Enhanced, where appropriate. For testing purposes, I set the preload interval to 900 secs and pages per interval to 100 (site only has less than 10 pages). I used the XML Sitemaps Generator by Auctollo and referenced it in Sitemap URL. Within /sitemap.xml, it includes a sitemap for each of the pages i.e. each time a new page or post is created, it appends a new xml sitemap within sitemap.xml. The Homepage appears to preload but internal pages don’t. To check, i visited the pages, then looked at the source to see the Cache timestamps. The Cache on the internal pages, showed the page was cached when i manually visited it i.e. it did not have an earlier preload stamp.

Would it be possible to point me in the right direction or help in resolving the issue?

Hello @bsthp

Thank you for your feedback.
Can you please share the website URL?
Have you added the sitemap URL https://yourwebsite.com/sitemap_index.xml or you just added sitemap.xml?

Thanks!

Marko, thanks for your response. Details are:

Website & Sitemap: https://www.webstarter1.co.uk/sitemap.xml

sitemap.xml is not a physical file on the server i.e. when viewing the site via ftp, it is not there.

Hello @bsthp

Thank you for the feedback.
You should add the sitemap URL: https://www.webstarter1.co.uk/sitemap.xml
The Preload is initiated once the website is visited if you are using native wp=cron.
You can also try using wp-cli
So, instead of calling wp-cron via crontab – you should call it directly via
wp w3-total-cache pgcache_prime
and track the output which pages/sitemap positions are primed by script as logging is available now with wp-cli
I hope this helps!
Thanks!

Marko, are you saying: to create a physical sitemap.xml file in the root i.e. public_html/sitemap.xml?

I have already deactivated native wp cron i.e. define(‘DISABLE_WP_CRON’, true); and created a scheduled cron job on the server to run every 15 minutes.

Hello @bsthp

I am not saying that. You already have the sitemap, you just need to add the URL to the W3TC settings.
Can you please confirm that the settings are set as per the screenshot below:

Thanks!

Thanks for feedback. Yes, the url has always been set to that and is exactly the same as your screenshot.

Hello @bsthp

Thank you for the information.
So if you added the define(‘DISABLE_WP_CRON’, true); and once this is done, you should create a cron job that will call wp-cron.php at a set interval?
To test this, delete the wp-content/cache/page_enhanced/ folder, and check after the set custom cron time if the new pages are cached.
Most likely the pages are already preloaded and since you have only a few pages (preload may not be needed here) all pages are preloaded over and over again in a 15 min preiod.
As for the time stamp, This depends on the Browser Cache last modified, so if you use disk enhanced with modified time, the user will always see an up-to-date page.
Thanks!

Marko, thanks for your feedback. Still no joy but making progress, I think. I tested my scheduled server job (wp-cron) to make sure it works, which it does. Set to run every 5 minutes at the moment. Checked the cron events in WP and w3_pgcache_prime runs every 15 minutes and is updated when the cron job is run every 5 minutes. I removed the folder you suggested, above, and when the cron events and cron job were run, it re-created the page_enhanced folder. I run this test for 2 cycles i.e. 2 x 15 minutes, then went to the /page_enhanced/www.webstarter1.co.uk folder. The only page that was cached, was the index page. Also, after the second cycle i.e. when the preload was run for the second time after 15 minutes, I checked the time stamp at the bottom of the source code of the index page and it was still the page that was cached in the first cycle i.e. It had teh same time stamp and I thought it would have moved it on by 15 minutes.

Hello @bsthp

Thank you for the feedback.
It seems that something is not set up correctly. Either the custom server cron or something with the sitemap.
I’ve tested this with the sitemap generated by YOAST and native wp-cron (triggered preload by visiting the website) and the pages are being preloaded.
I would advise using the sitemap https://www.webstarter1.co.uk/sitemap_index.xml and disable preload using wp-cli wp w3-total-cache pgcache_prime
and track the output which pages/sitemap positions are primed by the script as logging is available now with wp-cli
Thanks!

Thanks for your feedback. I removed the XML plugin that I was using and used Yoast’s xml sitemap (https://www.webstarter1.co.uk/sitemap_index.xml), instead. I then reactivated the native WP cron and visited the Homepage. Same issue, unfortunately. Only the homepage was preloaded. Am I right in thinking that preloaded pages should appear in the /wp-content/cache/page_enhanced/www.webstarter1.co.uk folder? Currently, this folder has the sitemap_index.xml folder and _index_ssl.html_gzip_old and _index_ssl.html_old files.

Hello @bsthp

Yes, that is correct. After a period of time, the pages should be cached in the /wp-content/cache/page_enhanced/www.webstarter1.co.uk folder.
How long have you waited before checking the folder?
Thanks!