W3 Total Cache – Page cache not preloading

Hello,

We have sent a request to HG and i am sure they will answer in several minutes.
The OCP if will work how it says will be the best solution for us, we update daily more than 400 posts and have in our DB about 200.000+ posts.

Again Thank you, i will inform for eventual changes.

Hello,

We installed python 2.7, the web we are testing is:
https://www.noa.al/noa-re/wordpress

I have set a cron job that the script run every 5 minutes , just to test it.

Here is the script: https://noa.al/avin/ocp.py

I have tryed it with “local” option and i got some errors, so i have changed it ” local = False “

Other errors at the xml map, the url of site map is here:

https://noa.al/noa-re/wordpress/sitemap.xml ,but i got another error so i have changed to : https://noa.al/noa-re/wordpress/sitemap.xml.gz

But i see that the script is not uploading the files fast as you said.
Have i done something wrong ?
Thank you

Hi Avin,

You must use Local mode — what makes it fast is that it’s checking the files on the disk instead of the pages via web requests. What error do you get when using local mode?

Make sure that ocp.py can read the files in the ‘pgcache’ folder, by the way — for example by running ‘sudo -u www-data ocp.py’.

Hello

Thank you for the answer, to run the script i use the cron job.
As i said i have installed the other version of python 2.7.
The XMLMAP path is correct, i have changed the var/www/ with https://wwww
same for the local path of pgcache, you can check the pgcache dir your self, url is here :
https://noa.al/noa-re/wordpress/wp-content/w3tc/pgcache/

Please take a look at the OCP.PY file its all ok ?
‘Here is the error : The folder /var/www/noa.al/noa-re/wordpress/wp-content/w3tc/pgcache doesn’t seem to exist. Please ensure that cache_dir points to the base of the local file cache.’

Thnx again.

Hi Avin,

It looks correct.

It sounds like ocp.py doesn’t have access to /var/www/noa.al/noa-re/wordpress/wp-content/w3tc/pgcache or /var/www/noa.al/noa-re/wordpress/sitemap.xml, which would explain why it can’t find the sitemap or the cache files on the disk. You should run ocp.py as a user that can access those folders and files — root or, preferrably, the file/folder owner, e.g. “www”, “www-data” (check ‘ls -la’ in /var/www) — THEN it should work ??

hello,
i haven’t understood it good, please can you tell me the steps if you can
Thank you

Hi Avin,

It’s hard to say without knowing what kind of access you have to the server, and what kind of server it is, but the problem is that the user running the cron job cannot access the files in /var/www, which means ocp.py can’t access them. How are you setting up the cron job? Can you specify that it should be run as an administrator? Do you have shell access?

HG should be able to fix it easily ?? It doesn’t sound like a problem with OCP, just basic file permissions. You don’t need to change the files or folders, ocp.py just needs to be run as the correct user. (Usually the user that owns the files in /var/www is www-data, apache, nginx, or similar.)

Sorry I can’t help more.

Hello,

Ok i will try to configure it, Thank you, i will give an answer if all will be ok

hello

this is the answer

bash-3.2$ /usr/bin/python2.7 ./ocp.py
The folder /var/www/noa.al/noa-re/wordpress/wp-content/w3tc/pgcache doesn’t seem to exist. Please ensure that cache_dir points to the base of the local file cache.

they have tryed with root user, are you sure there is nothing wrong at your script ?

Thank you

Hi Avin,

Sorry if this is a stupid question, but IS the website actually stored in /var/www?

What result do you get if you run the command find / -name "sitemap.xml" as root?

hello

this is their answer…

The convention of cPanel is to separate the document root of each domain by owner. For example, if user ‘fuber’ has a primary domain of ‘bazoids.com’ for its cPanel then the document root of ‘bazoids.com’ will be /home/fuber/public_html. Subdomains and addon domains usually occupy subfolders of public_html. So frob.bazoids.com will live in a folder like /home/fuber/public_html/frob. If you need to know which user own a domain cPanel provides a script for this purpose. The syntax is:
/scripts/whoowns domain.com

Hi Avin,

Then it seems if you change the paths to the sitemap and pgcache in ocp.py from /var/www/noa.al/noa-re/… to /home/yourusername/public_html/noa-re/…, it will work. If you get the same error message as before — that it can’t find the folder — it’s probably not the right path.

If you go to WordPress options -> W3 Total Cache -> Install, does it tell you the path under “Rewrite rules”?

heh, now a huge error

Couldn’t crawl https://noa.al/noa-re/wordpress/2009/10/qeveria-gruevski-distancohet-nga-enciklopedia-ristevski-te-jape-doreheqje/. Error: HTTP Error 404: Not Found
Couldn’t crawl https://noa.al/noa-re/wordpress/2010/11/ruci-berishes-kudo-dhe-kurdo-ti-flet-vetem-per-opoziten-dhe-edi-ramen/. Error: HTTP Error 500: Internal Server Error
Couldn’t crawl https://noa.al/noa-re/wordpress/2009/11/ps-shqiperia-ne-nje-krize-qe-po-thellohet-dita-dites/. Error: HTTP Error 404: Not Found

[Moderator:Cut the rest because it was breaking the forum page]

is any way that the script start caching all the latest posts first, and after start with the older one?

Hi Avin,

Well, it looks like local mode is working, at least. The 500 internal server error makes it sound like there was a temporary problem with the web server. When I run OCP on your website now, everything looks fine.

OCP crawls the pages with the highest priority first. I like your idea, though — I will make it so you can choose a crawl order in the next version.