Limit the visibility of minified files for Googlebot

Hello,

To prevent Googlebot from spending too much time on your minified CSS and JS files, you can use the robots.txt file. This will help improve your site’s crawl efficiency.

Here’s how to do it:

1. Open or create a robots.txt file in your website’s root directory.
2. Add the following lines:

User-agent: Googlebot
Disallow: /assets/cache/wpo-minify/

This tells Googlebot not to crawl the minified files in the /assets/cache/wpo-minify/ directory.

Thanks!

Hello wpmansour and thanks for your answer

I considered it, but I don’t think it’s good practice to block these files from Google. They most likely use them to compute the rendering, especially on mobile. They’re used to evaluate things such as Core Web Vitals. I expect more harm than good by blocking search engine bots from crawling that kind of file. What do you think ?

Would you have other ideas ?

Regards

These files created by merging. The first thing I would do is check if you site supports http2 using this tool: https://tools.keycdn.com/http2-test

If it does then you should not have merging enabled. It will be slowing your site down.

Thanks for your answer, i will try it since the HTTP2 test is positive

Should i then use a 301 redirection from these files to the corresponding css or js ?

I would not use a 301 redirection, I would leave them as they are.

Google render pages, meaning that they put together a page like a browser does, (it is basically a version of chrome), including executing JavaScript on the page.

That means they need to fetch the files to be able to execute them, and crawling is how googlebot gets things.

Sure, but wouldn’t these individual merged files become obsolete once i disable merging ?

Regards

Yes, but you will have lots of smaller files in their place

When loading a website from an HTTP1.1 server, A single request is processed at a single time period. therefore, the browser will first download the CSS, then another CSS, then another JS, then an image, and then the page will be rendered. This happens one after another, until all assets are loaded on the page. If you check a waterfall chart for this type of website loading, you will see that one request ends, and another one begins.

That’s why the conventional wisdom is that merging all of these requests into a single larger request is true, and will speed up the loading time of your website. And this is absolutely true. A single HTTP request on an HTTP 1.1 based server will always load quicker than multiple, as there is no waiting period.

With HTTP 2, All requests are loaded simultaneously. There is no waiting process, and there are the same number of connections for the number of assets being requested. Therefore, merging and minifying will simply create a larger file, taking a longer time to upload and download, leading to a longer render process for the website. By keeping all of your individual files unmerged and minified on WordPress, they can all load at the same time, and each upload and download is for a smaller file, resulting in a quicker loading time.

Thanks for your explanation. I’d like to mention, tho this is a parallel information, that Googlebot crawls using HTTP 1.0 and HTTP 1.1

So once i disable merging, shouldn’t I clean up the obsolete files ? Do you have a recommendation on how to proceed ; are they located in the same folder, different depending on whether they’re merged or not ?

Just clear the minifed cache by going clicking the “Purge the minified files” in the “Minify status” tab

https://snipboard.io/KJSPF1.jpg