AIO SEO and W3 Total Cache conflict?

Hi @berams,

Thanks for reaching out!

First, it’s important to understand that WordPress generates a dynamic robots.txt for every WordPress site.

This default robots.txt contains the standard rules for any site running on WordPress.

Second, because WordPress generates a dynamic robots.txt there is no static file to be found on your server. The content of the robots.txt is stored in your WordPress database and displayed in a web browser. This is perfectly normal and is much better than using a physical file on your server.

Lastly, All in One SEO doesn’t generate a robots.txt, it just provides you with a really easy way to add custom rules to the default robots.txt that WordPress generates.

I found the robots.txt loads correctly when accessing with the parameter https://travelacroatia.com/cms/?robots=1 – https://a.supportally.com/i/ZTDDQT.

Regarding the W3TC:

It could be a caching problem with W3TC.

Please check the plugin settings and make sure you’re not caching the robots.txt file.
Also, please let me know if the robots.txt file is visible from your end when W3TC is disabled.

Let me know what you find!

When the W3TC is disabled there is no error about the missing robot.txt. I’m not sure how to make sure that robots.txt is not cached.

Hi @berams,

Thanks for the update!

In the W3TC Performance menu, under Performance > Browser Cache, you should add the following if “Do not process 404 errors for static objects with WordPress” is enabled:

robots\.txtto “404 error exception list:”

Here’s a screenshot reference – https://a.supportally.com/i/RkqmJy.

Could you please try the above, purge caches, and recheck the results?

Let me know how it goes.

Thanks!

Hi @berams,

We haven’t heard back from you in a couple of days. I’m going to go ahead and close this thread for now. But if you’d like us to assist, please feel welcome to continue the conversation.

Thanks!