Physical Robots.txt File Detected

Hey @zavarzin,

Can you verify that, when the warning reappears, there is a physical robots.txt file in your website’s root directory? Perhaps another plugin keeps creating a new one?

– Arnaud

Yes, I verified and there is a physical robots.txt file in the website’s root directory and in the directory of each of my WordPress blogs.

Here are the screenshots:

https://zavarzin-libero.tinytake.com/tt/NTA2MTY0Ml8xNTkwODEwMw

https://zavarzin-libero.tinytake.com/tt/NTA2MTY0Nl8xNTkwODEwNw

It keeps recreating at 12:00 AM each day and its content is this:

User-agent: * 
Crawl-Delay: 20

I’d try to troubleshoot what service or plugin keeps regenerating the physical file. You can also reach out to your hosting provider to ask whether they are generating it.

I have contacted A2 Hosting support and they have replied this:

I checked some of the sites on the account and noticed that the robots.txt file contains references to your sitemap which is generated by All in One SEO. The same was happening on your other blogs.

You’ll see the same on your end. This leaves me to believe the file is being modified or created by the All In One SEO plugin. If you want to remove all your SEO plugins, we can try deleting the robots.txt again to see if it’s regenerated, but otherwise, there wouldn’t be any other service on your account that would create a robots.txt file. WordPress does not create this file on disk by default.

Please let me know.

Hey @zavarzin,

It’s definitely not created by AIOSEO. I know this because of two reasons –

1. AIOSEO does not have any code that can create a physical file, meaning that we only support dynamically/virtually generated robots.txt files;
2. We also do not support the crawl delay directive that’s included in your robots.txt file.

For those reasons, I can only conclude that it must be generated by another plugin or your theme (unlikely). Can you give me a list of the plugins you’re currently using?

– Arnaud

The problem got resolved by the hosting support.

Here’s what A2Hosting support staff has replied:

I’m sorry for the trouble caused by the robots.txt files being created. This is done nightly by a cron job on the server. The server will not create a robots.txt file if there is a file named robots.txt.ignore in the document root of the site. I have added robots.txt.ignore files to the document roots of your sites and these should no longer be recreated.

Just in case someone else will be having this issue I thougt to drop in the solution.

@zavarzin — I am having the same issue, also A2Hosting. Question – did they literally just create just a single file in the root of each of your sites called robots.txt.ignore?

@arnaudbroes — Does adding the robots.txt.ignore file in any way interfere with AIOSEO’s functionality? Meaning, could a crawler get confused by the presence of a ‘physical’ robots.txt.ignore file when you are dynamically creating and serving the robots.txt?

Thanks much to both of you,

John

@jalfr3dalasda1r Yes, that is correct. The hosting support (second line of advanced support) have literally added robots.txt.ignore files to each root directory of all WP sites. Doesn’t seem to make an impact on SEO as I’m on the first page of Google for hundreds of keywords with no issues.

@jalfr3dalasda1r physical files take precedence over dynamically generated ones (if the web server finds a physical file, it will serve that and immediately return before checking the rewrite rules), so AIOSEO won’t be able to serve its robots.txt content.

– Arnaud

I’m getting this error:” It looks like you are missing the proper rewrite rules for the robots.txt file. It appears that your server is running on Apache, so the fix should be as simple as checking the correct .htaccess implementation on www.remarpro.com.”

Any way you can help to troubleshoot this?

Iam facing this problem after i have updated w3 total cache plugin today. What can i do?

@tvhex I’d recommend opening a separate support thread so that we can follow up with you there.