mapped domain robots.txt 404

BTW, your link on Multisite Robots.txt Manager – Network Settings’ page to BWP google XML Sitemaps goes to ‘Google (XML) Sitemaps Generator for WordPress’ — not Better WordPress Google XML Sitemaps (BWP Google XML Sitemaps) .

Hello sslone,
This issue can sometimes happen when a different robots.txt manager was previously used. The other plugins leave a robots.txt file in the Websites settings (network admin > sites > edit site > settings) – if this is the issue, locate the old robots.txt file in the settings, make it blank, then save your settings.

My plugins robots.txt setting name is: MS Robots.txt – make sure all other robots.txt files are blank, other than the MS one.

….. I’ll correct the links, thanks.

Let me know if this helps ~Chris

Hi.

We’re experiencing the same issue : the default robots.txt is working but any other one (using WordPress MU Domain Mapping) is not.
It ends up in 404.

I’ve checked the settings, there’s nothing but “Ms Robotstxt” in the sites settings.

The default site has however another setting that is “Ms Robotstxt default”.

I actually had to go into the database tables for each site and make sure that the robots.txt was enabled – there was a difference for the sites that correctly displayed the robots.txt file, and the ones that did not. I didn’t record the table or field that needed tweaking, though, but i believe it was in options table.

There’s not a lot of fields in the option table.

The “autoload” field is set to “yes” on the working/default site…
While it’s set to “no” on the non working sites.

I’ve just tried to put “yes” instead of “no”…
Not working either, still 404.

Still not working 8(

Actually it seems what should not be working (sub-folder sites) is working while domain-mapped sites are not.

https://wp-mu.domain.tld/site/robots.txt works and contains the correct data.

https://www.domain-mapped-site.tld/robots.txt gives a 404.

We’re using this domain mapping plugin :
https://ocaoimh.ie/wordpress-mu-domain-mapping/

Hello networkstudio,

I will be releasing a new version of the plugin later today or first thing tomorrow morning. The new version removes the path check and doesn’t care if domains are mapped or not. It should correct the issues you’re having.

~Chris

Any update on the new version? I’ve got site that was working and now is no longer resolving – getting the 404 error again.

i actually went in to the options table (rewrite_rules) and added the text below – this fixed the issue for the single site that was delivering a 404 for the robots.txt

s:12:"robots\.txt$";s:18:"index.php?robots=1";

Hey sslone…
I thought it would be ready this morning but I ran into a small bug. I’m just doing some final testing and updating help documents. It should be ready Thursday morning.

Any idea how that rewrite rule was removed?

I will write function to detect the missing rewrite rule and a button to click that will add the rule back if it’s missing.

~Chris

This is _exactly_ our issue…

The rewrite rule is not added when a new site/blog is created.

I’ve just added it manually (once per site/blog) and it now works.

Quick update… I’ve created a function that detects and corrects missing rules and/or the missing robots.txt line. As well, a function that detects/removes robots.txt file data from old plugins.

And finally, I’m still testing the newest version. It should be ready today (and yesterday, and the day before), assuming I don’t find any more pain in the butt bugs.

~Chris

Hello again…

Thanks sslone for finding that rather odd problem.

The newly released version now detects the missing rewrite rule and it looks for old robots.txt plugin data. You can also manually run the checks via the Help tab. When an issue is found, an info box with buttons will display, allowing you to correct the issue.

Let me know if you have any issues with the new feature.

~Chris