On Multisite, new sites are throwing 404 errors after last update

Hey there,

It took a while to notice, but after some investigating, it appears when new sites are created in a multisite, those sites throw a 404 error for robots.txt when you try to access it when it didn’t before the last update. Sites launched before latest update are unaffected and working properly. They are also not affected if you make changes and update the robots.txt data.

I believe the changelog for that update indicated this was a fix for PHP7, and we have been running PHP7 this entire time and never had issues.

We already know it does not work while it has (what we consider) the “development” path of the site (networkdomain.com/site).

This problem has been occurring after we launch the site and map the domain for that site, including changing the site_url via DB and do a search and replace in the DB tables for what the launched url will be, etc. (e.g. from networkdomain.com/site to site.com)

Once we launched the last couple sites to their respective domains and try site.com/robots.txt it will take you to the 404 page.

Could you assist with this please?

Server Details:
Apache 2.4.29
PHP 7.1.21
WP 4.9.8
Plugin Version 1.9

Additional details:
Applicable DB records in options table mentioning plugin:
_transient_plugin_slugs: "pc-robotstxt/pc-robotstxt.php"
pc_robotstxt: `a:2:{s:11:”user_agents”;s:272:”User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Allow: /wp-includes/js/
Allow: /wp-includes/images/
Disallow: /trackback/
Disallow: /wp-login.php
Disallow: /wp-register.php

Sitemap: https://(omitted)/sitemap.xml”;s:15:”remove_settings”;b:0;}`

On sites where it is working, there is record for robots.txt in the row called rewrite_rules that has: "robots\.txt$";s:18:"index.php?robots=1";

but this does not exist in any of the sites currently not working. I strongly believe this is the problem since index.php?robots=1 will show what is supposed to be robots.txt.

I created a 301 redirect for this on engtech.services which is why on that site if you do /robots.txt you are taken to (and can see in URL) /index.php?robots=1 which is not ideal and need it to be /robots.txt

Websites currently experiencing this issue (that were launched after the latest update):
https://townofdish.com
https://engtech.services

Examples of a couple Websites on the same multisite that were launched pre-update that are working:
https://cityofmaypearl.org
https://www.texassteakcookoff.com

There are no errors in Apache error logs.
Access log shows status code returned as 404 for robots.txt
There are no errors in PHP error logs.

We have the same process that we follow each time we launch a site and has not changed in over a year now. Thanks for looking into this!

The page I need help with: [log in to see the link]