Disable Changes to Robots.txt

Hello @vsandersssi

Thanks for reaching out regarding your robots.txt file.

I assume you want to prevent the Yoast SEO plugin from outputting all the XML sitemap URLs for all the subdomains in your multisite network.

Can you confirm what version of Yoast SEO you have installed? Because from version 19.8, using the wpseo_should_add_subdirectory_multisite_xml_sitemaps filter, the functionality can be enabled or disabled.

add_filter('wpseo_should_add_subdirectory_multisite_xml_sitemaps', '__return_false' );

Do let us know how it goes

Hi Maybellyne,

Not exactly. There’s no issue right now with the sitemaps.

What I want to do is prevent the following from being written to robots.txt:

# START YOAST BLOCK
# —————————
User-agent: *
Disallow:

Sitemap: https://(oneofmydomains.com)/sitemap_index.xml
# —————————
# END YOAST BLOCK

I’m already writing the directives to the robots that I want (via functions.php) and I’d like to keep your plugin from also writing out the lines above.

Is that possible?

I can send you a link to the robots file privately, if possible.

Also, yes, I am using v19.8.

Hi @vsandersssi,

I would like to let you know that the Yoast SEO plugin, by default, uses the robots.txt file that WordPress creates dynamically. You or your server admin can override the dynamic file by adding a custom robots.txt file at the same server path.

You can create or edit the robots.txt file in the WordPress admin under Yoast SEO > Tools > File editor. You can read more about this here: https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo.

I hope this answers your question.

Again, I’m in a multisite network so I can’t create 1 single file (on disk) to represent all subsites. Also, as I said below, I’m dynamically creating a robots.txt for each subsite and I just want to keep your plugin from adjusting what’s contained within each.

I assume by the silence that there is no solution to this request.

Hello @vsandersssi, I’m sorry it took a while to get a response on the topic; I had to consult with senior support engineers.

Yoast SEO doesn’t create a robots.txt file automatically. WordPress dynamically includes one, which we hook into unless you manually create the file on the server; this is doable via our plugin with the click of a button. So, there’s no way to disable or prevent the creation.

Since you’re in a multisite environment with subdomains, that block of text in the file can’t be removed. But there’s a filter to disable the subsites sitemap link from robots.txt. Further information is available here: https://github.com/Yoast/wordpress-seo/pull/18909.

Please submit a feature request for our product team on GitHub at https://github.com/Yoast/wordpress-seo/issues

You may also consider customizing the robots.txt crawl rules using Yoast SEO’s hook instead of writing yours. But again, there’s no way to remove the existing crawl rules. Here’s the relevant hook to work with https://github.com/Yoast/wordpress-seo/blob/f43026341e91e40bae9950ff9f6f4dae616bf172/src/integrations/front-end/robots-txt-integration.php#L97

Once again, I apologize for the time it took to get back to you.

• This reply was modified 2 years, 4 months ago by Maybellyne.

This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.