Robots.txt missing default wp-admin lines after Yoast SEO 19.0

Hello Jos & Sean,

Thanks for reaching out regarding your robots.txt file. Indeed, when you update to Yoast SEO 19.0, the robots.txt now follows our recommendation for best practice. This involves relying on your robots.txt file as little as possible; it’s only really necessary to block URLs in your robots.txt file when you have complex technical challenges.

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap_index.xml

The wp-admin blocks are unnecessary and WordPress is increasingly moving away from admin-ajax. Also, there’s not really much value in hiding sitemaps anymore.

You can read more in our suggested best practice and release post

I hope that helps!

Hi @maybellyne,

Thanks for the update.

I already checked both links and as I understand that it’s not necessary anymore, I missed the explanation for it and the mention of that change in the changelog and/or release post. So I hope you can work something out and make a blog post about it with reasons why it was there and isn’t needed anymore now.

And maybe if we need to make some changes ourselves, because the only reference I see in the best practice article is about the x-robots HTTP header, but I can’t see that on my pages and don’t know how that works.

@registriran , @sean-h Please create your own topic for your questions to keep this topic clear.

Hi,

It is mentioned in the article that WordPress and Yoast SEO already automatically prevent indexing of some sensitive files and URLs, like your WordPress admin area.

If there’s any more modification to disallow any areas, you can check out our article about the x-robots-tag HTTP header and how to modify it – https://yoast.com/x-robots-tag-play/

Hi @mikes41720,

I know that it’s mentioned, but I was asking for an explanation, because that’s still missing.

Was the wp-admin folder blocked earlier for security reasons, to save the bots some work or another reason? What changed?

Thanks,
Jos

I’m with Josh on this one. Maybellene references ” Also, there’s not really much value in hiding sitemaps anymore.”, he/we are not hiding the sitemaps it has to do with preventing indexing of wp-admin URLs and preventing them from crawling them so not sure how that can be bad practice. Since we updated we have seen a huge drop in google traffic and only connection with start of the drop and now is this one change.