Is this robots.txt by yoast recommended?

Hello Swag,

Thanks for your question about the robots.txt. This isn’t the default directives that would be found in a fresh/default file by Yoast SEO. You can check out this helpful article to see what it should look like. However, if you are convinced that those extra directives are needed, you can leave them as it.

Thank you so much Maybellyne, for your reply.

I am not an expert, so I have no idea how to write robots.txt.

I can assure you that this robots.txt was created by yoast plugin, and this was done not just for one site but for two sites.

Actually yoast was already installed on my wordpress sites, but to explore other SEO plugins, I temporarily uninstalled Yoast.

However, I ultimately realized that Yoast is the BEST SEO plugin, therefore I reinstalled Yoast.

After reinstalling, when I checked my robots.txt file, I found the above mentioned elements in it.

Previously it looked like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

After reinstalling Yoast, it changed to this:

# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow: /?s=
Disallow: /page/*/?s=
Disallow: /search/
Disallow: /wp-json/
Disallow: /?rest_route=

Sitemap: xxxxxxxxxxxxxxxsitemap_index.xml
# ---------------------------
# END YOAST BLOCK

For further confirmation, you can check my sites with the following URLs:

homemade-circuits

.com

makingcircuits

.com

Kind Regards

Swag

• This reply was modified 1 year, 1 month ago by swagatam1975.

Since you uninstalled the plugin at some point, other plugins may have added directives to the WordPress dynamic/virtual file at yourwebsite.com/?robots=1. This is what the Yoast SEO plugs into when you are using our plugin.

But you can edit the file in WordPress > Yoast SEO > Tools > File Editor to match the Yoast SEO guidelines.

OK, Thank you very much for your kind reply!

I will try that!