Asking Yoast to look into my Robots.txt?

Hello Marbs,

Thanks for reaching out regarding your robots.txt file.

Indeed, any mistakes you make in your robots.txt can seriously harm your site. For most WordPress sites, the following example is best practice:

# This space intentionally left blank
# If you want to learn about why our robots.txt looks like this, read this post: https://yoa.st/robots-txt
User-agent: *

We even use this approach in our own robots.txt file. These help articles will be beneficial:

https://yoast.com/ultimate-guide-robots-txt/
https://yoast.com/wordpress-robots-txt-example/

Hi @maybellyne thanks for helping me out. So I have changed my robots.txt into this. Is that ok?

# This space intentionally left blank
# This space intentionally left blank
User-agent: *

# blocking search results
Disallow: /?s=*
Disallow: /search/*

• This reply was modified 2 years, 7 months ago by marbs.

Thanks for following up.

Are you trying to block your site’s search result pages from being indexed? Since you are using our plugin, you’re fine. We’ve got you covered. Your internal search results pages are noindexed by default in Yoast SEO. I do not think it is necessary to have extra directives that cover this in your robots.txt. You can learn more at https://yoast.com/blocking-your-sites-search-results/

@maybellyne I actually want to be found. I am trying to recover on search engine Google after using Rank Math.

So I don’t have to include the search. My robots.txt should be look like this to be discovered by search engines?

# This space intentionally left blank
# This space intentionally left blank
User-agent: *

Hey @marbs,

Ah okay. Yes, looks good and similar to our example and explanation here: https://yoast.com/wordpress-robots-txt-example/

I marked this thread as resolved due to a lack of activity by the original poster. The original poster is always welcome to re-open this topic.

If you are not the original poster but have a similar issue, please read this post before opening a new request.