Couldn’t find a robots.txt file

  • Resolved TWD

    (@twd)


    6 years, 1 month ago

    Running WP 4.9.8
    I couldn’t see any robots.txt file anywhere on the site.
    (doesnt WP install one by default?)
    Is it possible that Yoast removes the default robots.txt file?
    Because when i type in the URL https://www.mysite.com/robots.txt
    I see this:

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

    So I just created a new file, appended your code to the end so now I have this in my root directory. Does this look right?

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?blackhole
Viewing 1 replies (of 1 total)
  • Plugin Author Jeff Starr

    (@specialk)

    Yeah that looks fine. Also just to be sure, you can use a free online robots.txt checker to validate. Google provides a good robots checker in its Webmaster Tools (requires registration). Should be other free tools as well.

Viewing 1 replies (of 1 total)
  • The topic ‘Couldn’t find a robots.txt file’ is closed to new replies.