Google unable to fetch robots.txt

  • Resolved asen85

    (@asen85)


    3 years, 6 months ago

    Hi All,
    Google Search Console reports that Google is unable to fetch my robots.txt file.

    “You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file”
    Do you know how can I fix this?
    I’m using Yoast.
    Thanks in advance! ??

    The page I need help with: [log in to see the link]

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support devnihil

    (@devnihil)

    @asen85 We’re sorry to hear you are experiencing this issue on your site.

    We checked on your robots.txt at https://gearuphiking.com/robots.txt and see that it contains only the word ‘test’ as it’s contents (see image here).

    Can you please try editing the robots.txt file to contain valid syntax and then recheck it in Google Search Console? We have information on robots.txt file here: https://yoast.com/wordpress-robots-txt-example/

    You can edit the robots.txt file by performing the following steps:

      ? In WordPress, go to SEO>Tools
      ? Click on ‘File Editor’
      ? Under ‘robots.txt’ beneath ‘Edit the content of your robots.txt:’ is a field from which you can directly edit your robots.txt file
      ? After making any changes to the file, click ‘Save changes to robots.txt’

    We also have more information on this and editing the robots.txt file here: https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/

    Thread Starter asen85

    (@asen85)

    Thanks for your answer.
    My host provider Siteground re-created the file since it wasn’t present in File Manager.
    They modified it and entered “test”. But the issue was occurring before that.

    Should I edit the file like this

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Also how would that solve the error I’m seeing in Google Search Console stating that they are not able to fetch it?

    • This reply was modified 3 years, 6 months ago by asen85.
    Plugin Support Michael Ti?a

    (@mikes41720)

    Hi,

    You can refer to these articles for more information on what would be the ideal directives that you should add to your robots.txt file:

    Plugin Support devnihil

    (@devnihil)

    We are going ahead and marking this issue as resolved due to inactivity. If you require any further assistance please create a new issue.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Google unable to fetch robots.txt’ is closed to new replies.