Google index: “Failed: Robots.txt unreachable”

  • Resolved geekayhu

    (@geekayhu)


    1 year, 2 months ago

    Hi there,

    I hope you guys can help me with this because I have no idea why Google can’t index this site. I have already gotten in touch with the hosting company, I have several other sites with them, and I had no issue with sitemaps or this type of indexing issue, and no restrictions from the hosting side.

    Both the robots.txt and sitemap are reachable from the browser (for me).

    What is still strange to me is that (via the “URL inspection tool”) it already “seems to work” on the “Google Index” tab (https://ibb.co/RzN77J1) –> “page fetch: successful”, but the “Live test” tab says it’s blocked (https://ibb.co/wRz4xJc).

    And the sitemap can’t be read, either (https://ibb.co/HDCtcC2), however, if I click the “Open sitemap” link on the page, it opens the sitemap.

    Any help would be appreciated.
    Kind regards, Gabe

    The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Support Rank Math Support

    (@rankmathsupport)

    Hello @geekayhu,

    Thank you for your query and we are so sorry about the trouble this must have caused.

    We checked both of your robots.txt and sitemap URLs under the Googlebot user agent and they seem to be loaded without any issues.

    Can you please try clearing your website caches and clicking on the “REQUEST INDEXING” on the live test report?

    Let us know how that goes. Looking forward to helping you.

    Thread Starter geekayhu

    (@geekayhu)

    Hi there,

    thanks for your quick response.

    I rechecked within the Google Search console, and it is a bit strange to me that the page now seems to be indexed (https://ibb.co/Jqk2H7W), but the sitemap still cannot be fetched (https://ibb.co/7kNNxGY), and if I click the open sitemap, it successfully opens the sitemap. Unfortunately, I still don’t really understand what the “general HTTP error” really means, as there is no specific description in the help center, at least I can’t figure it out.

    And the URL inspection returns the error of the robots.txt (https://ibb.co/DM3PYQz)

    Any other suggestion you may come up with? Server/hosting does not block anything, and I have no idea what to do next. (Does this affect Google search hits? I mean, the page is indexed, but the sitemap cannot be fetched, so it means that Google should list this site in search, right? Sorry if the question is silly)

    Thank you for your assistance
    Kind regards, Gabe

    Plugin Support Rank Math Support

    (@rankmathsupport)

    Hello @geekayhu,

    The sitemap file will never be indexed and that’s perfectly fine. That file is only to help Google crawl the website and it should be added only to the sitemap section of GSC as you already did.

    Since the sitemap returned an HTTP error when Google parsed it, it’s possible that the bot encountered a captcha on the page or the sitemap was temporarily down.

    Due to the fact that it seems to be accessible now we would recommend that you remove the sitemap and add it back again to GSC and see what is the result of the next crawling.

    Hope this helps solve your issues.

    Don’t hesitate to get in touch if you have any other questions.

    Thread Starter geekayhu

    (@geekayhu)

    Hi there,

    Thank you for the suggestion. Finally, we reinstalled the whole site, actually deleted the domain itself, and readded due to it being “registered” as “parked…”. I don’t know if that was the problem, but now Google has indexed the site and has no problem with the sitemap.

    Have a wonderful day!
    Kind regards, Gabe

    Plugin Support Rank Math Support

    (@rankmathsupport)

    Hello @geekayhu,

    Glad to hear that your site is now indexed without any issues.

    Don’t hesitate to get in touch with us if you have any other questions.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Google index: “Failed: Robots.txt unreachable”’ is closed to new replies.