Robots.txt cannot be fetched

  • Resolved jvsouthwood

    (@jvsouthwood)


    3 years, 3 months ago

    Since 27 July Yoast update, Google Search console has stopped indexing a few pages and gives: ‘Page indexed without content’. The google robots.txt Tester gives: ‘robots.txt fetch failed. You have a robots.txt file that we are currently unable to fetch’.

    I have rewritten and resubmitted the robots.txt multiple times, no luck. Deactivated security plugins, removed caching, etc. No joy. You can see it working at link above, but google obstinately refuses to see it.

    On Mobile friendly tester https://search.google.com/test/mobile-friendly?id=73Y9Jpr0GhI7Z0rYDVz1Sw
    I get ‘Page cannot be reached. This could be because the page is unavailable or blocked by robots.txt’

    The Yoast update on 27 July apparently directly crashed my site, so I am tracing the issue back this way. Any way to get Google Search to recognise my robots.txt again? At wits end so any further tests or ideas welcomed.

    The page I need help with: [log in to see the link]

Viewing 6 replies - 1 through 6 (of 6 total)
  • Plugin Support Michael Ti?a

    (@mikes41720)

    Hi @jvsouthwood

    Upon checking your website https://londonsavate.co.uk/, we see it is accessible and it is using the latest version of Yoast SEO v16.8.

    Upon checking your robots.txt file here, it doesn’t look like it is using the standard directives or rules, which might be causing an issue as well.

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php  Disallow: /wp-content/uploads/wpo-plugins-tables-list.json

    You may want to check Google’s robots.txt file tester to get more information – https://support.google.com/webmasters/answer/6062598

    You might need to make changes to your robots.txt file to ensure that the crawlers and bots aren’t disallowed and will be able to crawl your website – https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/ & https://yoast.com/ultimate-guide-robots-txt/

    Thread Starter jvsouthwood

    (@jvsouthwood)

    It was by using the google robots Tester and not getting joy (original opening para) that prompted me to ask why it might be happening.

    When you say ‘doesn’t look look it is using standard rules’ what in particular are you seeing? I am using a recommended version. Thanks.

    Plugin Support Michael Ti?a

    (@mikes41720)

    Hi,

    This is the default WordPress robots.txt file:

    User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

    We’re not familiar with this other directive you’ve added:

    Disallow: /wp-content/uploads/wpo-plugins-tables-list.json

    When you test your robots.txt file in Google’s robots.txt tester, could you share more information on exactly what it is reporting? Perhaps there is information that could point as to why Google is unable to fetch your robots.txt file.

    Thread Starter jvsouthwood

    (@jvsouthwood)

    Yep, the result in Google’s robots.txt tester is the title of this thread:

    Robots.txt cannot be fetched
    You have a robots.txt file that we are currently unable to fetch. In such cases we stop crawling your site until we get hold of a robots.txt, or fall back to the last known good robots.txt file

    As I said, this all started when an update to yoast took my site down on 27 July, so I am trying anything, any ideas at all to get back my indexing.

    (Disallow: /wp-content/uploads/wpo-plugins-tables-list.json is from WP-optimise to guard against access to plugins, it’s fine and recommended, but I have removed the plugin now to eliminate possible causes).

    Any ideas why google will not even look at my robots.txt in live testing? Anyone come across this before?

    Plugin Support Md Mazedul Islam Khan

    (@mazedulislamkhan)

    Thanks for your confirmation. While it might be a coincidence that you’re seeing the issue on Google Search Console after the Yoast SEO plugin update on 27th July, we don’t see any issues at all on the robots.txt file on your website. We can confirm that it’s perfectly fine and crawlable as well.

    So, it might be an issue specific to Google Search Console as well. You may want to try resubmitting the robots.txt or reach out to the Google Webmaster Support for further assistance on this.

    Note: we just rested your website using the Google Mobile-Friendly Test tool and can confirm that your website is crawlable once again. You could test it on your own from here: https://search.google.com/test/mobile-friendly?id=nek75lINniMjFOMmNOR-OQ

    Thread Starter jvsouthwood

    (@jvsouthwood)

    Hi, it seems to be fixed, yes. Thanks for alerting me ??

    After trying everything, I come away thinking that the coincidence was that the bad gateway error caused by the plugin update (fixed within 10-20 minutes) just *happened* to coincide with a google crawl. And that subsequently google decided that the site didn’t exist for a few days. All’s well now though.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Robots.txt cannot be fetched’ is closed to new replies.