robots file is invalid

  • Resolved macpheek

    (@macpheek)


    1 year, 9 months ago

    Hi, I’ve had an issue with a few sites lately that are using Yoast. When running a scan through Google Page Speed insights, under the SEO section it says ‘robots file is invalid’.

    The issue seems to go away when I go into the Yoast Settings –> Tools –> File editor, and create a robots file in here.

    I just wondered if you know any further information into why Google would flag this up? I read that it’s not necessary to have a robots file unless you need to set up Allow / Disallow rules, so I’m confused why it would flag this up. Hope you can help shed some light on this

    Many thanks

Viewing 6 replies - 1 through 6 (of 6 total)
  • Thread Starter macpheek

    (@macpheek)

    To add to this, here’s a screenshot of the error in Page Speed Insights

    https://snipboard.io/qyYspT.jpg

    Plugin Support Maybellyne

    (@maybellyne)

    Hello @macpheek

    Thanks for reaching out regarding your robots.txt file. Search engines discover and index the web by crawling pages. But before a search engine visits any page on a domain it hasn’t encountered before, it will open that domain’s robots.txt file. That lets them know which URLs on that site they’re allowed to visit (and which ones they’re not).

    When a site does not have a user-defined?robots.txt?file (at the point of installation/activation), Yoast SEO generates a default output. That’s what you have done in WordPress > Yoast SEO > Tools > File editor.

    Do you mean that Page Speed Insights flagged an issue after or before creating the file in the Yoast SEO plugin?

    • This reply was modified 1 year, 9 months ago by Maybellyne.
    Thread Starter macpheek

    (@macpheek)

    Hi, thanks for your reply and info. Yes, Page Speed Insights flagged an issue before a robots file was created via Yoast. I’ve tried loading the robots url to check if anything exists, and I can see one exists at https://adviceacademy.co.uk/robots.txt, however I haven’t generated this via yoasts tool, and it’s flagging up the issue in Page Speed Insights.. I haven’t clicked ‘Generate Robots’ in yoast yet, as I want to leave it as it whilst this ticket is open. It looks like yoast has generated one by default itself, and it’s outputting an error on PSI ?

    Plugin Support Maybellyne

    (@maybellyne)

    Thanks for following up. This robots.txt file is valid. Next, test in the?robots.txt Tester?and let us know how it goes.

    Plugin Support Maybellyne

    (@maybellyne)

    This thread was marked resolved due to a lack of activity, but you’re always welcome to re-open the topic. Please read this post before opening a new request.

    Thread Starter macpheek

    (@macpheek)

    Hi, the robots txt tester comes up with no issues. Is this a false negative in google search console then?

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘robots file is invalid’ is closed to new replies.