robot.txt blocking but all looks fine

  • Resolved onecall

    (@onecall)


    3 weeks, 5 days ago

    Hi, I for some reason can’t get google to crawl the site I’m working on and search consol can’t fetch the sitemap even though it is there. I have tried a variety of fixes where I edit the robot.txt file but nothing seems to work. Are there any known issues with this? Or can someone help me resolve?

    Thanks
    Simon

    The page I need help with: [log in to see the link]

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Support Maybellyne

    (@maybellyne)

    Hello Simon,

    Thanks for using Yoast SEO. Here’s the directive you have in your robots.txt

    User-agent: *
    Disallow: /

    Instead, remove what you currently have and paste the following:

    # START YOAST BLOCK
    # ---------------------------
    User-agent: *
    Disallow:
    Sitemap: https://www.yoga-insure.co.uk/sitemap_index.xml
    # ---------------------------
    # END YOAST BLOCK

    Thread Starter onecall

    (@onecall)

    Hi Maybellyne,

    Thanks for coming back to me. I have pasted this text into the editor in Yoast and has no effect. When I go to https://www.yoga-insure.co.uk/robots.txt it only displays “User-agent: *” and the search console still can’t fetch the sitemap.

    Thanks
    Simon

    Plugin Support Maybellyne

    (@maybellyne)

    Hello Simon,

    Do you have a physical robots.txt file within your website files or the root of the server?

    Thread Starter onecall

    (@onecall)

    Hi Marbellyne,

    Yes there is a physical robots.txt file there. This has now been removed but still google console is unable to fetch.

    Thanks
    Simon

    • This reply was modified 3 weeks, 3 days ago by onecall.
    Plugin Support Maybellyne

    (@maybellyne)

    It must have been browser cache. I confirmed your robots.txt file is updated. See screenshot.

Viewing 5 replies - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.