Google can't crawl my site because it is unable to access site's robot.txt

  • Resolved sanaqureshi

    (@sanaqureshi)


    8 years, 6 months ago

    hi guys, i am having this issue:
    “Google couldn’t crawl your site because we were unable to access your site’s robots.txt file.”
    My site: TechGeekZone

    I have already tried all the suggestions provided in the related problems but nothing seems to be working. I am using a plugin that has my site in maintenance mode. Could that be the issue?

    I have made a robots.txt file and pushed it in the root directory but problem remains.
    Any kind of help is highly appreciated.
    Thank you!

Viewing 8 replies - 1 through 8 (of 8 total)

  • Please share the contents of your robots.txt file.

    Thread Starter sanaqureshi

    (@sanaqureshi)

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /xmlrpc.com
    Disallow: /wp-login.php
    Disallow: /comments/feed/
    Allow: /wp-admin/admin-ajax.php

    Sitemap: https://techgeekzone.com/sitemap_index.xml

    You are disallowing the bots to crawl… You have basically everything set to block the bots.

    Change your robots.txt file to look like this

    User-agent: *
Disallow:
    Thread Starter sanaqureshi

    (@sanaqureshi)

    But won’t that allow rogue bots to access everything they shouldn’t, as well?

    that will allow all bots to access the site.

    If you want only google bots to access it use this code. 

    User-agent: googlebot
Disallow: 

User-agent: *
Disallow: /
    Thread Starter sanaqureshi

    (@sanaqureshi)

    Thank you for your help!

    No problem at all. That is what the community is here for! Come back any time with questions or problems that you need help with!

    unable to crawl your website

    I did several reports for my clients and many of them saying this.

    On-Site SEO Report
    This section analyzes the On-Site SEO factors that affect your ability to rank high in search engines.
    On-Site SEO factors are things on your website – either visible on the page (which you can see) or in the code of your website (which you can’t see, but search engines can). It’s easier to improve SEO factors on your own site because you have control over it.
    Notice: due to an issue with your website structure we were only able to crawl the home page of your website. This issue may affect search engine crawlers and could be damaging to your SEO. We suggest that you look into this issue ASAP.

    Any help please???

Viewing 8 replies - 1 through 8 (of 8 total)
  • The topic ‘Google can't crawl my site because it is unable to access site's robot.txt’ is closed to new replies.