Googlebot can't access my website

  • * My website is working fine when browsing but googlebot can’t access my website and robots.txt file.

    –> i tried to fetch my webpages by “fetch as google” from my Google Webmaster Tools but it says “Temporarily unreachable”

    –> Robots.txt tester says “robots.txt fetch failed”.

    –> Also google can’t access the sitemap of my website. It says following errors:-

    “URL timeout: robots.txt timeout”
    “We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.”

    ** I have checked the installed plugins and everything from my side and robots.txt file also updated to new one by me today.

    ** I have contacted my hosting provided but they told that everything is OK also there is no issue in firewall.

    ** They are telling firewall is not blocking googlebot.

    Please help me! It’s serious!

Viewing 7 replies - 1 through 7 (of 7 total)

  • Hi Abrazz,

    Is this a new issue? I.e. was your robots.txt file and sitemap working and then suddenly it wasn’t? Or is this the first time your trying to use Google Webmaster Tools.

    Also, have you moved hosting or servers recently by any change?

    Finally, what’s the URL of your site?

    Thread Starter abrazz

    (@abrazz)

    This is not first time error.

    I checked sitemap and robots.txt is working fine.

    Also i have not moved my hosting service. I don’t know what happens to me.

    ** It’ll be helpful if you give me your IM or contact details

    I have several sites showing the same problem, but it spans multiple hosting accounts and even multiple hosting providers, including Host Gator.

    I’ve looked at the HTTP response with the Googlebot user agent, and pages are fetched with no problem, but “fetch as googlebot” claims “temporarily unreachable”.

    I’ve disabled my security plugin completely, including any filters in .htaccess, but nothing changed.

    I have have a site that’s not based on WordPress.

    Is your site on CloudFlare? I’ve just disconnected a could of sites from CloudFlare and googlebot can fetch pages, even after I activate CF again.

    Thread Starter abrazz

    (@abrazz)

    Thanks for your help. I have solved my problem after disabling cloudflare.

    Again many more thanks for you

    CloudFlare advises they have debugged the issue and applied measures to prevent it from recurrning. My sites all work now, so it’s probably safe for you to go back too.

    Is your site on CloudFlare? I’ve just disconnected a could of sites from CloudFlare and googlebot can fetch pages, even after I activate CF again

    galbaras, You Rock Man !!
    Many thanks ??

    additional tips for Wordfence Security plugin users :
    in case you were using Wordfence Security plugin, you would probably notice after a while that it is blocking Google crawlers..
    Go to /wp-admin/admin.php?page=WordfenceBlockedIPs and make sure you unblock IPs there blocked as fake Google bots, and checkout options page ( /wp-admin/admin.php?page=WordfenceSecOpt ) and un-thick this option : automatically block fake Google bot ...
    and there you go, submit a sitemap, update robots.txt and fetch as Google bot. ??

Viewing 7 replies - 1 through 7 (of 7 total)
  • The topic ‘Googlebot can't access my website’ is closed to new replies.

Tags