problems with googlebot not being able to crawl my site

  • I am at a loss here. trying to figure out why google keeps sending emails to me that they cannot crawl site? I thought the issue has been resolved yesterday but noticed today that google tried to crawl site yesterday and the crawl rate is only 40%.

    I corrected my robots.txt file ( I think?) and have deactivated most plugins.. My hosting co (godaddy) said that all looks good on their end. I can’t seem to figure out what is going on? Could someone help me with this?

    Thanks so much!

    url is foryourprivilegedpets.com

Viewing 5 replies - 1 through 5 (of 5 total)

  • I’d remove your robots.txt and install. wordpress seo plugin by yoast.

    This will generate your sitemap and also edit your robots.txt.
    you can then submt the sitemap url to google webmastertools.

    Typical robots.txt example for a WordPress site:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Thread Starter fypp

    (@fypp)

    Thanks Mike! appreciate your help.. I am going to do that. Should I delete my robots.txt file ( godaddy hosted website) first? or should I install the Yoast Seo plugin and it’ll overwrite it?

    Hmm thats an interesting question. I suggest you leave it for now and see wat happens when you install the plugin. (I have no experience with godaddy hosting)

    Thread Starter fypp

    (@fypp)

    ok, that is what I will do.. I have used his plugin before but removed it because I did not understand it.. I guess I need to take the time to understand since many people suggest it..

    Thanks again!!!

    Your welcome!

    here’s a link to the plugins’ guide: https://yoast.com/articles/wordpress-seo/

    If it fixes your problems, please return to set the status of this topic to resolved.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘problems with googlebot not being able to crawl my site’ is closed to new replies.