Robot.txt Blocking Search Engines

  • David

    (@fenceutah)


    12 years ago

    Okay, I’m having a bit of trouble launching a site of mine. Last Friday, I turned off the setting that discourages search engines from index my site. I cleared my cache. I also reinstalled SEO by Yoast. I also submitted my sitemap to Google Webmaster Tools. I know that things don’t happen over night, but it’s been nearly a week and I still get this error on Google Webmaster Tools. “URL restricted by robots.txt”

    I’ve looked everywhere on the site and in the source code and cannot figure out why robot.txt are still blocking my site.

    Does anyone have any suggestions on where I am missing information?

    https://www.fenceanddecksupply.com

Viewing 6 replies - 1 through 6 (of 6 total)

  • Hello FenceUtah,

    I’ve got curious and looked at your site. The main issue is that Google is stating this – so you will need to take a close link at what they’re stating in the search results. There’s an option there to click (at the end of the search result where it says:

    https://www.fenceanddecksupply.com/
    A description for this result is not available because of this site’s robots.txt – learn more.

    If you click on the ROBOTS.TXT link, you’ll see what’s probably causing the issue.

    Try going into the ROBOTS.TXT file and remove what Google is showing:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Additionally, if you haven’t already created an account with Google’s Webmaster tools, please use them. They have a robots.txt checker and will probably give you more succinct information.

    [ Signature moderated. ]

    Thread Starter David

    (@fenceutah)

    Okay, So I’ve edited the robots.txt file. I’ve resubmitted things to Webmaster Tools and cleared the cache.

    I’m still getting the error that it’s being blocked.

    Is it just going to take some time?

    Krishna

    (@1nexus)

    Currently your robots.txt contains:

    User-agent: *
/

    An entry that applies to all bots looks like this:

    User-agent: *

    So, it is advisable to use only the above entry. Review:
    https://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

    Thread Starter David

    (@fenceutah)

    Krishna,

    So, if I understand what you’re saying, by not stating

    User-agent: *
/

    that I’m blocking all the robots?

    I’ve changed it to 

    User-agent: *
Disallow:

    based on the recommendation of https://tools.seobook.com/robots-txt/.

    Hello FenceUtah,

    Checkout https://www.robotstxt.org/robotstxt.html and you’ll see all of the options that you can use to block or not block bots.

    Krishna is right though – you’re blocking everything with your previous entry.

    Arnel C.

    Thread Starter David

    (@fenceutah)

    Thanks! I think it’s fixed now… we’ll see though.

Viewing 6 replies - 1 through 6 (of 6 total)
  • The topic ‘Robot.txt Blocking Search Engines’ is closed to new replies.