robots.txt

  • I’m having a problem with my robots.txt

    Well that ain’t new of course, but the stupid thing is that I did everything right. Well at least I think I did. My robots.txt file allows everything. It says:

    User-agent: *
    Disallow:

    That should do it right? Wel it doesn’t. I changed it over a month ago, but still my site isn’t crawled and google says:

    A description for this result is not available because of this site’s robots.txt – learn more

    HELP!

Viewing 2 replies - 1 through 2 (of 2 total)

  • I don’t think having a literal robots.txt file is going to work, at least not in the traditional sense.

    If I’m remembering correctly, WordPress generates a “virtual” robots.txt based on the privacy settings in the dashboard.

    The exact location will depend upon how current your WordPress version is, but in 3.5.1, it should be:

    Settings -> Reading -> “Discourage search engines from indexing this site” (If you see a checkmark there, you’ll need to remove it.)

    If you need more fine control over the generated robots.txt file, there are plugins that claim to give you that.

    i have same problem ! everything but it not success! help me!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt’ is closed to new replies.