Robots.txt Question

  • Hi, I’ve recently been working on reducing the amount of duplicate content in my blog by denying robot access to category/archive pages etc.

    My website has a robots.txt in the root with the blog sitting in a subfolder. I’ve noticed that WordPress generates a robots.txt file that I can’t seem to delete this file.

    Would it being there effect my robots.txt file in the root folder?

    Example of root robots:

    Disallow: /blog/wp-admin/
Disallow: /blog/wp-content/
Disallow: /blog/wp-includes/

    WordPress generated robots:

    User-agent: *
Disallow:

Viewing 1 replies (of 1 total)

  • I’m facing the problems with this virtual robot.txt in www.remarpro.com 2.8.6
    I have the privacy settings to Allow… and when I see the robot.txt file it says: Disallow (??) – I’m totally perplexed by that one. I swear I have hit the save settings button atleast 3 times here. Then I installed a plug in and that doesnt help either. Can anybody please give a solution to this?

    Thanks

Viewing 1 replies (of 1 total)
  • The topic ‘Robots.txt Question’ is closed to new replies.