Robots.txt? Why is it not included by default?

  • Hello Yall,

    Anyone know why there is no robots.txt file included in WP? Arguably there are a few directories any admin wouldn’t want showing up for security and SEO reasons. A few examples are:
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /rss/
    Disallow: /*?*

    I have found that various engines will index places such as wp-* that one might not want indexed. =(

    Any idea on why there is no default robots.txt?

Viewing 5 replies - 1 through 5 (of 5 total)

  • A robots.txt goes in the root directory, while WordPress may be installed in a subdirectory. Also, a site may already have a robots.txt with other non-WordPress things in it.

    WordPress will create a basic robots.txt if it’s installed in the root directory, there is no existing robots.txt, and your privacy settings are set to private.

    BTW, I’d recommend not blocking your feed if you want to be indexed quickly by Google. Google is smart enough now not to consider a feed to be duplicate content, and if your feed is being indexed instead of your regular posts/pages, then you have something else wrong with your site.

    Thread Starter thedevnull

    (@thedevnull)

    Hello Iridiax,

    Assumptions are a big problem here. We cannot assume where one is going to install WP or how it will be configured. Not having a default WP robots.txt file allow many files that should NOT be indexed to show up in most engines.

    FYI,that was just an example and I don’t have that exact config. =P The “privacy” feature you are speaking doesn’t address the issue that I am attempting to illuminate. A default WP robots.txt would give a user a general guide on what to NOT allow indexing and now no such info exists.

    Is robots.txt file necessary? While setting up XML sitemap generator plugin, I had noticed that a virtual robots.txt is generated by WordPress & a real robots.txt file should NOT exist in the blog directory. I also felt that after installing real robots.txt file, my blog started performing poorly in google search.

    Gouri: The sitemap plugin will be using the virtual robots.txt to specify the sitemap location, rather than using it to restrict what’s indexable, so in some cases a real one is necessary.

    Thanks, Kawauso.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Robots.txt? Why is it not included by default?’ is closed to new replies.