sitename.com/robots.txt error 404 – not found

  • Hello,

    whenever I type mysitename.com/robots.txt in my browser I get error 404 – not found. It’s a new, clean installation of wordpress 2.8 with the following plugins active: contact form 7, customize(for freshy 2.0 theme), google xml sitemaps, wordpress.com stats and wp-mythumshot
    site is not yet properly indexed by google although it is only a few days old.
    I tried installing PC Robots.txt plugin but still get error 404 when trying to reach sitename.com/robots.txt
    why is this and what should I do? should I create robots.txt manually and add all content including sitemaps.xml to it? (and disable “Add sitemap URL to the virtual robots.txt file.” in google sitemaps xml plugin)
    My site is not big, it’s only like 5 or so pages in size.

Viewing 3 replies - 1 through 3 (of 3 total)

  • should I create robots.txt manually and add all content including sitemaps.xml to it? (and disable “Add sitemap URL to the virtual robots.txt file.” in google sitemaps xml plugin)

    That’s what I would do but then I’m used to creating a robots.txt file anyway.

    Thread Starter julius7

    (@julius7)

    But why it’s not created automatically(or virtual version of it)? And if I would to create it manually, what examples I can look at to learn how to create the best robots.txt file for search engines?

    A robots.txt file can cover an entire site – not just the directory containing WP.

    https://www.robotstxt.org/orig.html is the definitive guide. And there’s a validator at https://tool.motoricerca.info/robots-checker.phtml.

    It’s basically just a plain text file. I normally use a boilerplate file along the lines of:

    # robots.txt for https://your_domain/

Sitemap: https://your_domain/sitemap.xml.gz

User-agent: *
Disallow: /css/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /temp/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /themes/
Disallow: /plugins/
Disallow: /upgrade/
Disallow: /uploads/
Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘sitename.com/robots.txt error 404 – not found’ is closed to new replies.