Robots.txt

Define “easy”. If you mean “create it in a text editor and put it on your site”, then yeah, that’s pretty easy. ??

Notepad?

Notetab is much better.

Put a robots.txt file in your WP root.
Something like:
User-agent: *
Disallow: /aboutme/
Disallow: /stats/

Do I need to worry about the admin area?

Up to you. I don’t have my whole wp-admin folder indexed.

So would it be like so?

User-agent: *
Disallow: /aboutme/
Disallow: /stats/
Disallow: /wp-admin/

Yes exactly (though “stats” and “aboutme” were examples)
If you want to do a specific file:
User-agent: *
Disallow: /aboutme/
Disallow: /stats/
Disallow: /wp-admin/
Disallow: /wp-includes/index.php

is there something in the codex about this for indepth reading?

Google has lots of links for robots.txt. It certainly isn’t a WordPress only thing — it applies to all web sites. ??

https://www.robotstxt.org/

Thanks for the link. I’ll check it out.

Hi all!

Sorry for the late bump of this… but I have a domain in which I have a Joomla installation at / and WordPress at /blog. I only have 1 robots.txt file in / and here is what is currently in it:

*****************
User-agent: *
Allow: /sitemap.xml
Allow: /blog/sitemap.xml
Allow: /index.php
Allow: /blog/index.php
Disallow: /*.php$
Disallow: /blog/*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php-dist$
Disallow: /*.ico$
Disallow: /blog/*.js$
Disallow: /blog/*.inc$
Disallow: /blog/*.css$
Disallow: /blog/*.wmv$
Disallow: /blog/*.cgi$
Disallow: /blog/*.xhtml$
Disallow: /blog/*.php-dist$
Disallow: /blog/*.ico$
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /editor/
Disallow: /help/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /mambots/
Disallow: /media/
Disallow: /modules/
Disallow: /templates/
Disallow: /installation/
Disallow: /cgi-bin/
Disallow: /blog/wp-content/
Disallow: /blog/wp-admin/
Disallow: /blog/wp-includes/

Please take a look at the items highlighted in bold (particularly the first few)–and let me know if this is allowed (and especially if it will work) in the robots.txt file?

Also, should I have a separate robots.txt file for the /blog directory?

Many thanks!

David

1. you can only have ONE robots.txt and it should me at the root of your domain. Robots will not look anywhere else for it.

2. You dont need the “allows”

As far as this:

Allow: /blog/index.php
Disallow: /*.php$ <– that blocks index.php, archives.php, etc.. WHY are you doing that? Joomla and WordPress are both PHP based apps.