Disallow: /comments on robots.txt? What's that?

It would keep bots from scanning/indexing

domain.com/comments (whatever that is on your site)

thanks, so I will Allow it. But… what are the advantages/disadvantages of it?

I read somewhere for SEO that it’s better to Disallow it, but I don’t see the logic.

If I allow it, what happens?

It may get picked up by bots (bots crawl the text on your site looking for things to index – google, yahoo, bing, etc)

Which would mean it might get indexed and show up in google searches.

You will probably find a million different opinions on what should, and should not go in to robots.txt.

I have 1 site where I disallow a bunch of stuff, and another where I don’t block anything. Both sites get indexed fine. You can do a Google search for ‘wordpress robots.txt’ to read various tutorials/opinions and make up your own mind what is best.

Just remember, in robots.txt, you don’t allow anything.

To block something from showing up in a search, you would put Disallow: whatever in there…. but the command Allow: whatever doesn’t exist. So to allow something…. you just don’t disallow it, make sense?

Hmm… I am thinking… right now I have Disallow: /comments on the robots. But it looks like that the comments are being indexed because the post pages are indexed.

So I don’t get what does the Disallow: /comments really disallows. Comments? Or other stuff?

it would disallow a page named comments really…

because if you go to: domain.com/comments that doesn’t get you to your comments or anything….

Disallow: /comments would specifically stop the url domain.com/comments from being indexed…. if you have such a url

If that is supposed to be stopping individual comments from being indexed…. I don’t think blocking /comments does it…

Thanks. I’m starting allowing everything but the plugins.

Take care ??