[Plugin: WordPress SEO by Yoast] Robots

Hi – I have the same question: when sitemaps are activated in WordPress SEO plugin, a path to the sitemap is not added to robots.txt. Do we need to add it manually? If so, is there any recommended practice for doing so?

Thanks!

Never mind – I found the answer in your post here:

https://yoast.com/xml-sitemap-in-the-wordpress-seo-plugin/

shnozolla, the reason it doesn’t appear in robots.txt is that it’s apparently not necessary.

Thanks!

Hi Miriam.

I think we were asking different questions:

I was asking why no CONTENT has been written in robots.txt.
How do the search engines know which links to follow and which ones not to follow?

Thanks.

shnozolla, I believe that the plugin depends on robots meta info that appears at the top of every page to tell the search engines which content to index and follow. The reason, if I understand correctly, is that robots.txt files full of links is great fodder for spammers and other shady types to find content you don’t want them to find.

Miriam

OK, that makes sense.

Part of my original question was: “…and I have not seen any robots meta tags added to any of the pages.”

Any ideas?

Thank you Miriam.

shnozolla, are you sure you’re not seeing any robots meta tags? Can you share a link? What version of WP are you running?

If they are not showing up, could be a number of things (I’m partially guessing here):

1. Incompatibility with the version of WP
2. Conflict with some other plugin(s)
3. Something in your theme that doesn’t allow the plugin to do its stuff (I’m not sure if this is possible)

Miriam

Here is the link.
Version 3.1.1

Pretty!

I see – no robots info. I’m guessing again, but I think it only appears when you need to tell the bots something, i.e. to follow but not index, or vice versa. If the command is follow,index, then it doesn’t appear.

To see if that’s the case, try no-indexing a page or post as a test. Then see if the command shows up.

Maybe Joost can confirm at some point…

Thank you Miriam for your tips (and for the compliment ?? )

You are right…the robots tags only appear on the pages that are not meant to be followed.

I just checked the source code for

https://www.strawberryswan.com / wp-login

and the no follow robots appear there!

I remember reading somewhere that having a robots.txt file increases your search engine rankings…I guess that has changed, as you wrote previously.

Thank you!