Robots, index and following

  • Hello. I’m using the WordPress SEO plugin by Yoast and am just wondering in general, which posts/pages should I have set to the following for Meta Robots Index and Meta Robots Follow:

    Index, Follow:
    Index, No Follow:
    No Index, Follow:
    No Index, No Follow:

    And then Part 2 of the question is should I change anything in the Meta Robots Advanced for any posts/pages? I’ve never altered this setting from None for any post/page.

    Can someone please give me the 101 and the basic guidelines? Obviously, this is for both SEO and user experience. Many thanks.

Viewing 5 replies - 1 through 5 (of 5 total)

  • Basically, the meta robots directive is used to specify which pages you want a search engine to index and pass link value through.

    So, the directive <meta robots=”index, follow”/> on a page tells a crawler that you want that page indexed in search results, and you want the links on that page to pass the link value.

    “index, no follow” means “index the page, but don’t pass link value from that page”
    “no index, follow” means “don’t index the page, but pass the link value from that page”.
    “no index, no follow” means “don’t index, don’t pass link value”

    This directive is mainly an SEO directive and will not affect user experience at all. Pages that are “noindex, nofollow” can still be viewed by users, as it is simply a meta tag in the source.

    As to which pages to “index, follow”/”no index, follow”, etc. is up to your discretion, but a good rule of thumb is to use it in combination with the link canonical tag to minimize duplicate content pages. For example, wordpress “tag” pages that duplicate the post content from other aggregate pages, are typically good pages to “no index, follow”, or e-commerce pages that create duplicate urls from query parameters.

    If you haven’t already, I’d recommend signing your site up for Google Webmaster Tools, and take a look at the “HTML improvements” section. This will give you a good snapshot of duplicating pages you might want to remove from search results.

    Thread Starter michaelborger

    (@michaelborger)

    Thanks so much for the detailed explanation. That clears up a lot.
    Can you please explain in a bit more detail what you mean by combining with canonical tags?

    The link canonical tag is an indicator to search engines as to which url is the “preferred version” of a page. I couldn’t explain it better than Google already has here: https://support.google.com/webmasters/bin/answer.py?hl=en&answer=139394

    Rai

    (@raiacuarioprocom)

    Hi there SpaceDog, I just install the plugin but I have one dude about robots.txt. It is create it automagically by the plugin or I need to create it and then the plugin will changed for me?..

    Thanks.
    Rai.

    @rai@acuariopro.com I am discussing the robots meta tag directive, not the robots.txt file. There is a difference between the two.

    See https://www.robotstxt.org/

    And yes, the robots meta directive is created with the Yoast plugin and there are many options to adjust them.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Robots, index and following’ is closed to new replies.