robots.txt how to exclude a Page being indexed and other robots.txt questions

  • Hi,

    I looked through the forums regarding robots.txt and I see a bunch of them posted and have some questions regarding why, how, and what.

    1. If I have a page with a slug of “info” and I didn’t want it to be indexed how would I do that, like this?:
    Disallow: /info

    2. If I Disallow: /wp-content/ wont all my “pages” be not indexed? Or does that just tell it not to index any of the actual files like index.php in those directories?

    3. If I have a custom homepage called home.php won’t Disallow: /wp-content/ make my homepage not be indexed? If so how do I make it so everything except the homepage is not indexed?

    Thanks for your help.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Moderator Samuel Wood (Otto)

    (@otto42)

    www.remarpro.com Admin

    1. Yes.

    2. No. Remember that robots only see your URLs, not the actual location of things. Disallowing /wp-content/ will prevent them from indexing anything with an actual wp-content URL. This includes bits of your theme (CSS file) and theme images. It will also include any images you post using the normal image uploader (since those go to wp-content/uploads/ by default).

    3. No. The URL is important, not the location of the document. Disallow: /wp-content/ will make anything with a URL of https://example.com/wp-content/whatever not get indexed. The homepage does not have a /wp-content/ URL.

    Thread Starter jrm213

    (@jrm213)

    Thanks Otto42, that information is very helpful!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt how to exclude a Page being indexed and other robots.txt questions’ is closed to new replies.