Blocking Crawling

  • I have a site with a public section and a members only section. The members only section is accessed by a login/logout plugin. That works fine. However, Google etc. still crawls the pages and posts in the members section thus defeating the purpose of a ‘protected’ area.

    Is there a way to restrict search engines from crawling the content in the members section?

Viewing 2 replies - 1 through 2 (of 2 total)
  • Michael

    (@mikeyhoward1977)

    You can manually update your robots.txt file and add entries to discourage search engines from crawling – https://www.robotstxt.org/robotstxt.html

    You also need to ensure that the pages you do not want crawled are not included in your sitemap.xml file.

    The SEO plugins that are available (free) can handle all of this for you. There are a few good ones out there. i’d suggest you take a look. They will enable you to exclude pages being crawled from the Admin UI and also dictate which pages should not be included in the sitemap. (As well as lots of other great stuff)

    Two good SEO plugins are;
    Yoast WordPress SEO and the All in One SEO Pack

    Thread Starter ritch47golf

    (@ritch47golf)

    Many thanks Mike.

    I’ll have a got at that this evening.

    Ritch

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Blocking Crawling’ is closed to new replies.

Tags