What is the best way to handle robots.txt on Multisite?

  • I know there are plugins, and some other things that people do, but is it straight the best just to keep one robots.txt in the root folder for all the sites in the network?

    Thank you.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    ?????? Advisor and Activist

    Depends on how your site is built. Most people only need the one.

    Thread Starter jtt89

    (@jtt89)

    Thanks. I have the following cofiguration (I did not look at the names not to confuse anything):

    define( ‘SUBDOMAIN_INSTALL’, false );

    and I basically make new sites of main_domain.com, like

    main_domain.com/001
    main_domain.com/002
    main_domain.com/003 etc…

    I use domain mapping plugin on shared hosting, so I just went to “Parked Domains” in cPanel and then added this domain in Multisite configuration…

    I am gonna be starting with something like this:

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: /tag/
    Disallow: */trackback
    Disallow: */comments

    Disallow: /2006/
    Disallow: /2007/
    Disallow: /2008/
    Disallow: /2009/
    Disallow: /2010/
    Disallow: /2011/

    (Second part is just an example)But that’s probably gonna be changing too…

    Thank you.

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    ?????? Advisor and Activist

    So you’re mapping domains? Yeah, one robots is great for that ?? Full domains and subdomains are great with the one robots. If you put it in your root HTML, it’ll easily handle all sites.

    (BTW, I wouldn’t disallow the yearly archives, Google’s smart enough to know it’s not duplicate content, and it helps people find things.)

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘What is the best way to handle robots.txt on Multisite?’ is closed to new replies.