robots.txt set to disallow, can't change

  • I’m working with wordpress multisite, and have verified that the primary blog is set to allow crawlers in the privacy settings. Unfortunately, the generated robots.txt file is still showing disallow for all the sites. Any ideas on why this would be the case and how to fix it?

Viewing 4 replies - 31 through 34 (of 34 total)
  • angiewheeler

    (@angiewheeler)

    Hey Rick –
    Did you ever get a resolution for this problem? I have a client having the same problem.

    rickfish182

    (@rickfish182)

    Hi Angie – I did. Kind of.

    1. Based on the support moderator’s suggestion, I disabled and deleted the Google XML sitemap plug in.

    2. I then created a manual robots.txt file and set it to allow everything.

    3. Then I waited a few days for Google to re-index the site. It it took 3 or 4 days, but it happened. All the pages and posts are now indexed.

    Next steps:

    Download and re-install the Google XML sitemap plugin.

    **I have not had time to do this yet**

    Hope this helps ! ??

    angiewheeler

    (@angiewheeler)

    This does help! Thank you so much for the response Rick ??

    Moderator Ipstenu (Mika Epstein)

    (@ipstenu)

    ?????? Advisor and Activist

    rickfish182 – I’m glad the manual robots punted Google over. It’s so annoying when it gets STUCK.

    FWIW, I did the same thing and have since turned on Google XML sitemaps. It’s working as expected.

Viewing 4 replies - 31 through 34 (of 34 total)
  • The topic ‘robots.txt set to disallow, can't change’ is closed to new replies.