mazondo

Thanks for the reply. This hit the backburner temporarily but I’ll play with it tonight!

Thanks!

I just checked google webmaster tools again and it seems like everything is working ok now. There may have been more than a few days lag between when I turned off privacy and when it showed as such in google. I have no idea why.

For anyone looking into this in the future, WordPress DOES generate it’s own robots.txt file when you have privacy on, but you do have the option of creating your own and adding it into the root directory to override the generated one.

Thanks everyone for all your help!!!! I learned a lot about robots.txt files from that link ipstenu, really appreciate it.

Really ipstenu? I didn’t realize that was set to allow all. Google webmaster tools is saying I have crawling blocked. Can you think of anything else that would be causing that?

Yes, it is available at https://brandedlocal.com/robots.txt.

You can tell Its generated by wordpress and not a physical file because of the sitemaps line. Every blog in the network receives it’s own robots file with it’s own sitemap line. That wouldn’t be the case with a static file.

Sorry, I guess I can see the issue with my first post. I meant generated robots file as in the one created on the fly by wordpress. There is no physical robots file in my root directory.

There isn’t a physical robots.txt file in the wordpress root, and it isn’t being generated by a plugin. It’s the virtual robots file generated by wordpress when your blog is set to private. Unfortunately, my blog is now set to public but is still generating the wrong robots file.

I’ve deactivated all plugins and confirmed that the file is still being created, Which leads me to believe I must have done something wrong with the multisite install somehow. I can’t create my own robots file to override it because I’m depending on the goole XML sitemaps plugin which requires you to use wordpress for generating the robots file. Any ideas?

The thing is that I don’t have a manual robots.txt file. It’s definitely being generated by wordpress and added to by the google sitemap xml for multisite plugin. I deactivated all plugins but the file is still generated.

The thing is that when I first started the multisite I had it set to private. I don’t recall ever changing this, but now it’s set to public and the robots.txt file is being generated to disallow.

My guess is that this is a www.remarpro.com problem, not a specific plugin problem now. I’ve found 3 plugins now, all with the same issue. Anyone have any ideas or another way to get the plugin?