Problem with Robots.txt

First, here is the best place for Wordfence Support:
https://www.wordfence.com/forums/forum/wordfence-support-questions/

I also use Bulletproof and Wordfence, and neither plugin has any effect upon robots.txt. So, first be sure you do not have “Discourage search engines from indexing this site” checked at Dashboard > Settings > Reading, then also be sure to uncheck “Add sitemap URL to the virtual robots.txt file” at Dashboard > Settings > XML-Sitemap if you are using the Google XML Sitemaps plugin. As best I can figure out, having that checked causes Google to be distracted by the default WordPress “virtual robots.txt” output that is somehow affected (over-ridden) by the plugin that can negatively affect that virtual file (like some kind of circular reference) when “Add sitemap URL to the virtual robots.txt file” is checked and thereby actually triggering it in conflict with itself (the plugin).

Edit PS: It is also best to leave “Immediately block fake Google crawlers” un-checked inside Wordfence.

Thanks leejoseph for your words and help, but I think I’ve discovered what happened, and will post it here for everyone to see:

Yes, I had checked every point you’ve said in your post before creating this thread, but it kept prompting the robots.txt issue.

So after some more in-depht research, I found this:

It seems that, for WordPress 3.61, some changings have been added to the functions.php file in wp-includes, NOT in wp-content.

In this functions.php file, there’s a function called “do_robots()”, which builds a robots.txt file if there is none. So it really doesn’t exist, it is created virtually with the settings made by this function in functions.php. These are the settings (remember I didn’t touch anything, this was a fresh new WordPress 3.61 install from zero):

header( ‘Content-Type: text/plain; charset=utf-8’ );

do_action( ‘do_robotstxt’ );

$output = “User-agent: *\n”;
$public = get_option( ‘blog_public’ );
if ( ‘0’ == $public ) {
$output .= “Disallow: /\n”;
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url[‘path’] ) ) ? $site_url[‘path’] : ”;
$output .= “Disallow: $path/wp-admin/\n”;
$output .= “Disallow: $path/wp-includes/\n”;
}

echo apply_filters(‘robots_txt’, $output, $public);
}

The bold line $output .= “Disallow: /\n”; is the important one for this issue.

I simply have erased the /, so now it remains like this:

$output .= “Disallow: \n”;

Now Google can send its crawlers with no problem at all. I hope this shall be useful for other people having this issue. Problem solved.

Cool beans!

What would have been best coded to meet most or all WordPress installs? I would suspect that many have the same issues but do not have a Google account to see this incompatibility.

“Disallow: /\n”;
Compared to: “Disallow: \n”;

I had the same problem, but the suggestion above didn’t work. I ended up copying the functions.php from another site and dropping it in. Don’t know why this would make a difference, but it did.

ive the same issue right now with only one of my site.

2 people have dug up an very old thread and added vague comments with no clear indication what they are talking about. You are not likely to get any help unless you start your own thread and clearly describe your problem.

If you require assistance then, as per the Forum Welcome, please post your own topic instead of tagging onto someone else’s topic.

This 7 month old resolved topic is now closed.