[Plugin: Bad Behavior] Blocking Google Bot

Man, also all these:

IP address: 66.249.66.187
Host name: crawl-66-249-66-187.googlebot.com
66.249.66.187 is from United States(US) in region North America

————————

IP address: 66.249.66.247
Host name: crawl-66-249-66-247.googlebot.com
66.249.66.247 is from United States(US) in region North America

————————

IP address: 66.249.71.164
Host name: crawl-66-249-71-164.googlebot.com
66.249.71.164 is from United States(US) in region North America

————————-

IP address: 66.249.66.133
Host name: crawl-66-249-66-133.googlebot.com
66.249.66.133 is from United States(US) in region North America

Tired of looking at all of them. Bad, bad behavior…

This happened to me on one of their updates. I was told I had to make some kind of adjustment because I am using Cloudflare. In the end, I removed this plugin because Google indexing is far more important than spam prevention. My traffic dropped by 50% when it started blocking thousands of index attempts by Googlebot.

I can confirm this issue. Our traffic dropped significantly, and most of our pages stopped being indexed… Damned…

I very rarely put any ones work on plugins down. But this plugin caused me HUGE issues with Google.

Sure the author responded and asked me to give them the IP’s, but the damage was already done.

Lesson learned, but was an expensive one.

We noticed some strange activity when used with cloudflare. The plugin was setup as per the instructions cloudflare provide.
Our server includes the cloudflare mod.
So, everything is configured as it should.

3 of our sites had massive volumes of errors, one site had 180’000 403 errors per day!
Cloudflare claimed that bad behavior was causing the issues. So we checked the log and it showed no google IPs or cloudflare IPs.

So cloudflare blamed our server as causing the issue. We pulled the server logs and saw no google IPs or cloudflare IPs that were being given any 403.

When we supplied stats for our server and bad behavior, cloudflare closed all our tickets and did not address the issue further, including one ticket that had received no reply from their staff or anyone else. We have had to resort to contacting their press office as their support is immensely lacking (presently 14 days with no reply, tickets closed, issues ignored, questions ignored, the list goes on and on…).

The problem is that cloudflare do not know their own system. I noticed that they made a post stating that sites with errors would be moved to IP blocks so that Google knew they were error prone and did not base it on the IP. When I asked their support staff about this, they claimed the opposite. I asked them to clarify this, but they simply ignored the question repeatedly.

I joined cloudflare with the view of testing it and then upgrading some of our sites to one of their paid options. However, after the support and errors we experienced, I am now looking at moving to an alternative.

There is not enough information here to determine the cause of the problem.

Though I got a report this morning that Google AdSense had changed their bot user agent, and so I released an immediate fix. You can download 2.2.11 today.

If you see instances where a legitimate bot is blocked, you should report them to the author immediately. This forum is not a good support channel for such issues.

Hi error,

First, I reported these to both plugin developers listed on this forum, Mark and Andrew. They both got back to me and said that they only contributed a small amount of code to this project a while ago and should not be considered developers on the project, yet they are listed as co-devs.

Second, it is NOT CloudFlare. When I upgraded to the last version of Bad Behavior, my traffic logs and Google Webmaster Tools both reported a serious drop in traffic as well as Googlebot being blocked from crawling the site. This happened sometime last month. My site was completely dropped (no re-instituted) in Google search results.

How do I know this? I turned off Cloud Flare and tested using FETCH domain through Google Webmaster Tools and the domain was still being blocked.

Then, I turned off Bad Behavior and turned on Cloud Flare and FETCH correctly got through to identify my domain and allowed me to re-submit Netmix.com, which finally showed back up in search today, thank god!

So, this is absolutely a Bad Behavior issue. The reason I know this is that when doing a FETCH, the error returned clearly mentions that Bad Behavior is blocking and questioning the authenticity of both GoogleBot and MSN Bot.

I’m not sure which IPs to whitelist for Google, MSN and Yahoo or another other service for that that matter. But, I also use CloudFlare and WordFence too, so I’m pretty well protected to a point.

I’d love to recommend Bad Behavior again. It is a sweet plugin, but killing my SEO and Google Page Rank, as well as the listing, is certainly worrisome.

Hey CouponCode, the only way you’re going to see where Bad Behavior blocks Googlebot is running a FETCH domain through Google Webmaster Tools. It will display the message from Bad Behavior to Google blocking the bot. That’s how I figured it out.