Bad Bot Alert for Google

Doing a forward/reverse lookup, I see that IP corresponds to 68.27.195.35.bc.googleusercontent.com, which does not seem to be listed in the official documentation for Google user agents:

https://support.google.com/webmasters/answer/1061943?hl=en

What is the reported user agent in the Blackhole log?

• This reply was modified 7 years ago by Jeff Starr. Reason: clarity

Jeff, thanks for help. See complete message below. I’m a business guy with enough web knowledge to be dangerous. Just wanted to make sure.

Request URI: /?blackhole=2b86ac1407
IP Address: 35.195.27.68
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:52.0) Gecko/20100101 Firefox/52.0

Whois Lookup:

ARIN WHOIS data and services are subject to the Terms of Use
available at: https://www.arin.net/whois_tou.html
If you see inaccuracies in the results, please report at
https://www.arin.net/public/whoisinaccuracy/index.xhtml

The following results may also be obtained via:
https://whois.arin.net/rest/nets;q=35.195.27.68?showDetails=true&showARIN=false&showNonArinTopLevelNet=false&ext=netref2

NetRange: 35.192.0.0 – 35.207.255.255
CIDR: 35.192.0.0/12
NetName: GOOGL-2
NetHandle: NET-35-192-0-0-1
Parent: NET35 (NET-35-0-0-0-0)
NetType: Direct Allocation
OriginAS:
Organization: Google LLC (GOOGL-2)
RegDate: 2017-03-21
Updated: 2017-03-21
Ref: https://whois.arin.net/rest/net/NET-35-192-0-0-1
OrgName: Google LLC
OrgId: GOOGL-2
Address: 1600 Amphitheatre Parkway
City: Mountain View
StateProv: CA
PostalCode: 94043
Country: US
RegDate: 2006-09-29
Updated: 2017-10-16
Comment: *** The IP addresses under this Org-ID are in use by Google Cloud customers ***
Comment:
Comment: Direct all copyright and legal complaints to
Comment: https://support.google.com/legal/go/report
Comment:
Comment: Direct all spam and abuse complaints to
Comment: https://support.google.com/code/go/gce_abuse_report
Comment:
Comment: For fastest response, use the relevant forms above.
Comment:
Comment: Complaints can also be sent to the GC Abuse desk
Comment: ([email protected])
Comment: but may have longer turnaround times.
Comment:
Comment: Complaints sent to any other POC will be ignored.
Ref: https://whois.arin.net/rest/org/GOOGL-2

OrgTechHandle: ZG39-ARIN
OrgTechName: Google LLC
OrgTechPhone: +1-650-253-0000
OrgTechEmail: [email protected]
OrgTechRef: https://whois.arin.net/rest/poc/ZG39-ARIN

OrgNOCHandle: GCABU-ARIN
OrgNOCName: GC Abuse
OrgNOCPhone: +1-650-253-0000
OrgNOCEmail: [email protected]
OrgNOCRef: https://whois.arin.net/rest/poc/GCABU-ARIN

OrgAbuseHandle: GCABU-ARIN
OrgAbuseName: GC Abuse
OrgAbusePhone: +1-650-253-0000
OrgAbuseEmail: [email protected]
OrgAbuseRef: https://whois.arin.net/rest/poc/GCABU-ARIN

ARIN WHOIS data and services are subject to the Terms of Use
available at: https://www.arin.net/whois_tou.html
If you see inaccuracies in the results, please report at
https://www.arin.net/public/whoisinaccuracy/index.xhtml

More research reveals that googleusercontent.com is used for Google services like translations and cached web pages. The problem is twofold: 1) they are not tagging the user agent with “google”, so it looks like a typical user browser (which it may be). And 2), Google uses a gaggle of different IP addresses. So that makes it very difficult to whitelist googleusercontent.com; there is no discernable user agent, and way too many IPs that probably are constantly changing.

The question is, why is anything from googleusercontent.com (e.g., cached pages or translations) hitting the hidden Blackhole link? I assume that you have added the required Disallow rule to your site’s robots.txt file, correct? If so, then googleusercontent.com is disobeying your rules and should be blocked, IMO.

Follow-up, are you able to check your site’s/server’s access log and get the “referrer” of the request? If so, I could possibly add an option to whitelist by referrer, which then would work to always allow requests coming from Google’s googleusercontent.com service. But to do so, we need to know exactly what the referrer is..

I hope you got this resolved one way or another. Gonna go ahead and mark this thread as resolved to keep things organized. Feel free to post again if any further issues, thank you.