Banned google bot

Hi there, thanks for your feedback.

Firstly can you go to the Pareto Security control panel and look through the log file for any mention of Googlebot. It will look something like this:

[Banned] Crawler/USER-AGENT: 'Something Something Googlebot....'

And in the meantime while I look into this can you verify that you are using Pareto Security in the default Standard Mode rather than Advanced Mode.

Standard Mode does not prevent crawlers from accessing your website.

If you were using Advanced Mode, once you switch back to Standard Mode, go to plugins and deactivate then activate it again. This will reset the IPs in your banned list.

I have tested Pareto Security now against over 250 versions of Googlebot (see https://developers.whatismybrowser.com/useragents/explore/software_name/googlebot/ for the full list) and Pareto Security has not triggered a ban in advanced mode. So definitely will need your help to be able to identify your specific issues.

Hi!

Thanks for your replay. Pareto Security works in Advanced mode + – Hard Ban Mode at my site. So far I have not noticed “Googlebot” in the title Crawler / USER-AGENT, I’ll still look and write here. But maybe Google can use Crawler / USER-AGENT without “Googlebot” in the title? In any case, it would be convenient for your plug-in to see which useful search engine bots were allowed and skipped. Thanks

In my .htaccess file already [224] unique IP addresses are banned by Pareto Security. Are these all harmful bots? Could not ordinary visitors be banned by mistake? Or does your plugin accurately identify violators? Thanks

Google always at least uses the term “Google” in their user-agent.

Just for your own info, go to https://www.dnsqueries.com/en/googlebot_simulator.php and test two websites, your website, and mine (https://hokioisecurity.com). We both run Pareto Security in Advanced/Hard Core mode. You will see that your site does not return any data when simulating Googlebot.

My best guess then is another plugin is causing this. Can you list the plugins you are using for me thanks, I may be able to help you there.

Hi!
At my site installed next plugins:

Google AdSense

Google Analytics Dashboard for WP (GADWP)

Google XML Sitemaps

Markup (JSON-LD) structured in schema.org

NextScripts: Social Networks Auto-Poster

Pareto Security

Recent Pages

Redirection

Server IP & Memory Usage Display

Simple Custom CSS

Subpage Listing

UpToLike Social Share Buttons

VKontakte API – crossposting, comments, social buttons, login

WP Super Cache

Thanks.

I have installed all those plugins on two of my websites on two different web hosting services and in each case Googlebot is bot prevented from accessing the site.

Can you go to Settings / Reading

Then confirm that there isn’t a tick in the box “Discourage search engines from indexing this site”. This is the only other possibility.

I just took a closer look at your websites server. It appears your website is running the mail.ru page counter. This may be the source of the problem. Your site is most certainly talking to Google Analytics, but Google cannot connect back.

Can you disable Pareto Security, then let me know, so I can run some tests on your website without it enabled. Once I am finished I will message here. Won’t take me long.

• This reply was modified 6 years, 3 months ago by te_taipo.

In fact it appears that a service on the webserver is returning the empty page to Googlebot before any PHP pages are loaded. So it may also be something your web host is running. Check with your host as well please.

Hello! The settings of my site allow indexing by search engines. I suspect that the problem is only with google, the main traffic to my site goes through another search engine (yandex). I temporarily, until tomorrow deactivated your plugin. Thanks.

Ok, so now that you have disabled Pareto Security, this means all of the banned IPs will be removed from your HTACCESS, including any IP addresses that belong to Google.

If you go to https://www.dnsqueries.com/en/googlebot_simulator.php and enter your website in there, you will be able to see that Googlebot still cannot access your website. I tested this and it is clearly *not* Pareto Security that is blocking Googlebot.

Your next step should be to contact your web host and ask them if they are blocking the IP range that Google uses.

Thanks for the help! This is a small problem for me, Google plays a small role in the attendance of my site.

Ok, no problem.