Google bots blocked by iThemes ??

  • Hello,

    This plugin does a great job si far. It already allowed me to block IPs attempting to access my site.
    But today I’ve received a “blocking notification” because of “to much attempts to reach a file that doesn’t exist” (to reach ONE file…?).
    The problem is that it seems to be/they are google bots…
    (“Hostname” : 59.84.154.xxx.bc.googleusercontent.com ; “Organization” : Google ; “ISP” : google cloud)

    I’ve search trough the documentation, the forum and other ressources, couldn’t find a reason nor a workarround.

    So, the excluded pages (6) of the search console are (I had to look for to find their urls):
    – I only have one 404 page (wich is the “exemple article by default on wordpress”). I’ve deleted this page, don’t know why it still appears to google (and don’t know how to do the erase/delete this url ?). Anyway, don’t found other 404 errors !
    – I do have 2 pages with redirections, wich are well configured.
    – I do have some “noindex” pages, but by default, google shouldn’t try to reach them ?
    – I do have one page I don’t want, but don’t know how it did appear nor how to get rid of… ( https://example.com/accueil-xxx-dordogne-24/feed/) ; when I try to open it, it’s only allowed to download or to ‘look at’ because it’s a “application/rss+xml” ???

    So how and why does iTheme Security decided to “block” that IP ??? What can I do to avoid it ?

    Thanks very much to who will be able to help me out !!

Viewing 9 replies - 1 through 9 (of 9 total)

  • Hi,

    The plugin blocked the IP because it was attempting to access a file that does not exist and viewed that IP as malicious.You’ll need to remove that IP from the Banned Users field, and disable 404 Detection to prevent this from happening until the 404s are fixed.

    Thanks,

    Matt

    @lsphoto,

    I recommend removing all Ban Hosts and Ban User Agents (i.e., IPs) from your “Banned Users” setting (fresh start), but leaving 404 Detection on with proper rules in place to block IPs that try to access a file that doesn’t exist more than “X” times. This is what we use (click here).

    Once those IPs are blocked or detected, then perform an IP WHOIS LOOKUP analysis to determine if the offending IPs belong to a good or bad bot (i.e., crawler) or malicious source (i.e., some obscure IP located in China, Germany, etc.).

    To further assist, below please find additional resources that will help you make that determination (many others can be found online).

    https://www.keycdn.com/blog/web-crawlers
    https://www.abuseipdb.com
    https://whatismyipaddress.com/blacklist-check

    One last thing, time permitting and once you have identified or confirmed a malicious IP, don’t hesitate to report it via proper channels. Together, we need to fight back and ensure those behind these malicious IP have their day in court or their account suspended by their hosting company.

    Thread Starter lsphoto

    (@lsphoto)

    Hello !
    I’m sorry for my late reply..

    @beardedginger : my site is new, and the banned IPs logs are already long ! I’ve searched hours yesterdeday and didn’t find the google bot IP I spoke of, I don’t understand…
    Also, I don’t have problems. I only have a few page site and the few ones are redirected correctly
    But there is new about hoster bots, see below.

    @jetxpert Thanks for the great resources !!
    About the settings, I’ve some differences but not much (“how long lockouts…” : 12 ; 30min to “remember 404”).
    You say something really interesting : to “remove all Ban Hosts and Ban User Agents (i.e., IPs) from your “Banned Users” setting (fresh start)”. I today had another hoster blocked, using a proxy but located in Germany, another in USA (I’m in France) etc but I didn’t find on your link if it is malicious or not ??? I do find others too, why are hosters have bots crawling my site, how to know if they are malicious ? Secondly, you speek about User Agents, how to know wich they are, and again why don’t block them ??
    Then you say “take a fresh start”, do you want me to delete all logs ?
    Finally, I have to learn more about these logs, I do have some results for “brute force”, and also two red lines with “fatal error”, without IP, for “altereted files” and “bad analysis”..

    How (where) to report malicious IPs, if I’m sure I find one ?

    Thanks again !!

    @lsphoto,

    (1) Please provide the name and/or IP of the blocked host. I will guide you further and explain above better.

    (2) If you have a very large log, use the plugin, Log Cleaner for iThemes Security, to delete your logs and start again to find out which bots, etc. are crawling your site. It’s a great plugin and safe to use.

    (3) Have you contacted your hosting company to find out if the can help you?

    (4) Does your website use a CDN such as Cloudflare? If not, you should sign up (it’s free and highly effective in containing malicious bots, etc.) Your hosting company can help you with that.

    Thread Starter lsphoto

    (@lsphoto)

    Hi @jetxpert
    Thanks again !

    1) for example:
    – This one has been blocked some days ago, and for the second time today! (the IP blocked was 94.130.51.22, seems to be a german hoster..)
    “inetnum: 94.130.51.0 – 94.130.51.63
    netname: HETZNER-fsn1-dc1
    descr: Hetzner Online GmbH
    descr: Datacenter fsn1-dc1
    country: DE
    admin-c: HOAC1-RIPE
    tech-c: HOAC1-RIPE
    status: ASSIGNED PA
    remarks: INFRA-AW
    mnt-by: HOS-GUN
    mnt-lower: HOS-GUN
    mnt-routes: HOS-GUN
    created: 2018-03-15T13:48:07Z
    last-modified: 2018-03-15T13:48:07Z
    source: RIPE

    role: Hetzner Online GmbH – Contact Role
    address: Hetzner Online GmbH
    address: Industriestrasse 25
    address: D-91710 Gunzenhausen
    address: Germany”

    It seems to be a hoster but… ?

    – Another, seems to be a NL society (via VPN) but located in Italy..
    “inetnum: 213.152.161.0 – 213.152.161.184
    netname: NL-AIR
    descr: AirVPN.org
    country: NL
    descr: ****************************************************
    descr: Alblasserdam datacenter
    descr: AirVPN IP Space
    descr: NL, Europe
    descr: ****************************************************
    admin-c: PB18435-RIPE
    tech-c: PB18435-RIPE
    status: ASSIGNED PA
    remarks: INFRA-AW
    mnt-by: GLOBALLAYER
    created: 2017-01-10T21:06:09Z
    last-modified: 2017-01-10T21:06:09Z
    source: RIPE # Filtered

    person: Paolo Brini
    address: c/o Studio Papa Via Vecchi, 53
    address: I-06100 PERUGIA
    address: Italy”

    -last exemple : seems to be Google, but different IPs are used from day to day
    “Query terms are ambiguous. The query is assumed to be:
    # “n 66.249.93.86”
    #
    # Use “?” to get help.
    #

    NetRange: 66.249.64.0 – 66.249.95.255
    CIDR: 66.249.64.0/19
    NetName: GOOGLE
    NetHandle: NET-66-249-64-0-1
    Parent: NET66 (NET-66-0-0-0-0)
    NetType: Direct Allocation
    OriginAS:
    Organization: Google LLC (GOGL)
    RegDate: 2004-03-05
    Updated: 2012-02-24
    Ref: https://rdap.arin.net/registry/ip/66.249.64.0

    OrgName: Google LLC
    OrgId: GOGL
    Address: 1600 Amphitheatre Parkway
    City: Mountain View
    StateProv: CA
    PostalCode: 94043
    Country: US
    RegDate: 2000-03-30
    Updated: 2019-10-31″

    I do have 30 pages of log (site is online less than one month)

    2) I’ll have a look at this plugin, thanks !

    3) I’ve written to my hoster to ask how to detect malicious IPs, but I said I was using iThemeSecurity… So they answered “you’ve to ask the plugin support”…

    4) I don’t use CDN because, first I only have a very little site, with almost ‘green’ everywhere (google insight, Lighthouse, mobile ergonomy, GTMetrix, Pingdom etc), so don’t think it’s worth ??
    Secondly, I wanted to keep my own URLs.
    Waht are the cons/pro of this solution ? (how does it “contain malicious bots”??). I should learn more about it…

    Thanks again for your support !!!

    @lsphoto,

    (1) Thank you for the information. In response:

    (a) Hetzner Online: We have also been attacked by them. Continue to block them. You can also block them using Cloudflare. To report them, you can visit the following links:

    https://www.abuseipdb.com/ (use the offending IP addresses)
    https://www.whatismyip.com/ip-whois-lookup/ (enter the offending IP address, then send an email to the “[email protected]” email address that appears in the results)

    (b) Global Layer: We have also been attacked by them. Same as above. In this case, “[email protected]” is the corresponding address.

    (c) Google: Legitimate. Well-known bot and crawler. OK to whitelist them. Sometimes they are blocked because they are looking for website links that may be part of your sitemap (helpful for SEO).

    (2) OK

    (3) Based on your hosting company’s feedback, recommend purchasing iThemes Security Pro. They can help you with all of your concerns. Also, follow the links I provided above. There’s plenty of good info online as well.

    (4) Your choice, but will help you a lot to sign up for Cloudflare. At first, it may seem intimidating, but once set up, you’ll enjoy the benefits. Remember, it’s free. Your hosting company can help you set this up as well.

    Best wishes.

    Thread Starter lsphoto

    (@lsphoto)

    I must add, I just found back the previous IP the first example has used some days ago (as you see, not the same “IP range”…) :
    “inetnum: 94.130.91.128 – 94.130.91.191
    netname: HETZNER-fsn1-dc1
    descr: Hetzner Online GmbH
    descr: Datacenter fsn1-dc1
    country: DE
    admin-c: HOAC1-RIPE
    tech-c: HOAC1-RIPE
    status: ASSIGNED PA
    remarks: INFRA-AW
    mnt-by: HOS-GUN
    mnt-lower: HOS-GUN
    mnt-routes: HOS-GUN
    created: 2018-03-15T14:12:33Z
    last-modified: 2018-03-15T14:12:33Z
    source: RIPE

    role: Hetzner Online GmbH – Contact Role
    address: Hetzner Online GmbH
    address: Industriestrasse 25
    address: D-91710 Gunzenhausen
    address: Germany
    phone: +49 9831 505-0
    fax-no: +49 9831 505-3
    abuse-mailbox: [email protected]
    remarks: *************************************************
    remarks: * For spam/abuse/security issues please contact *
    remarks: * [email protected], not this address. *
    remarks: * The contents of your abuse email will be *
    remarks: * forwarded directly on to our client for *
    remarks: * handling. *
    remarks: *************************************************
    remarks:
    remarks: *************************************************
    remarks: * Any questions on Peering please send to *
    remarks: * [email protected] *
    remarks: *************************************************
    org: ORG-HOA1-RIPE

    What could be their goal ???

    And a third entry, again a couple of days earlier, again with another IP range…
    “Information related to ‘88.99.57.128 – 88.99.57.191’

    % Abuse contact for ‘88.99.57.128 – 88.99.57.191’ is ‘[email protected]’

    inetnum: 88.99.57.128 – 88.99.57.191
    netname: HETZNER-fsn1-dc1
    descr: Hetzner Online GmbH
    descr: Datacenter fsn1-dc1
    country: DE
    admin-c: HOAC1-RIPE
    tech-c: HOAC1-RIPE
    status: ASSIGNED PA
    remarks: INFRA-AW
    mnt-by: HOS-GUN
    mnt-lower: HOS-GUN
    mnt-routes: HOS-GUN
    created: 2018-03-15T14:19:23Z
    last-modified: 2018-03-15T14:19:23Z
    source: RIPE

    role: Hetzner Online GmbH – Contact Role
    address: Hetzner Online GmbH
    address: Industriestrasse 25
    address: D-91710 Gunzenhausen
    address: Germany
    phone: +49 9831 505-0
    fax-no: +49 9831 505-3
    abuse-mailbox: [email protected]
    remarks: *************************************************
    remarks: * For spam/abuse/security issues please contact *
    remarks: * [email protected], not this address. *
    remarks: * The contents of your abuse email will be *
    remarks: * forwarded directly on to our client for *
    remarks: * handling. *
    remarks: *************************************************
    remarks:
    remarks: *************************************************
    remarks: * Any questions on Peering please send to *
    remarks: * [email protected] *
    remarks: *************************************************
    org: ORG-HOA1-RIPE”

    • This reply was modified 4 years, 5 months ago by lsphoto.

    @lsphoto,

    One other thing concerning Cloudflare … not only will it block malicious IPs, it will also increase your website’s speed. You also get to keep your URLs.

    Cheers!

    Thread Starter lsphoto

    (@lsphoto)

    @jetxpert

    Some more questions, because you’re a great helper !!! ?? :
    – I do use cache plugin, can’t Cloudfare have some conflict with it ??
    – I found -while testing a page on google insight- a “UNPKG file”. After some research, it seems (?) that it belongs to Cloudfare ??? How can that happen ??
    – last : if I install Cloudfare, what other ‘conflict’ could happen with my already installed plugins ? (as “Redirections”, “iThemesSecurity”, “WP-Optimize”, “Imagify”, “Formidable forms”, “Updraft backup”, “Yoast”…..???). I ask because these kind of plugins do have some kind of cache, or settings that “can move with the way you use my site” (sorry for bad English)

    In fact, with a site with ~10 pages, how could Cloudfare help me out ? OK about malicious IPs, but as I do use of cache plugin, plus plugins wich need to react immediatly (security, redirects, backup, even Yoast..), I’m afraid of what Cloudfare could “defer”??? Am I totally wrong?

    I’ve read the ressources you gave me earlier, very interesting, but I still don’t understand what for instance how to detect a malicious IP with “IP WHOIS LOOKUP” ? Yes there are informations, but … What do you think about the example I gave, particularly with that “Hetzner german company”??? They came back using different range of IPs each time, I find it strange ?

    A very big thanks once more time !!!

Viewing 9 replies - 1 through 9 (of 9 total)
  • The topic ‘Google bots blocked by iThemes ??’ is closed to new replies.