very difficult to limit facebookexternalhit

  • Resolved volkerforster

    (@volkerforster)


    5 years, 5 months ago

    Hi,
    I found a thread here a year old, and it wasn’t completely addressing the issue I have with the Facebook bot. Therefore, I will start a new one:

    facebookexternalhit fires from about 50 different IPs, and it looks like Wordfence would ignore this fact, and only goes by the IPs. I have set crawl limits to 30 per minute for anyone except Google, but since Facebook “attacks” from many different IPs, “facebookexternalhit” still hits almost 100 times within just 10 seconds, and spikes my server CPU load to the limit, causing my site to crash for some minutes. My hosting company’s advice to upgrade my cloud service to more CPUs is very nice, because they make big Dollars just like that. But this can’t be the fix. I don’t want to block Facebook’s crawler completely, since most advertising is on FB and therefore most visitors come from there. But I need to limit it on “crawler level”, not on IP level. I think this is a flaw in Wordfence, to allow a bot unlimited access, when the bot uses unlimited IPs to crawl your site. This should be fixed and updated to limiting a crawler regardless the IP used for the visit. Any idea how to achieve the limiting without blocking?

Viewing 3 replies - 1 through 3 (of 3 total)

  • Hi,
    I have such kind of attacks/hits from facebookexternalhit IP’s (they pretend to be) on all my sites. I have block them all. From what I read, it’s a DDoS attacks from malware botnets actively scanning sites from past two weeks maybe. It’s not a common Wordfence flaw, I think.

    Thread Starter volkerforster

    (@volkerforster)

    How to get any support to Wordfence? I am using the paid version, I really would like to get some support here!!

    I keep blocking huge IP ranges from where the attacks happen, but Wordfence does not work as expected. It shows the certain IP ranges blocked, but the requests still cause high load on my server until it crashes.

    How is this even possible?

    They fire from many IPs, and it is NOT POSSIBLE to block them on IP level (because of that), but Wordfence shows always the same name for the bot:

    facebookexternalhit/1.1 (+https://www.facebook.com/externalhit_uatext.php)

    Why is Wordfence not capable of blocking this completely?

    I also miss a useful setting for request limitations. I can limit a bot to a value per minute, but this is not good enough. When I set the limit to 10 per minute, it still means they can send 10 requests within 1/10 of a second, which is too much, specially since they use many IPs, probably 100s of different IPs.

    It turns out that Wordfence is completely useless to help with these Facebook attacks!! The question is, which other tool provides a better protection and better blocking against attacks like that?

    Plugin Support wfphil

    (@wfphil)

    Hi @volkerforster

    If you are a premium user you can access premium support from your account and open a ticket.

    A good quality hosting package should not crash just because of too many visits from Facebook’s crawler bot.

    Wordfence will block IP addresses if they exist in an IP range that you have blocked but Wordfence cannot prevent requests being made to your server, so Wordfence will be working as expected. However, Wordfence blocking naturally uses less server resources than if those requests weren’t blocked at all and WordPress was fully loaded.

    You could use your robots.txt file to rate limit Facebook’s crawler bot using the example guide here below:

    https://blog-en.openalfa.com/how-to-limit-the-crawl-rate-of-bots-in-a-website

    https://developers.facebook.com/docs/sharing/webmasters/crawler/

    We recommend that you set the Rate Limiting option “If a crawler’s page views exceed” to 240 per minute. If you have set that to 30 per minute then you will be disallowing friendly crawlers from crawling your site properly. Please read our recommended settings here:

    https://www.wordfence.com/help/firewall/rate-limiting/

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘very difficult to limit facebookexternalhit’ is closed to new replies.