facebookexternalhit/1.1 thousands of requests

Hello @wizard71,

Thank you for reaching us.

Please give us 2-3 business days to investigate this case.

We’ll get back to you as soon as we have any news.

Hello @wizard71,

We’re still working to resolve this issue. As a temporary solution, you can add the following text to your robots.txt to block Facebook bots:

User-agent: facebookexternalhit
Disallow: /

We’ll get back to you as soon as we finish.

We are facing the same problem… just lurking for a solution, thank you!

we are also facing the same issue where we are getting many hits which eats up my servor resources

We have disallow it- blocked it but still its is consuming my resources and finally slowing down the website performance

Hello @bds1975,

Find out the name of the user agent of these crawlers and add them to robots.txt. I’m guessing it was another Facebook crawler called Facebot.

@bds1975 some articles say that Facebook bot does not respect robots.txt so I found a different solution.

Use this plugin: https://github.com/nadimtuhin/Facebook-Request-Throttle-WordPress-Plugin
I set the limit to 60 seconds, and it’s working.

Hello everyone,

same issue here with my eshop..for a couple of days. I check the plugin, how you set the limit to 60 secs, like this:

define(‘FACEBOOK_REQUEST_THROTTLE’, 60.0);

Is that correct? And why 60 secs and not something else? Could you give us a feedback about your site, I mean is it working ok, performance etc etc ??? thank you for your time!!

@mariakos correct. I used 60 seconds because it gave me a more less optimal result and I don’t see any drop in Facebook traffic.

Hello.

I notice the same problem on several stores. The correlation I see is that queries are being generated to links like: ‘shop/[paramsfilter]’

Do Facebook crawlers have any errors, because this appeared recently?

Same experience here. A client’s website got over 1/2 million hits from facebookexternalhit/1\.1 starting 6/21/24, eating up 33GB of bandwidth in a few days. All hitting the same page in their website every few seconds. They don’t even have a Facebook account, nor do they desire the traffic.

So, we just blocked the facebookexternalhit BOT in .htaccess with this:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^facebookexternalhit/1\.1 [NC]
RewriteRule ^ – [F,L]

If you don’t care about Facebook, then that might be an option for you.

There must be something wrong with Facebook’s BOT that they’re not aware of…

Hi all,

We apologize for the delay, we are still working on a solution. It may take 1-2 weeks more. We will inform you.

We have removed the restriction on adding any addresses. You can use your personal lists to block these IPs:

https://cleantalk.org/help/security-firewall

Hello, I had the same problem.

I managed to stop the phenomenon with the help of the https://botspy24.com/ application, I first observed and then banned the agents causing the overload..

Hello @habaj,

Have you tried using our personal lists that we mentioned in the previous comment? Did it help you?

We recently encountered an issue where our cloud server crashed due to an aggressive passage of the Facebook bot. Here is the user agent:

facebookexternalhit/1.1 (+https://www.facebook.com/externalhit_uatext.php)

As discussed with Infomaniak support, blocking the bot via robots.txt is not an effective solution since the Facebook bot ignores these directives. However, we cannot completely block this bot because we rely on it for our shop products that are featured on Facebook.

Could you please guide us on how to limit the access of the Facebook bot to avoid server overloads while still allowing it to crawl our essential product pages?