facebookexternalhit bot

Hi @siribakke01,

The rate-limiting settings can be adjusted to be more or less strict depending on how the site is configured. You can find details and our recommended settings here: https://www.wordfence.com/help/firewall/rate-limiting/

For “If a crawler’s page views exceed…“, we recommend setting the limit to 120 per minute to start, but you may need to use a stricter setting. Use the “throttle” option in most cases, which will rate-limit crawlers rather than block them.

Please keep in mind that as a web application firewall, while Wordfence can prevent users from visiting the site, it can’t prevent bad actors from accessing the server altogether. If the bot is causing high CPU use even when the rate limiting is active, your host may be able to help by adjusting the server firewall.

Please let us know if you have any questions!

Thanks,
Margaret

Hi Margaret,

Thank you for your help greatly appreciated!

I have set the “If a crawler’s page views exceed…“ limit to 120 per minute which helped a bit. I then reduced it to 60 howevere it seems like the facebook bot is still a problem..

I noticed under ‘Advanced Firewall options” there is a ‘allowlisted serviced’ option. Would unticking Facebook here help and force the facebook bot not hitting so much?

Thank you so much for your help.

Best regards,

Siri

Hi @siribakke01,

Unticking Facebook in Allowlisted Services will ensure Wordfence fully throttles Facebook. But removing Facebook from the allowlisted services can lead to issues with how content from your website displays on Facebook, such as when someone shares your site, if the connection is throttled or blocked.

I recommend adding a crawl delay to your robots.txt file first:
https://developers.facebook.com/docs/sharing/bot/

Facebook crawlers should respect the crawl delay to automatically throttle themselves on your site, which may allow you to leave Facebook as an allowlisted service on your site.

Let me know how it goes!

Thanks,
Margaret

–

Thank you so much for your help! I’m trying out the

User-Agent: FacebookBot
Crawl-delay: x

It seems to get a little better then goes back up again. What is the range of delays one can add? eg in the article it says Crawl-delay: 5. I been trying out 5, 10, 30 and 120. Will higher the number slow it down further? Or is there a number where Facebook bot don’t follow the delay and just keep going?

Any advice would be greatly appreciated. I seem to get it to slow down a little then it spikes back up!

Best regards,

Siri

Hi @siribakke01,

The crawl delay is how many seconds the bot should wait before crawling the next page, so the higher the number is the slower the bot should crawl. A quick search reveals some reports that Facebook’s bot may not always respect the crawl delay in robots.txt.

If the crawl delay isn’t helping, you may need to untick Facebook in Allowlisted Services. Since a web application firewall can’t prevent bots from hitting the server altogether, you may also want to reach out to your host to see if there are any adjustments they can make to the firewall, or consider the use of a CDN that can block or throttle bots before they reach the server.

Thanks,
Margaret