Rate Limiting facebookexternalhit

Hello @rifatspir and thanks for reaching out to us!

I generally set my Rate Limiting Rules to these values to start with:
If anyone’s requests exceed 240 per minute
If a crawler’s page views exceed 120 per minute
If a crawler’s pages not found (404s) exceed 60 per minute
If a human’s page views exceed 120 per minute
If a human’s pages not found (404s) exceed 60 per minute
How long is an IP address blocked when it breaks a rule 30 minutes

I also always set the rule to Throttle instead of Block. Throttling is generally better than blocking because any good search engine understands what happened if it is mistakenly blocked and your site isn’t penalized because of it. Make sure and set your Rate Limiting Rules realistically and set the value for how long an IP is blocked to 30 minutes or so.

https://www.wordfence.com/help/firewall/rate-limiting/ is an amazing resource for rate limiting rules.

You could use your robots.txt file to rate limit Facebook’s crawler bot using the example guide here below:
https://blog-en.openalfa.com/how-to-limit-the-crawl-rate-of-bots-in-a-website
https://developers.facebook.com/docs/sharing/webmasters/crawler/

Let me know if this helps!

Thanks!

@wfadam I have already set your recommend rate limiting and using crawl delay in robots txt. Nothing can slow down the Facebook bot. It hits once a day and use cpu (5 core) resources in that time.

If I block the bot, my site link preview does not work when someone share links in Facebook. I do not know why wordfence cannot identify the misbehavior of the bot.

Hello again @rifatspir

If you do a google search for Facebookbot not respecting robots.txt files, you should see that other people are apparently reporting this also. You might want to contact Facebook directly for a resolution to the issue.

Let me know if there is anything else I can do to help.

Thanks!