Hi @rsl89, thanks for getting in touch.
You can limit the amount of traffic that attempts to hit your site with our Rate Limiting Rules on the Firewall Options page. This configures how crawlers and humans are treated.
I generally set my Rate Limiting Rules to these values to start with:
Rate Limiting Screenshot
- If anyone’s requests exceed – 240 per minute
- If a crawler’s page views exceed – 120 per minute
- If a crawler’s pages not found (404s) exceed – 60 per minute
- If a human’s page views exceed – 120 per minute
- If a human’s pages not found (404s) exceed – 60 per minute
- How long is an IP address blocked when it breaks a rule – 30 minutes
I also always set the rule to Throttle instead of Block. Throttling is generally better than blocking because any good search engine understands what happened if it is mistakenly blocked and your site isn’t penalized because of it. Make sure and set your Rate Limiting Rules realistically and set the value for how long an IP is blocked to 30 minutes or so.
Remember there is no hard and fast, one size fits all set of rules for every site. This is just a good place to start. During an attack you may want to make those rules stricter. If you see visitors, like search engine crawlers getting blocked too often, you might want to loosen them up a little.
Thanks,
Peter.