Wordfence Crawler – False Positive for Human

  • Resolved mpiette

    (@mpiette)


    6 years, 6 months ago

    Hi there,

    I am having problems with visitors getting an error for limited access with the message “blocked for Exceeded the maximum number of requests per minute for crawlers”

    The error they see is for “limited” but in my live scan summary it shows as a “blocked” bot.

    I had previously set the Rate Limiting for Crawler’s page view to 4 per minute. So this is what could have set it off, but is it possible that this user (which I know exists) was incorrectly identified as a crawler? How do I change that without completely whitelisting the IP?

    Thanks,
    Michelle

Viewing 2 replies - 1 through 2 (of 2 total)

  • Determining the difference between a human and bot is an inexact measure at best, so chances are high humans can be interpreted as bots.

    Best bet is to set the Rate Limiting value much higher to avoid this…

    Hi Michelle,

    Exactly as @bluebearmedia mentioned, distinguishing between human/bot visit is made by running certain scripts via the browser, sometimes due to different reasons (JS issue, browsers disabling JS, JS failure on some mobile devices, etc..) some visits aren’t detect accurately.

    Setting crawler’s page view to 4 per minute is way too low, my recommendation is to set “If a crawler’s page views exceed” option to 240 minutes, and to use the “throttle” option in most cases, which will simply rate limit crawlers instead of blocking them.

    Thanks.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Wordfence Crawler – False Positive for Human’ is closed to new replies.