Search engine crawler request overload

  • Resolved fabiopwl37

    (@fabiopwl37)


    5 years, 8 months ago

    The plugin is doing what it is intended to. It’s block multiple bots for me.

    However, whitelisted bots are sending a ton of requests to the blackhole pages. My access log is flooded with HTTP requests to pages ending in “/?blackhole=”. Server CPU utilization has been very high since it started.

    • This topic was modified 5 years, 8 months ago by fabiopwl37.
Viewing 1 replies (of 1 total)
  • Plugin Author Jeff Starr

    (@specialk)

    Yep, whitelisted bots can do whatever they want. It always is your decision whether to whitelist (always allow) any bot(s) that are necessary. To block them, simply remove them from the whitelist. But be careful to know exactly what you are doing, etc.

Viewing 1 replies (of 1 total)
  • The topic ‘Search engine crawler request overload’ is closed to new replies.