Blocking “normal” crawlers?

  • Resolved Heiko Mitschke

    (@antares7)


    7 years, 10 months ago

    Hello!

    I am currently testing WP Ninja Firewall before buying the +-Edition for my sites.

    In my firewall log I find these entries with level medium:

    GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; spbot/5.0.3; +https://OpenLinkProfiler.org/bot )]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; spbot/5.0.3; +https://OpenLinkProfiler.org/bot )]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; spbot/5.0.3; +https://OpenLinkProfiler.org/bot )]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; spbot/5.0.3; +https://OpenLinkProfiler.org/bot )]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; AhrefsBot/5.2; +https://ahrefs.com/robot/)]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +https://www.semrush.com/bot.html)]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; seoscanners.net/1; [email protected])]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; seoscanners.net/1; [email protected])]
GET /index.php - Suspicious bots/scanners - [SERVER:HTTP_USER_AGENT = Mozilla/5.0 (compatible; AhrefsBot/5.2; +https://ahrefs.com/robot/)]

    Are they really “bad bots” or crawlers or just normal ones? I thought e.g. ahrefs.com is just a common backlink scanner. Am I wrong?

    Thanks in advance!

    Heiko

Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Author nintechnet

    (@nintechnet)

    Hi,

    They are crawlers but a bot such as AhrefsBot, for instance, does not follow the robots.txt file, hence we added it to the list of annoying bots.

    Semrushbot is actually OK. Or at least I’d like to be able to allow it.

    Is the only solution to disable the Bad Bots rule (# 531). Or is there some way to whitelist this particular user agent / IP address. I have disabled the rule for now.

    • This reply was modified 7 years, 10 months ago by peripatetic.
    Plugin Author nintechnet

    (@nintechnet)

    In the WP Edition, you need to disable the whole rule: “Firewall Policies > Block suspicious bots/scanners > No”.
    In the WP+ Edtion, you can remove/add/modify bots individually.

    Thread Starter Heiko Mitschke

    (@antares7)

    Thanks for your answer! I am glad that it is possible to change rules for bots individually in WP+.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Blocking “normal” crawlers?’ is closed to new replies.