Wrong Bot Detection: Fake Web Crawler

  • Resolved dimal

    (@dimalifragis)


    1 year, 5 months ago

    Hi.

    I see this:

    85.208.98.17: Bot Detection: Fake Web Crawler – Fake Web Crawler detected at “/”. Fake Crawler misrepresented itself as “SemrushBot”.

    But it IS Semrush bot (rDNS bot.semrush.com)

    Thanks

Viewing 5 replies - 1 through 5 (of 5 total)
  • Plugin Author Paul

    (@paultgoodchild)

    This will be fixed in the next release, thanks for reporting it.

    Plugin Author Paul

    (@paultgoodchild)

    Just to edit our earlier response, upon further investigation, this isn’t a problem we can fix, assuming it is actually problem. There’s nothing to show that this IP address is an official semrush bot.

    Just so you’re aware, rDNS doesn’t prove that an IP address is from a given location. It’s very easy to fake rDNS.

    If you review semrush documentation:

    That IP address you specified isn’t provided in their documentation as an official bot.

    You may want to reach out to semrush for clarification.

    We used to reach out to providers like ahrefs etc. when they weren’t doing things right, but that’s a full time job and we don’t do it any longer.

    If “official” spiders and bots want to prevent fake web crawlers, then they need to do better at providing access to official IP address lists or improve their rDNS/forward DNS resolution.

    A simple rDNS isn’t sufficient to verify a bot.

    Thread Starter dimal

    (@dimalifragis)

    Ok, still it should be semrush bot

    https://ipinfo.io/85.208.98.17

    Plugin Author Paul

    (@paultgoodchild)

    it’s not on their list.

    Can you suggest ways we can verify it’s semrush bot automatically if the IP isn’t on their list? Simple rDNS isn’t a solution

    Thread Starter dimal

    (@dimalifragis)

    I completely understand what you say but i have no idea. I will (try to) email them with this topic link and see how it goes.

Viewing 5 replies - 1 through 5 (of 5 total)
  • The topic ‘Wrong Bot Detection: Fake Web Crawler’ is closed to new replies.