How are bots, crawlers, spiders etc identified?

  • Resolved digbymaass

    (@digbymaass)


    6 years, 2 months ago

    In my battle with bandwidth consuming bots I’m constantly looking at live traffic. I have various bot blockers in place.
    Traffic which is local to us and is often evidently a normal user is being identified as a bot. eg 129.215.224.30 which Whois Lookup identifies as ‘The Edinburgh University local area network’ is flagged as being a bot.
    Are these mis-identifications? or bots spoofing addresses?

    How does Wordfence identify a bot? And if a bot spoofs an address would that show as a bot?

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
  • Ambyomoron

    (@josiah-s-carberry)

    Perhaps you are aware that most http requests include an agent field, containing information such as:

    Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +https://yandex.com/bots)
Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; SemrushBot/3~bl; +https://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; localsearch-web/2.0; +https://www.localsearch.ch/en/legal-notice)
Sogou web spider/4.0(+https://www.sogou.com/docs/help/webmasters.htm#07)

    …and so forth.

    Of course, malevolent visitors may seek to hide their identity by providing a different string, as well as spoofing other fields. Remember, too, that third party computers can be infected and participate in networks of attacks.

    In any case, Wordfence support will let you know more precisely how it determines that a visit is by a bot. There are many aspects of a visit that could be exploited to qualify the visitor as a bot.

    Thread Starter digbymaass

    (@digbymaass)

    The ones I’m thinking of are displaying normal viewing behaviour – a couple of visits to topical pages, not attacking behaviour.
    And yes, live traffic clearly shows the agent fields, and these ones don’t show any bot-like names.
    eg:

    Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1
    Location: Edinburgh, United Kingdom

    Hi @digbymaass,

    The bot detection in Wordfence uses JavaScript to detect if the user is clicking, scrolling, or interacting with the page.

    If the user has JavaScript disabled, or is using a different client such as a mobile browser, it may not register these events and will be flagged as a bot.

    Dave

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘How are bots, crawlers, spiders etc identified?’ is closed to new replies.