bingbot causing problems on my site

  • Resolved liquid8

    (@liquid8)


    10 years, 12 months ago

    Hi,

    I got message from my host provider today that my site is having so much traffi that they had to close it.
    I checked apache’s log and found a lot of rows like this:

    157.55.33.93 – – [01/Apr/2014:19:45:28 +0300] “GET /kalenteri/action~agenda/cat_ids~14/tag_ids~17,37,46,25,64/ HTTP/1.1” 200 13308 “-” “Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)”

    66.249.78.51 – – [01/Apr/2014:19:45:29 +0300] “GET /kalenteri/action~posterboard/time_limit~1386453600/tag_ids~81/ HTTP/1.1” 200 10784 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)”

    157.55.33.93 – – [01/Apr/2014:19:45:33 +0300] “GET /kalenteri/action~agenda/cat_ids~14/tag_ids~17,23,19,24,28,47/ HTTP/1.1” 200 13345 “-” “Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)”

    Could somebody tell me what’s going on and should I try to block bingbot’s requests? Any ideas would be very welcome.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Moderator bcworkz

    (@bcworkz)

    You can easily prevent Google and Bing bots from accessing all or part of your site by using a robots.txt file. Preventing access will result in people not being able to find your site’s content using the related search services. There’s plenty of examples online on how to use this file.

    You might try the crawl-delay directive. It is not an official directive, but I understand most of the major bots honor the directive. YMMV

    Thread Starter liquid8

    (@liquid8)

    Thank you for your answer. I added robots.txt with crawl-delay -definition. Here’s how my file looks like now in case someone is interested about this too:

    User-agent: Googlebot
    Crawl-delay: 20

    User-agent: Slurp
    Crawl-delay: 20

    User-Agent: msnbot
    Crawl-Delay: 20

    User-Agent: bingbot
    Crawl-Delay: 20

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘bingbot causing problems on my site’ is closed to new replies.