Viewing 11 replies - 1 through 11 (of 11 total)
  • Plugin Author AITpro

    (@aitpro)

    Go to the BPS Security Log page and post the security log entry that shows what was blocked.

    The standard BPS code does not block any good Search Engine bots or affect crawling or indexing. Have you added any custom hotlink protection code that is blocking the Yahoo bot?

    You can use the “site” operator to double check that Yahoo Slurp is crawling and indexing your website pages successfully.

    Go to https://search.yahoo.com/
    Enter this search in the Yahoo search window: site:your-domain-name.com

    Plugin Author AITpro

    (@aitpro)

    Is the issue/problem still occurring or is it resolved?

    Thread Start Date: 5-7-2015 to 5-8-2015
    Current Date: 5-11-2015

    Thread Starter portwally1

    (@portwally1)

    Hello,

    thanks for the quick reply. Yes, the bot which was blocked was the yahoo slurp bot.
    h074.crawl.yahoo.net

    I think he is not an important one.

    Thans a lot
    Walter

    Plugin Author AITpro

    (@aitpro)

    The Yahoo Slurp bot is important and a legitimate bot. BPS standard htaccess code does not block legitimate bots like the Yahoo Slurp bot, Googlebot, etc from crawling and indexing website pages or in general. What can happen is that if you add additional custom htaccess code then that custom .htaccess code can block legitimate bots. Or it could simply be that a script (in a plugin or something else either internal or external) is doing something “additional” that is being blocked, but that does not affect crawling or indexing of your site by legitimate bots. This scenario is considered a nuisance error since it does not affect your website negatively in any way. You can disregard these nuisance errors.

    Plugin Author AITpro

    (@aitpro)

    Assuming all questions have been answered – thread has been resolved. If the issue/problem is not resolved or you have additional questions about this specific thread topic then you can post them at any time. We still receive email notifications when threads have been resolved.

    Thread Start Date: 5-7-2015 to 5-8-2015
    Thread Resolved/Current Date: 5-13-2015

    Thread Starter portwally1

    (@portwally1)

    Well, thanks. The slurp bot from yahoo comes when ever I do a new post.
    And in the logs he is blocked.

    Cheers
    Walter

    Plugin Author AITpro

    (@aitpro)

    BPS does not block the Yahoo Slurp bot or the Googlebot or any other legitimate search engine bots from crawling or indexing your website or any other legitimate thing that a legitimate bot does.

    let me try and explain this again.
    If you have some sort of script on your site or something else on your site that is related to sitemaps or SEO or something that relates to SERPS’s or the bot crawl and index is being done in a way that several different things are happening (ie 10 different things are happening and 1 of those things is blocked) or if the bot is not really the Yahoo Slurp bot and is faking the Yahoo Slurp User Agent / bot name.

    If any of these things above are happening then the end result is the same: The Yahoo Slurp bot crawls your site successfully and indexes your site successfully. Whatever “other” thing whether it is internal (something installed on your site) or external (some additional thing that is being done BESIDES just crawling and indexing your site) it is being blocked and not the Yahoo Slurp bot crawl or index and the same applies to all legitimate bots – Google, Bing, etc – BPS does not block any legitimate bots from crawling or indexing a website.

    Plugin Author AITpro

    (@aitpro)

    You can use the “site” operator to double check that Yahoo Slurp is crawling and indexing your website pages successfully.

    Go to https://search.yahoo.com/
    Enter this search in the Yahoo search window: site:your-domain-name.com

    You can also use other tools built into Yahoo and Google to verify and double check that.

    Plugin Author AITpro

    (@aitpro)

    Yahoo Site Explorer merged with Bing Webmaster Center (4 years ago) regarding the old Yahoo Site Explorer that no longer exists. So you would need to use the Bing Webmaster Center to check Bing and Yahoo indexed URL’s and Google Webmaster Tools for Google indexed URL’s.

    Thread Starter portwally1

    (@portwally1)

    Hello, thanks!

    Just want to give you the log of what I got from it.

    [403 GET / HEAD Request: May 13, 2015 2:36 pm]
    Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
    Solution: N/A - Hacker/Spammer Blocked/Forbidden
    REMOTE_ADDR: 74.6.254.134
    Host Name: h074.crawl.yahoo.net
    SERVER_PROTOCOL: HTTP/1.1
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /tourismus-seiten-portugal/
    QUERY_STRING:
    HTTP_USER_AGENT: Mozilla/5.0 (compatible; Yahoo! Slurp; https://help.yahoo.com/help/us/ysearch/slurp)
    Plugin Author AITpro

    (@aitpro)

    Yep something is being blocked, but not Yahoo Slurp crawling or indexing your website. Thanks.

Viewing 11 replies - 1 through 11 (of 11 total)
  • The topic ‘How do I whitelist an IP’ is closed to new replies.