Block IPs that access these URLS
-
please see my previous support ticket: Question about Immediately block IPs that access these URLs. I wasn’t sure if it actually reopened it so I’ve added this new request.
I added /ecwd_calendar/* to the Immediately block IP’s that access these URLS option and it is the only URL that I’ve added there. However, it appears that MSN crawlers are getting blocked by this rule when trying to access other URLs. See the following example
United States was blocked for Accessed a banned URL at https://www.MYDOMAINNAME.ca/tutorial-from-watercolours-to-pastel/
11/26/2019 12:55:14 PM (1 hour 44 mins ago)
IP: 207.46.13.39 Hostname: msnbot-207-46-13-39.search.msn.com
Human/Bot: Bot
Browser: undefined
Mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)
Do you know why this would happen?
Kathy
-
Hi @pacadmin01
Please respond to Dave in your other topic to let him know that you have opened a new topic so that he doesn’t spend time responding as I have resolved it here.
If Bing’s crawler crawled a URL that it has in it’s index containing /ecwd_calendar/ then it will be blocked for accessing a banned URL.
It will be blocked for the duration set by the option How long is an IP address blocked when it breaks a rule in the Rate Limiting section on the All Options page.
Therefore whilst the block is in place then Bing’s crawler will be blocked for that duration for any request it makes to your site.
Because it is blocked, all visits to other URL’s will say blocked for Accessed a banned URL because that is what it was originally blocked for.
As you have removed the WD Events Calendar and you don’t want to block search engine crawlers I recommend that you remove that rule.
As the WD Events Calendar has been removed and all requests for it will result in a 404 Not Found response status code then there isn’t a security threat.
- The topic ‘Block IPs that access these URLS’ is closed to new replies.
