BPS Pro blocking uptime monitor

If you want to allow the uptimebot and all other bots just remove HEAD from this nuisance filter in your Root .htaccess file. This is NOT a security filter and is just for blocking junk and spam bots and in your case this good bot that you want to allow.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F,L]

And I guess you could actually do something like this below, but I have not tested this so I don’t know if it would work on not. In theory it should work. So give it a try and let me know if it works. Be prepared to delete your Root .htaccess file if something goes wrong.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot|another-example-good-bot) [NC]
RewriteRule ^(.*)$ - [F,L]

Hmm I wonder if you would need to also add the version number – /1.0 ? If the code does not work above then try this code.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot\/1\.0|another-example-good-bot) [NC]
RewriteRule ^(.*)$ - [F,L]

Actually it does work and you do NOT need to add the version number. I just tested Web Sniffer and I got a 200 response.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(Web-sniffer|another-example-good-bot) [NC]
RewriteRule ^(.*)$ - [F,L]

Status: HTTP/1.1 200 OK
Date:	Thu, 24 Jan 2013 23:28:24 GMT
Server:	Apache
X-Pingback:	https://forum.ait-pro.com/xmlrpc.php
X-Powered-By:	W3 Total Cache/0.9.2.4
Set-Cookie:	w3tc_referrer=http%3A%2F%2Fweb-sniffer.net%2F; path=/
Set-Cookie:	bp-message=deleted; expires=Wed, 25-Jan-2012 23:28:24 GMT; path=/
Set-Cookie:	bp-message-type=deleted; expires=Wed, 25-Jan-2012 23:28:24 GMT; path=/
Vary:	Accept-Encoding,User-Agent
Connection:	close
Content-Type:	text/html; charset=UTF-8

I’ll add an additional “NOTE” in the next release of BPS for folks.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# If you want to allow ONLY specific good bots to make a HEAD Request then do this:
# RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
# RewriteCond %{HTTP_USER_AGENT} !^(example-good-bot|another-example-good-bot) [NC]
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.

This method did indeed work for me.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(HEAD|TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot|another-example-good-bot) [NC]
RewriteRule ^(.*)$ - [F,L]

Thanks as always for the great support!

Great! You can remove the “another-example-good-bot” example and just use:

RewriteCond %{HTTP_USER_AGENT} !^(UptimeRobot) [NC]

If you need to add another good bot at a later date/time then you would follow this example fomat – Bot name separated by the Pipe Operator – (bot1|bot2|bot3|bot4)

Looks like I spoke too soon. I monitored it for a bit and got no new errors. Saved everything and left it alone for 2 days, came back to an log file over 100kb with the same error over and over. It seems very sporadic. Sometimes it’s able to get through I think and others it gets a 403 error. Thinking I might have to disable the security log ability all together unless there is some work around. Bummer! ??

Well disabling the error logging does not fix the issue/problem. The error log just makes it easy for you to see what the exact problem is. So actually what you would want to do is allow all bots to make a HEAD request, but before you do that is the error the exact same one or are they different errors.

On average we log 1,500+ blocked hacking attempts per day so our log files fill up real fast. ??

You just made me think of a hilarious analogy. My gas guage “out of gas” light keeps telling me I am out of gas – solution remove the light bulb. ha ha ha. Just kiddin of course. ??

LOL Well I know you’re right! Here is the entry from a minute ago:

>>>>>>>>>>> 403 Error Logged – January 28, 2013 – 2:07 pm <<<<<<<<<<<
REMOTE_ADDR: 74.86.158.106
Host Name: 74.86.158.106-static.reverse.softlayer.com
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0+(compatible; UptimeRobot/1.0; https://www.uptimerobot.com/)

Same error. I tried changing around the way the user agent was listed to be UptimeRobot/1.0 and even https://www.uptimerobot.com/ just to see (grasping at straws) if it would work, but no dice!

Good I did not piss you off so you must have a good healthy sense of humor too. ??

Well actually try the first thing I posted and just remove HEAD from the nuisance filter altogether. It is possible that some other type of Request is being made by this UA / Bot, but before going any deeper do this first.

# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F,L]

Hmm looking at the documentation for uptime it may be doing something more sophisticated then just making a HEAD request. There appears to be an API and you may actually have to install something on your website.

Did you install something on your site for uptime? I see that you can sign up for free, but what happens after that?

I am actually using ManageWP which uses the UptimeRobot API. So, yes – I had to install a plugin on my site, but I am not certain what that plugin does outside of giving me access to all of my sites from one dashboard.

Yep because it just dawned on me that there is not a Referrer in the error you posted. When I tested web-sniffer you can see that the Referrer is the web-sniffer site. So what this probably means is the error is actually occuring on your site itself and is not a HEAD Request issue at all. I am not familiar with ManageWP so I only have a very general idea about what it does. I could not find any help info on the ManageWP site that tells me how it works. I see they have a free trial available so when i have some spare time I will test it and see what the issue is with uptime.

>>>>>>>>>>> 403 Error Logged - January 24, 2013 - 3:26 pm <<<<<<<<<<<
REMOTE_ADDR: 37.200.98.39
Host Name: j31966.servers.jiffybox.net
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER: https://web-sniffer.net/
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Web-sniffer/1.0.44 (+https://web-sniffer.net/)