"The User Agent" part of the plugin won't work for me and botsvsbrowsers.com

Yeah, most proxy services change identifying headers like user-agent, so you’re going to want to test using a service like https://www.hurl.it/ – there you can verify that BBQ is working correctly for the strings you mention.

Thanks, Jeff.

hurl.it script didn’t work and caused my browser crashing when I tried the website in question.

However, when I picked another website, hurl.it script worked seamlessly, showing 403/200 when I activated/deactivated BBQ. Even my curl worked in unison with hurl.it

Thanks again.

M.

Wait a minute, Jeff.

I have found and you confirmed that BBQ “The User Agent” part doesn’t protect the website in question behind the proxy.

We have direct hits from banned bots.

There are enough proxies all over the world to make BBQ “The User Agent” part useless.

Is there any workaround?

Thank you.

M.

I have found and you confirmed that BBQ “The User Agent” part doesn’t protect the website in question behind the proxy.

Actually, that is incorrect. BBQ does block malicious requests coming from proxy servers, and if the proxy is reporting a user agent that is blocked by BBQ, it should be blocked as well. In other words, BBQ will deny any request that reports a user agent that is included in the blocked-user-agents array.

There are enough proxies all over the world to make BBQ “The User Agent” part useless.

Lol, it may seem that way, but again, incorrect. As explained previously, BBQ denies requests from some of the worst user agents, regardless of whether the requests are made from behind a proxy.

Updated to add: to block tough proxies, check out this technique:

https://perishablepress.com/block-tough-proxies/

Sorry, Jeff.

There are tools.

I can’t agree with you as long as curl with BBQ banned user-agents dumps everything from the site like BBQ doesn’t exist.

The access log proves that the banned user agent has reached the site and hasn’t been denied.

Let’s take a closer look at this. Here is the actual user-agent array to which you refer:

$user_agent_array = apply_filters( 'user_agent_items', array( 'binlar', 'casper', 'cmswor', 'diavol', 'dotbot', 'finder', 'flicky', 'nutch', 'planet', 'purebot', 'pycurl', 'skygrid', 'sucker', 'turnit', 'vikspi', 'zmeu' ) );

As you can see, there is no “curl” user agent on the list, so requests that include curl in the user agent are not blocked by BBQ. So if the curl request is using the default user agent (e.g., curl/1.2.3), it is not blocked by BBQ, by design. Otherwise, if the curl request sets a custom user agent using something like CURLOPT_USERAGENT, then BBQ will block only if the specified user agent is in the previously stated array.

Sure, it’s easy to add ‘curl’ to array and check it out.

What I found that when I clear cache (WP Super Cache) and send curl, it gives 403.

curl -I https://xxxx
HTTP/1.1 403 Forbidden

As soon as a site being viewed in a browser and a cached file has been created, it gives 200.

curl -I https://xxxx
HTTP/1.1 200 OK

I was able to duplicate such pattern on other sites running WP Super Cache as many times as I tried.

I played with another site without WP Super Cache at all, ‘curl’ was blocked all the time.

I removed ‘curl’ from the array and continued sending requests using
‘–user-agent <agent string>’ to check the rest of $user_agent_array.

Sure enough, BBQ blocks whatever is in the array.

curl -I –user-agent “casper” https://xxxx
HTTP/1.1 403 Forbidden

Jeff, could you please explain why a cached file makes such a difference like there is no BBQ?

Thank you.

Removed all BBQ instances.

Problem solved.

Unsubscribed.

Sorry for the delay on this! Thatnks for the reminder ??

I’m not a cache expert, but would guess that it’s because the pages are static, so some scripts aren’t run (e.g., BBQ).. again, not an expert on caching but that would be my best guess. I hope it helps!