There is an older security filter that was not well enough defined that did block malformed queries by accident instead of just hacking attempts. The old security filter looked like this:
RewriteCond %{THE_REQUEST} \?\ HTTP/ [NC,OR]
RewriteCond %{THE_REQUEST} \/\*\ HTTP/ [NC,OR]
And is/was automatically replaced with this new security filter during a BPS upgrade unless something interfered with that automatic filter rule change:
RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)HTTP(:/|/) [NC,OR]
Other possible filters that may block malformed / bad / dangerous queries could be these security filters below. It just depends on how malformed/bad/dangerous the query is. You can try commenting out these filters to see which one is causing the issue.
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=https:// [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
RewriteCond %{QUERY_STRING} http\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]