GET requests issued by plugin unnecessarily waste bandwidth

In general, BLC will always start with a HEAD request. However, if the server returns an error (e.g. 405 Method Not Allowed), it will try that link again and send a GET request instead.

It’s implemented this way because a significant fraction of sites either don’t support HEAD requests on some URLs, or they have buggy/incomplete HEAD support. I’ve seen several servers that would respond with a 4XX/5XX to any HEAD request while GET requests work just fine.

When the plugin falls back to sending GET requests, it will include a Range: bytes=0-2048 header to limit the amount of transferred data and reduce bandwidth usage.

There’s one more exception: the plugin will use GET requests with HTTPS links.

If you see lots of bandwidth being wasted due to this plugin, it’s probably because one of the following:

1. Some of the links use HTTPS and some don’t.
2. The server doesn’t support HEAD requests for some file types.
3. The server doesn’t support Range headers, so BLC gets more data than it actually asked for.
4. The plugin checks links far more often than it should, which makes the ~3 KB of data per request add up to large amounts. Check the plugin configuration. Also, based on your logs, how often does it request each file?
5. The links in question incorrectly an “error” status despite working fine, so BLC wastes time and bandwidth retrying them with GET requests.

Thanks for the speedy reply. I issued a “Force Recheck” a few minutes ago and the only situations in which BLC issued HEAD were for:
WP permalinks to all pages of my site
.zip files
.rtf files
All other accesses were GET requests without any prior unsuccessful HEAD. The links involved are not WP permalinks but for explicit files with names ending .jpg .xls .pdf etc).

I’ve defined BLC to check every 72 hours, and the Apache access log shows that this is indeed what happens. I realise I could reduce this frequency.

None of my links are defined as HTTPS, and the Apache access log reports all BLC accesses as having Protocol HTTP/1.1

The Apache log says that the full file size was downloaded for all GET requests, so I guess that my server doesn’t support Range headers ??

Any ideas? I can supply the Apache log and any other info you need.

Pete

Can you post a couple of example links from either group (HEAD vs GET requests)? I’ll add them to a test site and see if I can figure out why the plugin would issue a GET request without trying HEAD first. If you don’t want to make the links public, you can send them via my contact form.

More info that might give you some clues:
If I temporarily modify my .htaccess RewriteRules to reject all HEAD requests with HTTP404 (not found), then for .zip and .rtf filetypes (the only ones for which HEAD is issued), the Apache logs show:
– HEAD request being rejected HTTP404, followed by a
– GET request with HTTP200 (success), and length 2049
So your logic in http.php appears to be getting control, but not for all filetypes. Also it seems my server honours Range request headers.

In the next few minutes I will email you further information (Apache logs, screenshots, etc) plus some links you can test with.

Regards,
Pete

It looks like the problem might be related to CloudFlare. I’ve sent you more details via email.

Hello Janis,

Thanks for your in-depth analysis and conclusions that you emailed to me. It is indeed CloudFlare.

If I set Development Mode (=no caching) in CloudFlare, my Apache log shows that BLC issues only HEAD requests for all files regardless of filetype, and the bandwidth attributed to BLC is now almost zero.

I will raise this with CloudFlare and let you know the result. In the meantime, I have unset CloudFlare Development Mode and created CloudFlare Page rules to bypass cache for 2 URLs:
– *.[domain-name]/directory1/directory2/* everything in that directory
– *.[domain-name]/*.pdf all PDFs anywhere on the website

It’s a compromise between the benefits of CloudFlare caching for normal visitors and any requirement to minimise the bandwidth used by Broken Link Checker. Other websites might want to define things differently.

Let’s hope that CloudFlare come up with a decent permanent solution.

Thanks again,
Pete (ABTUK webmaster)

In my previous post (immediately above) I wrote “It is indeed CloudFlare.”

While it’s true that CloudFlare wasn’t caching the responses to GET requests (thereby being unable to respond to later HEAD requests without issuing another GET to the origin server), this happened because I was unaware that .htaccess at my site included Header set Cache-Control “…, private, …” – which basically says “proxies: do not cache”. After I located and changed that to Header set Cache-Control “…, public, …” everything worked as I hoped it would.

So it was a simple user error, aka my fault, all along. If my .htaccess settings had not prevented CloudFlare from doing its job, the “problem” described in this topic just wouldn’t have happened.

In case anyone’s interested, here’s the response I got from CloudFlare that enabled me to find the error:

Asset is a cacheable file extension, and is in our cache – GET & HEAD requests are returned without hitting the origin
Asset is not a cacheable file extension – GET & HEAD requests are proxied to the origin
Asset is not a cacheable file extension, and it is NOT in our cache – GET or HEAD request is proxied to the origin and converted to a GET, so that we can serve subsequent GET/HEAD requests from our cache.

If you think the behaviour you’re seeing is different, it may be that your origin is serving headers that are instructing us not to cache. As an example, typically servers send no-cache headers with most non-200 response codes. E.g. a 404 will be served with no-cache. In that case we’ll respect that.

Regards, Pete