[W3 Total Cache] Does HEAD request cause caching?

  • Some of our blogs that use W3 Total Cache are being hammered by search engines that throw 5 to 10 HTTP HEAD requests at the site a second, causing small cloud virtual servers to wedge and get very slow until the backlog is processed. Each HEAD seems to cause WordPress to build the entire page (regardless of whether it’s cached? can’t tell…), but then a subsequent GET seems to take just as much time as the original HEAD took.

    First question: Does (or “could”) a HEAD request simply look at the cached copy of an existing cached page and return the appropriate info rather than building the entire page? The .htaccess rewrite rules that W3TC inserts seem as they are checking the cache, but when I observe the time it takes to return a HEAD, it suggests that perhaps the rule isn’t working.

    Second question: Does HEAD cause W3TC to create a new cached copy (if the old one is expired) that a subsequent GET will use?

    If this question is hard to answer, I can look at the code myself to try to figure it out, but I have seen comments elsewhere that WP itself does create an entire page in response to a HEAD (which it probably should) and that searchbots consequently clobber the performance of WP sites when hammering WP sites that don’t have caches in place.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Contributor Frederick Townes

    (@fredericktownes)

    Can you fill out a bug submission form from the support tab of the plugin so I can see your settings?

    Thread Starter jimsky7

    (@jimsky7)

    I’ll do that later, if necessary. I think there’s nothing that you’d have to do… I just did some testing using a direct (telnet) connection to the web server and found that HEAD does (as we would expect) cause the creation of a cached copy, and a subsequent HEAD or GET will retrieve the cached copy. So it’s working as expected. The W3TC .htaccess rules are and were correctly written when I first looked at them.

    Our problem is really with searchbots that hit us 5 to 10 times a second with HEAD requests, presumably thinking that HEAD is more economical than GET. It’s only more economical -for them- not for us, especially if they’re looking at old WP posts that are not in the cache, which might take a couple of seconds (each) to recreate (our site is very “backoffice heavy”).

    I’m working with the particular searchbot owner to try to get them to throttle their bot a bit, but it’s hard for them to even think about throttling because it basically feeds iPads through their app and on an iPad the usual human behavior is to flick-flick-flick right through page after page, expecting them to come up instantly. Not the same expectations as a computer-based site visitor who has the understanding that they’re on a big old heavy computer with a limited-bandwidth Internet connection. The iPadder expects it all to be instant. The world is going to be more and more like this … and W3TC will be more and more needed!

    Plugin Contributor Frederick Townes

    (@fredericktownes)

    Thanks for your input.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘[W3 Total Cache] Does HEAD request cause caching?’ is closed to new replies.