Is there a way to have Varnish 4 ignore any hit-for-pass objects in the cache if all backends have been marked as "sick"?
Here's the failure scenario I'm trying to improve:
- At first backends are all healthy and behaving nicely, returning valid content with status of 200. Varnish is caching these pages and serving them according to their TTL.
- Something breaks, say a database query starts taking longer than usual. The backends start returning pages slowly. Then eventually requests start to time out altogether, and the backends start returning "Internal Server Error" (status 500).
- Varnish sees these responses and marks them as hit-for-pass with a TTL of 120s, according to the default
vcl_backend_response
. - Finally the health checks kick in, and Varnish eventually marks all the backends sick.
- Now as more requests come in, Varnish sees a hit-for-pass object in the cache, and decides it needs to do a backend fetch. Except all the backends are sick, so that results in a 503 "Backend fetch failed."
These 503 responses continue for up to 2 minutes, depending on the timing of the first non-cacheable response (marked as hit-for-pass) and when the backends all get marked sick. - After the hit-for-pass objects have expired from the cache (120s), Varnish starts treating those requests as regular hits again, and serves up the cached 200-status pages in grace mode (according to the default
vcl_hit
-- "if (obj.ttl + obj.grace > 0) ....
)
One workaround I've come up with is to shorten the TTL of hit-for-pass objects if they came from a status 500 response:
sub vcl_backend_response {
if (beresp.ttl <= 0s && beresp.status == 500) {
set beresp.ttl = 10s;
set beresp.uncacheable = true;
# return inside this if statement to allow builtin vcl_backend_response to run
return (deliver);
}
}
Other possibilities are adjusting the interval and threshold of the health checks, or coming up with a better health check.
But aside from that, is there a way to explicitly tell Varnish "yeah, you have a hit-for-pass, but look -- all the backends are sick! Don't bother, switch to grace mode now."