1

Is there a way to have Varnish 4 ignore any hit-for-pass objects in the cache if all backends have been marked as "sick"?

Here's the failure scenario I'm trying to improve:

  • At first backends are all healthy and behaving nicely, returning valid content with status of 200. Varnish is caching these pages and serving them according to their TTL.
  • Something breaks, say a database query starts taking longer than usual. The backends start returning pages slowly. Then eventually requests start to time out altogether, and the backends start returning "Internal Server Error" (status 500).
  • Varnish sees these responses and marks them as hit-for-pass with a TTL of 120s, according to the default vcl_backend_response.
  • Finally the health checks kick in, and Varnish eventually marks all the backends sick.
  • Now as more requests come in, Varnish sees a hit-for-pass object in the cache, and decides it needs to do a backend fetch. Except all the backends are sick, so that results in a 503 "Backend fetch failed."
    These 503 responses continue for up to 2 minutes, depending on the timing of the first non-cacheable response (marked as hit-for-pass) and when the backends all get marked sick.
  • After the hit-for-pass objects have expired from the cache (120s), Varnish starts treating those requests as regular hits again, and serves up the cached 200-status pages in grace mode (according to the default vcl_hit -- "if (obj.ttl + obj.grace > 0) ....)

One workaround I've come up with is to shorten the TTL of hit-for-pass objects if they came from a status 500 response:

sub vcl_backend_response {
  if (beresp.ttl <= 0s && beresp.status == 500) {
    set beresp.ttl = 10s;
    set beresp.uncacheable = true;
    # return inside this if statement to allow builtin vcl_backend_response to run
    return (deliver);
  }
}

Other possibilities are adjusting the interval and threshold of the health checks, or coming up with a better health check.

But aside from that, is there a way to explicitly tell Varnish "yeah, you have a hit-for-pass, but look -- all the backends are sick! Don't bother, switch to grace mode now."

2

If you want that behaviour, you should avoid creating hit-for-pass objects when you get 500 responses from the backend.

Let's assume that (1) you're always requesting the same cacheable URL / object X; and (2) object X is currently stored in the Varnish storage with a TTL of 1h and a grace of 24h. Let's now assume the following timeline:

  • t=0: some client requests X to Varnish. The object is already cached and fresh, so Varnish returns it to the client. No backend requests.
  • t=1: the backend gets broken and it replies with 500 to all requests from now on.
  • t=3601: some client requests X to Varnish. The object is cached but stalled, so Varnish returns it to the client and triggers a background backend request to update the cached object X.
  • t=3602: the background backend request gets a 500 response. A hit-for-pass object is stored with a TTL of 120s. This object overwrites the stalled object X.
  • t=3603: some client requests X to Varnish. A hit-for-pass object is found and a backend request executed. A 500 response is sent to the client side.
  • t=4000: some client requests X to Varnish. The object X is not in cache anymore and the backend was marked as sick some time ago. A 503 response will be sent to the client side: Varnish cannot contact with the backend (it's marked as sick) and Varnish cannot return grace content (it was overwritten by the hit-for-pass object on t=3602).

The solution here would be avoid creating hit-for-pass objects when you get 500 responses from the backend. Instead simply abandon the request. That way on t=3603 and t=4000 you'll send a stalled copy of the object to the client side.

  • 1
    Thanks, Carlos Abalde! The keyword for me here was "abandon". Adding this to my vcl_backend_response seems to do what I want: if (beresp.status == 500) { return(abandon); } -- The 500s are never served to the client, whether the backends are healthy or sick. – larcher Sep 2 '15 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.