Is it safe to use proxy_ignore_headers Vary?

Question

I use a Nginx server that acts like a cache/reverse proxy for my Apache origin server.

I was strugling to debug high rate of cache MISS when I found that it is because the response header Vary: Accept-Encoding, different www clients with different Accept-Encoding were not getting HIT, even thou same exact url was previously requested by another client.

This can be fixed by adding nginx config:

proxy_ignore_headers Vary;

After that, I get HIT cache replies.

a) Does this disable the different cached page variations just between my cache server and origin server ? (e.g it is safe to use)

b) Or does it cache one version (gzipped or not-gzipped file) for all final clients regardless of client's encoding/decoding capabilities ? I am thinking this could result in a non-gzip capable client receiving a gziped file ?

Even thou the cache is improved / fixed by the above directive, the final client still receives the Vary: Accept-Encoding, it can be hidden with proxy_hide_header Vary, but should I ?

Answer 1

I did some investigation on this subject. I think I found some valuable info on GitHub discussion in Yichun Zhang (author of OpenResty nginx fork) reply:

I'd suggest the following options:

1. Cache both compressed and uncompressed responses, and add (a canonicalized) value of the Accept-Encoding to your cache key. So that clients which do not expect gzip'd responses result in a cache miss (and trigger caching the uncompressed version).
2. Cache compressed responses and use the standard ngx_gunzip module to uncompress the response body for those clients which do not accept gzip encoding:. See http://nginx.org/en/docs/http/ngx_http_gunzip_module.html But you need to be careful about nginx output filter order (between ngx_srcache's filter and ngx_gunzip's filter).
3. Only cache uncompressed content and rely on the ngx_gzip module to compress the data for those clients expecting gzip. Set proxy_set_header Accept-Encoding ''; for example, if the proxy module is used as the content handler.

So, it looks like it isn't safe, and you should cache both (compressed and non-compressed) replies.

But what happens if you receive several requests with the following Accept-Encoding headers:

• Accept-Encoding: gzip
• Accept-Encoding: gzip, br
• Accept-Encoding: gzip;q=0.8, br;q=0.9
• Accept-Encoding: gzip, deflate, br

Does it mean that each of responses will be cached separately? I don't know yet, this needs more extensive testing that I already did. What I'm found so far:

• nginx does not respond to Accept-Encoding: deflate with deflated response (which is only a little different from gzipped one).
• nginx does not respect the weight qvalue part of header. It will respond with compressed response on request with Accept-Encoding: identity;q=0.8, gzip;q=0.5 header. If you have a ngx_brotly module installed, nginx will always respond with brotly-compressed response if your Accept-Encoding request header contains a br compression method no matter what weights are specified for compression methods.

Yichun Zhang mentions about adding (a canonicalized) value of the Accept-Encoding to your cache key. This can be something like

map $http_accept_encoding $encoding_key {
    ~[\s:,]br(?:[\s,\;]|$)       2; # brotli
    ~[\s:,]gzip(?:[\s,\;]|$)     1; # gzip
    default                      0; # uncompressed
}

If you don't use ngx_brotly module, things can be simplified:

if ($http_accept_encoding ~ "[\s:,]gzip(?:[\s,;]|$)") {
    set $encoding_key 1;
}

After that adding $encoding_key variable to proxy_cache_key expression can do the trick, but as I already said, this needs more extensive testing.