Varnish not caching (MISS)

Question

We have a varnish server in front of a few websites running Nginx/CraftCMS. All the websites cache as expected, except one.

Over the past week we've read every tutorial, checked every answer and must have made 100 config tweaks. Currently we're unsetting almost every header but still can't put a finger on it.

The site doesn't have user specific content and cookies can be completely ignored.

BerespStatus   200
-   BerespReason   OK
-   BerespHeader   Server: nginx/1.14.0 (Ubuntu)
-   BerespHeader   Date: Tue, 20 Aug 2019 11:14:35 GMT
-   BerespHeader   Content-Type: text/html; charset=utf-8
-   BerespHeader   Transfer-Encoding: chunked
-   BerespHeader   Connection: keep-alive
-   BerespHeader   Vary: Accept-Encoding
-   BerespHeader   Set-Cookie: CraftSessionId=XXX; path=/; secure; HttpOnly
-   BerespHeader   Expires: Thu, 19 Nov 1981 08:52:00 GMT
-   BerespHeader   X-Powered-By: Craft CMS
-   BerespHeader   Cache-Control: no-cache
-   BerespHeader   Pragma: no-cache
-   BerespHeader   charset: utf-8
-   BerespHeader   Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
-   BerespHeader   X-XSS-Protection: 1; mode=block
-   BerespHeader   X-Content-Type-Options: nosniff
-   BerespHeader   Access-Control-Allow-Origin: *
-   BerespHeader   X-UA-Compatible: IE=Edge
-   BerespHeader   Content-Encoding: gzip
-   TTL            RFC 0 10 0 1566299675 1566299675 1566299675 375007920 0
-   VCL_call       BACKEND_RESPONSE
-   BerespUnset    Set-Cookie: CraftSessionId=XXX; path=/; secure; HttpOnly
-   BerespUnset    Cache-Control: no-cache
-   BerespUnset    Pragma: no-cache
-   BerespUnset    Vary: Accept-Encoding
-   BerespUnset    Date: Tue, 20 Aug 2019 11:14:35 GMT
-   BerespUnset    Expires: Thu, 19 Nov 1981 08:52:00 GMT
-   TTL            VCL 120 10 0 1566299675
-   VCL_return     deliver
-   BerespHeader   Vary: Accept-Encoding
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      200
-   ObjReason      OK
-   ObjHeader      Server: nginx/1.14.0 (Ubuntu)
-   ObjHeader      Content-Type: text/html; charset=utf-8
-   ObjHeader      X-Powered-By: Craft CMS
-   ObjHeader      charset: utf-8
-   ObjHeader      Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
-   ObjHeader      X-XSS-Protection: 1; mode=block
-   ObjHeader      X-Content-Type-Options: nosniff
-   ObjHeader      Access-Control-Allow-Origin: *
-   ObjHeader      X-UA-Compatible: IE=Edge
-   ObjHeader      Content-Encoding: gzip
-   ObjHeader      Vary: Accept-Encoding
-   Fetch_Body     2 chunked stream
-   Gzip           u F - 21276 75774 80 80 170140

If anyone had a suggestion we'd be enormously grateful!

Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Age: 0
charset: utf-8
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Wed, 21 Aug 2019 07:54:10 GMT
Server: nginx/1.14.0 (Ubuntu)
Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
Transfer-Encoding: chunked
Vary: Accept-Encoding
Via: 1.1 varnish (Varnish/5.2)
x-Cache: MISS
X-Content-Type-Options: nosniff
X-Powered-By: Craft CMS
X-UA-Compatible: IE=Edge
X-Varnish: 858580
X-XSS-Protection: 1; mode=block

Answer 1

Your website is setting a cookie (at least so from your varnishlog entry):

-   BerespHeader   Set-Cookie: CraftSessionId=XXX; path=/; secure; HttpOnly

When Varnish sees your backend setting a cookie, it will flag the response as uncacheable, in the built-in VCL.

The built-in VCL fires "after" you own, and you can override/skip its execution by returning early from your VCL.

Evaluate why your app is sending the cookie. Since you expect the page to be cached, then this is likely neither a login page nor somewhere you really need to set a cookie.

The common workaround would be to unset the cookie using VCL. This alone won't be enough though, because PHP sets negative Cache-Control while a cookie is being set. So you will also need to adjust ttl and specify how long those responses are to be cached.

The following config unsets all backend-emitted cookies, from any page except /login/.

sub vcl_backend_response {
    if (beresp.http.Set-Cookie && bereq.url !~ "^/login/") {
        unset beresp.http.Set-Cookie;
        unset beresp.http.Cache-Control;
        # cache for 1 hr
        set beresp.ttl = 3600;
    }
}

All that said, it is best to simply adjust your app to only send Set-Cookie when it's absolutely required, e.g. when POST-ing to your login page, and only there.