1

I have 2 varnish 3.0.4 servers on edge of an Apache cluster (fallback director). On one of my sites, users log in to download static content such as pdf, images (jpg,jpeg,png, gif), xls/xlsx etc. All but the exel files are told to be cached.

When downloading an Excel file everything works as expected. When trying to download a cached object the behavior is different. Even if the site is correctly displayed (all images appears) the user is disconnected and the file can't be fetched (404 from varnishlog). No issues browsing the site when logged in.

Using host file to point directly on Apache and the files download perfectly.

As a temporary solution told varnish not to cache for this site and banned all content for it.

Thank you.

EDIT : full vcl and varnishlog.

Thanks for your help. Sadly it's still not working. Here is my full vlc

backend srv1 {
        .host = "srv1";
        .port = "80";
        .first_byte_timeout = 120s;
        .probe = {
                .url = "/";
                .interval = 5s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

backend srv2 {
        .host = "srv2";
        .port = "80";
        .first_byte_timeout = 120s;
        .probe = {
                .url = "/";
                .interval = 5s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

backend srv3 {
        .host = "srv3";
        .port = "80";
        .first_byte_timeout = 120s;
        .probe = {
                .url = "/";
                .interval = 5s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

director cluster fallback {
        { .backend = srv1; }
        { .backend = srv2; }
        { .backend = srv3; }
}

sub vcl_recv {
        set req.backend = cluster;

        if (req.url ~ "\.(htm|html|xml|jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|pdf|flv|swf)$") {
        return (lookup);
        }

        if (req.http.Authorization || req.http.Cookie) {
        return (pass);
        }

        # Normalize encoding/compression
        if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip";     }
        elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; }
        else { remove req.http.Accept-Encoding; }
        }

        if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
                error 405 "Not allowed.";
        }
        return (lookup);
        }
}

acl purge {
        "localhost";
}

#
sub vcl_hit {
        if (req.request == "PURGE") {
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}
#
sub vcl_miss {
        if (req.request == "PURGE") {
                error 404 "Not in cache.";
        }
}

sub vcl_fetch {
if (req.url ~ "\.(htm|html|xml|jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|pdf|flv|swf)$") {
        return (deliver);
        }
}
sub vcl_deliver {
        if (obj.hits > 0)
                { set resp.http.X-Cache = "HIT from Varnish"; }
        else
                { set resp.http.X-Cache = "MISS from Varnish"; }
}

And here is the full varnishlog trace when trying to download a file :

   50 BackendOpen  b srv1 10.24.1.11 40049 10.24.1.95 80
   50 TxRequest    b GET
   50 TxURL        b /de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf
   50 TxProtocol   b HTTP/1.1
   50 TxHeader     b Via: 1.1 ClientSiteProxy:3128 (squid/2.7.STABLE4), 1.0 proxy-1_5 (squid/3.1.19), 1.1 cache-3:80, 1.0 SPRX0008
   50 TxHeader     b Host: mysite.domain.com
   50 TxHeader     b Referer: http://mysite.domain.com/de-ge/website-material/vet_basic/produktdatenblaetter
   50 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
   50 TxHeader     b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
   50 TxHeader     b Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
   50 TxHeader     b X-Teacup: eFDRDF7Vx0755xk=
   50 TxHeader     b X-Forwarded-For: CLIENT_IP, PUBLIC_IP
   50 TxHeader     b X-Varnish: 1630421623
   50 TxHeader     b Accept-Encoding: gzip
   50 RxProtocol   b HTTP/1.1
   50 RxStatus     b 404
   50 RxResponse   b Not Found
   50 RxHeader     b Date: Wed, 08 Jan 2014 10:02:24 GMT
   50 RxHeader     b Server: Apache
   50 RxHeader     b Expires: Thu, 19 Nov 1981 08:52:00 GMT
   50 RxHeader     b Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, public, no-transform
   50 RxHeader     b Pragma: no-cache
   50 RxHeader     b X-Powered-By: eZ Publish
   50 RxHeader     b Served-by: mysite.domain.com
   50 RxHeader     b Content-language: de-DE
   50 RxHeader     b Set-Cookie: PHPSESSID=ob5a4jaic2ubpvala2qacksl17; path=/
   50 RxHeader     b Content-Length: 7030
   50 RxHeader     b Connection: close
   50 RxHeader     b Content-Type: text/html; charset=utf-8
   50 Fetch_Body   b 4(length) cls 0 mklen 1
   50 Length       b 7030
   50 BackendClose b srv1
   24 ReqStart     c REVERSE_PROXY_IP(NOT VARNISH) 57364 1630421623
   24 RxRequest    c GET
   24 RxURL        c /de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf
   24 RxProtocol   c HTTP/1.1
   24 RxHeader     c Via: 1.1 ClientSiteProxy:3128 (squid/2.7.STABLE4), 1.0 proxy-1_5 (squid/3.1.19), 1.1 cache-3:80, 1.0 SPRX0008
   24 RxHeader     c Host: mysite.domain.com
   24 RxHeader     c Cookie: ISAWPLB{515B9AC4-0CEE-4B2B-B32F-DB0AD870EC2F}={DF5E7FFB-0F55-466C-9DDC-2983CFDF75AE}; PHPSESSID=nfcqe0tcq7gqiu253ef3o9svu4; is_logged_in=true
   24 RxHeader     c Referer: http://mysite.domain.com/de-ge/website-material/vet_basic/produktdatenblaetter
   24 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
   24 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
   24 RxHeader     c Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
   24 RxHeader     c X-Teacup: eFDRDF7Vx0755xk=
   24 RxHeader     c X-Forwarded-For: CLIENT_IP, PUBLIC_IP
   24 RxHeader     c Cache-Control: max-age=259200
   24 RxHeader     c Connection: Keep-Alive
   24 VCL_call     c recv lookup
   24 VCL_call     c hash
   24 Hash         c /de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf
   24 Hash         c mysite.domain.com
   24 VCL_return   c hash
   24 VCL_call     c miss fetch
   24 Backend      c 50 cluster srv1
   24 TTL          c 1630421623 RFC 0 -1 -1 1389175345 0 1389175344 375007920 0
   24 VCL_call     c fetch deliver
   24 ObjProtocol  c HTTP/1.1
   24 ObjResponse  c Not Found
   24 ObjHeader    c Date: Wed, 08 Jan 2014 10:02:24 GMT
   24 ObjHeader    c Server: Apache
   24 ObjHeader    c Expires: Thu, 19 Nov 1981 08:52:00 GMT
   24 ObjHeader    c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, public, no-transform
   24 ObjHeader    c Pragma: no-cache
   24 ObjHeader    c X-Powered-By: eZ Publish
   24 ObjHeader    c Served-by: mysite.domain.com
   24 ObjHeader    c Content-language: de-DE
   24 ObjHeader    c Set-Cookie: PHPSESSID=ob5a4jaic2ubpvala2qacksl17; path=/
   24 ObjHeader    c Content-Type: text/html; charset=utf-8
   24 VCL_call     c deliver deliver
   24 TxProtocol   c HTTP/1.1
   24 TxStatus     c 404
   24 TxResponse   c Not Found
   24 TxHeader     c Server: Apache
   24 TxHeader     c Expires: Thu, 19 Nov 1981 08:52:00 GMT
   24 TxHeader     c Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0, public, no-transform
   24 TxHeader     c Pragma: no-cache
   24 TxHeader     c X-Powered-By: eZ Publish
   24 TxHeader     c Served-by: mysite.domain.com
   24 TxHeader     c Content-language: de-DE
   24 TxHeader     c Set-Cookie: PHPSESSID=ob5a4jaic2ubpvala2qacksl17; path=/
   24 TxHeader     c Content-Type: text/html; charset=utf-8
   24 TxHeader     c Content-Length: 7030
   24 TxHeader     c Accept-Ranges: bytes
   24 TxHeader     c Date: Wed, 08 Jan 2014 10:02:25 GMT
   24 TxHeader     c X-Varnish: 1630421623
   24 TxHeader     c Age: 0
   24 TxHeader     c Via: 1.1 varnish
   24 TxHeader     c Connection: keep-alive
   24 TxHeader     c X-Cache: MISS from Varnish
   24 Length       c 7030
   24 ReqEnd       c 1630421623 1389175344.435734510 1389175345.230693817 2.376755714 0.794904470 0.000054836

EDIT2 : I can't use curl because I need to be logged in the application. But I'm 200% sure the ressource is available since when I use my host file to hit the web server I can download the files.

Here's the output of Chrome dev tools :

    Request URL:http://mysite.domain.com/de-ge/content/download/1004658/18185577/version/2/file/RC_PB_CHN_Junior.pdf
    Request Method:GET
    Status Code:200 OK

    Request Headers
    Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
    Accept-Encoding:gzip,deflate,sdch
    Accept-Language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
    Connection:keep-alive
    Cookie:__utma=65825073.183772452.1389198084.1389262355.1389265715.3; __utmc=65825073; __utmz=65825073.1389198084.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=3jr13d7cqmn0kv86iho4n030i0; is_logged_in=true
    Host:mysite.domain.com
    Referer:http://mysite.domain.com/de-ge/website-material/retail_basic/produktdatenblaetter
    User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36


  Response Headers
    Accept-Ranges:bytes
    Cache-Control:public
    Connection:close
    Content-Disposition:attachment
    Content-language:de-DE
    Content-Transfer-Encoding:binary
    Content-Type:application/pdf
    Date:Thu, 09 Jan 2014 14:36:30 GMT
    Expires:Thu, 09 Jan 2014 14:46:30 GMT
    Pragma:
    Served-by:mysite.domain.com
    Server:Apache
    Transfer-Encoding:chunked
    X-Powered-By:eZ Publish

EDIT 3 : I can't get it to work. I and I really don't understand why all images are correctly diplayed in the user zone whereas downloading (litterally, with a download prompt) images or another resource failed.

Thank you.

1 Answer 1

1

I suspect that your backend is overwriting the cookie, so I'll try changing vcl_fetch to:

sub vcl_fetch {
  if (req.url ~ "\.(htm|html|xml|jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|pdf|flv|swf)$") {
    unset beresp.http.set-cookie;
    return (deliver);
  }
}

Anyway it's hard to tell without having a varnishlog trace and your full vcl config.


After seeing your full VCL and the varnishlog trace, its clear that the backend is answering a 404 status.

So please try to issue a curl call to the backend in order to inspect the response, like:

curl -I -H "Host: mysite.domain.com" srv1/de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf

After seeing your EDIT2:

If you need to be logged in to get the resource and logins are cookie based, it's failing because you're stripping the cookies on:

if (req.url ~ "\.(htm|html|xml|jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|pdf|flv|swf)$") {
  return (lookup);
}

BTW: You can issue curl calls with cookies:

curl -I -b "PHPSESSID=3jr13d7cqmn0kv86iho4n030i0; is_logged_in=true" -H "Host: mysite.domain.com" srv1/de-ge/content/download/992186/17973237/version/1/file/PB_CAN_EDUC_Trockennahrung.pdf

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .