Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've setup a Varnish cache mirror to sit in front of a file server, but it seems to be endlessly re-downloading data from my file server.

There's about 100GB of data in total, but so far Varnish has downloaded 800GB from my file server.

I'm using the default VCL file that comes with Varnish and the response headers for files served by the file server are similar to the following:

HTTP/1.1 200 OK
Cache-Control: max-age=290304000, public
Content-Type: image/jpeg
Expires: Wed, 29 Dec 2010 21:38:33 GMT
Server: Microsoft-IIS/7.0
E-Tag: "8b4723296ab697530768f18b1378b269"
Content-Disposition: inline; filename=image046.jpg;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 23 Dec 2010 05:38:33 GMT
Content-Length: 100592

I'm starting varnishd with the following options:

varnish/sbin/varnishd -a 0.0.0.0:80 -f varnish/etc/varnish/default.vcl -s file,varnish/var/lib/varnish/varnish_storage.bin,100G
share|improve this question
    
Can you paste in your default.vcl (I can't find an unmodified one on the internets)? –  Tom O'Connor Dec 25 '10 at 8:54

1 Answer 1

up vote 3 down vote accepted
sub vcl_recv {
    if (req.url ~ "\.(cur|gif|jpg|jpeg|swf|css|js|flv|mp3|mp4|pdf|ico|png)(\?.*|)$") {
      unset req.http.cookie;
      unset req.http.Accept-Encoding;
      unset req.http.Vary;
    }
}

If your requests contain cookies, they will be passed to the backend by default. Removing the encoding and vary will normalize things a little more so that you don't need to maintain a few copies of your images in cache. Also, I think 100G might be a bit excessive for your backing store, you might consider something closer to the amount of RAM you have in the machine.

share|improve this answer
    
How do you know his Cache server doesn't have 100 GB of ram? It's quite easy to get 100+GB of ram in a server now. –  Tom O'Connor Dec 25 '10 at 8:55
1  
Thanks, the cookies were what was making Varnish not cache. I didn't think about it, but it turns out that because the Varnish mirror was on a subdomain of a site setting cookies for the whole domain, they were affecting requests to Varnish too. –  Charlie Somerville Dec 25 '10 at 9:23
    
Funny that aaa didn't receive the bounty for answering the question. It really is a boys club around here. –  aaa Jan 14 '11 at 11:59
1  
@aaa - I need to make you clear on a few points sorry. This is a question and answer site not a chat board, can I ask you to read our FAQ please as you shouldn't comment in the form of an answer (I've converted your answer into the above comment). Secondly I don't understand your accusation, if you have a query regarding bounty management then you can contact us without being rude. Thirdly you appear to have used two different OpenID's to create two different accounts, each called 'aaa', this may have caused the bounty problem - would you like them merged? –  Chopper3 Jan 14 '11 at 12:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.