Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a slow web app that I've placed Varnish in front of. All of the pages are static (they don't vary for a different user), but they need to be updated every 5 minutes so they contain recent data.

I have a simple script (wget --mirror) that crawls the entire website every 15 minutes. Each crawl takes about 5 minutes. The point of the crawl is to update every page in the Varnish cache so that a user never has to wait for the page to generate (since all pages have been generated recently thanks to the spider).

The timeline looks like this:

  • 00:00:00: Cache flushed
  • 00:00:00: Spider starts crawling to update cache with new pages
  • 00:05:00: Spider finishes crawling, all pages are updated until 00:15:00

A request that comes in between 0:00:00 and 0:05:00 might hit a page that hasn't been updated yet, and will be forced to wait a few seconds for a response. This isn't acceptable.

What I'd like to do is, perhaps using some VCL magic, always foward requests from the spider to the backend, but still store the response in the cache. This way, a user will never have to wait for a page to generate since there is no 5-minute window in which parts of the cache are empty (except perhaps at server startup).

How can I do this?

share|improve this question
up vote 9 down vote accepted

req.hash_always_miss should do the trick.

Don't do a full cache flush at the start of the spider run. Instead, just set the spider to work - and in your vcl_recv, set the spider's requests to always miss the cache lookup; they'll fetch a new copy from the backend.

acl spider {
  /* or whereever the spider comes from */

sub vcl_recv {
  if (client.ip ~ spider) {
    set req.hash_always_miss = true;
  /* ... and continue as normal with the rest of the config */

While that's happening and until the new response is in the cache, clients will continue to seamlessly get the older cache served to them (as long as it's still within its TTL).

share|improve this answer
Thanks for the answer; I was mid-way through typing the solution I came up with when I saw your answer. Yours is clearly superior than mine (and I am already using yours). Can you take a look at my answer and let me know any problems with it just so that I can learn? – Tom Marthenal Sep 8 '12 at 6:51

Shane's answer above is better than this one. This is an alternative solution which is more complicated and has additional problems. Please upvote Shane's response, not this one. I am just showing another method of solving the problem.

My initial thought was to return (pass); in vcl_recv and then, after the request has been fetched, in vcl_fetch, somehow instruct Varnish that it should cache the object, even thought it was specifically passed earlier.

It turns out this isn't possible:

If you chose to pass the request in an earlier VCL function (e.g.: vcl_recv), you will still execute the logic of vcl_fetch, but the object will not enter the cache even if you supply a cache time.

So the next-best thing is trigger a lookup just like a normal request, but make sure it always fails. There's no way to influence the lookup process, so it's always going to hit (assuming it is cached; if it's not, then it's going to miss and store anyway). But we can influence vcl_hit:

sub vcl_hit {
    # is this our spider?
    if (req.http.user-agent ~ "Wget" && client.ip ~ spider) {
        # it's the spider, so purge the existing object
        set obj.ttl = 0s;
        return (restart);

    return (deliver);

We can't force it not to use the cache, but we can purge that object from the cache and restart the entire process. Now it goes back to the beginning, at vcl_recv, where it eventually does another lookup. Since we purged the object we're trying to update already, it will miss, then fetch the data and update the cache.

A little complicated, but it works. The only window for a user getting stuck between a purge and the response being stored is the time for the single request to process. Not perfect, but pretty good.

share|improve this answer
Yup, that should work! One extra thing that you can do that ought to prevent a user from getting a slow response due to hitting the gap between the delete and re-cache would be to set a short grace mode timer; since the restart will have fired, Varnish will have a request already in process for the content. Grace mode kicks in and serves the stale content past its TTL, instead of waiting for the spider's request to finish. – Shane Madden Sep 8 '12 at 7:07
@ShaneMadden thanks for the note about grace mode—I'm still trying to get the hang of Varnish and hadn't heard of that yet. Your solution is still clearly better, but I appreciate the opportunity to learn more about Varnish. – Tom Marthenal Sep 8 '12 at 7:34

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.