What's the best way to run Drupal and Django sites behind the same Varnish server?

Question

I have a high traffic website running with Drupal and Apache, five web servers behind a Varnish server load balancing. Let's say this site is example.com. I'm using five backends and a director like this in my default.vcl:

director balancer round-robin {
  {
   .backend = web1;
  }
  {
  .backend = web2;
  }
  {
  .backend = web3;
  }
  {
  .backend = web4;
  }
  {
  .backend = web5;
  }
}

Now I'm working on a new Django project that will be a new section of this site running on example.com/new-section.

After checking the documentation I found I can do something like this:

sub vcl_recv {
     if (req.url ~ "^/new-section/") {
         set req.backend = newbackend;
     } else {
         set req.backend = default;
     }
}

That is, using a different backend for a subdirectory /new-section under the same domain.

My question is, how do I make something like this work with my director and load balancing setup?

I'm probably going to run two or more web servers (backends) with my new Django project, each one with a mix of Gunicorn, Nginx, and a few Python packages, and would like to put all of those in their own Varnish director to load balance.

Is it possible to do use the above approach to decide which director to use?, like this:

sub vcl_recv {
     if (req.url ~ "^/new-section/") {
         set req.director = newdirector;
     } else {
         set req.director = balancer;
     }
}

All suggestions welcome.

Thanks!

Answer 1

I read Varnish's documentation a few more times and started playing with my local setup and I got this working. Varnish, at least version 2.1 which is the one I used for my tests, considers a director as a backend, so you can check set request.backend and that will set the correct director to use. So I did this:

if (req.url ~ "^/new-section") {
   set req.backend = django_balancer;
} else {
   set req.backend = balancer;
}

Where django_balancer is a new director created in a similar way as my original balancer (see details in the question). Obviously this new director has to point to web servers setup to serve the Django project.

Now Varnish will serve everything using the director for the Drupal servers except for URLs starting with /new-section, which will be served by the director for Django servers.

There's some extra setup in the default.vcl for Drupal and Django to correctly handle cookies with Varnish but that's another issue and there are already a few resources online with details on how to handle it.

And by the way, I got the hint from Varnish's wiki page about load balancing, this was the key part:

 You can send traffic to your brand new director in vcl_recv, like this:

 sub vcl_recv {
    if (req.http.host ~ "^(www.)?mysite.com$") {
      set req.backend = baz;
    }
 }

I think the use of req.backend is confusing, there should be something like req.director to make it clearer.