Automatic reverse/caching proxy configuration update?

Question

I've never done something so ambitious before so please let me know if this is simply a bad idea, and alternate solutions I should consider.

So I have three servers (and more later), all hosting sites with the same TLD. As it stands, I can't link a subdomain and a server. (There are more than one server because of different subscription levels and expected performance, etc.)

My ideal scenario is this: All web traffic hits a Varnish cache, and Varnish caches it from whichever server is hosting the site. I can use Apache or Nginx for the web servers, whichever is more convenient.

There will always be new sites being created on the web nodes, so I want my Varnish cache to know which site is where without having to manually enter them. (All the servers are on Debian.)

I have been considering Puppetmaster for this, but I can't find any documentation (maybe I'm searching for the wrong thing?). I've also been considering a custom daemon, but that seems like reinventing the wheel.

I could use Nginx as a caching proxy if it's more convenient, Varnish just seems more popular for this use case.

Thanks

Answer 1

On Varnish, the only pitfall I can foresee is when adding new servers (backends)... Varnish is "host insensitive" so given that all servers can process requests for all hosts you won't have any problems.

You can also make use of DNS director [1] (if you don't need healthchecks) to add a network range (cidr notation) of backend servers:

director directorname dns {
    .list = {
            .port = "80";
            .connect_timeout = 0.4s;
            "192.168.15.0"/3;
    }
    .ttl = 5m;
}

Probably using Puppet for this kind of setup is a good idea, but I think you'll need a custom module for that.

[1] https://www.varnish-cache.org/docs/3.0/reference/vcl.html#the-dns-director