I'm looking into setting up a system something like this:
/ [Load Balancer] \ /\/\/\/\/\/\/\ [LAN1] - ----- | Server Pool | \ [Load Balancer] / \/\/\/\/\/\/\/ | [Redundant ] [Load Balancer]
In this setup, there would be an active-active configuration of load balancers (to ensure no Single Point of Failure) and a variable number of web servers in the pool. Servers can be set up and running at any time (through network booting) and so the Balancers should be able to add those servers to the backend without having to restart the service. As you can tell I'm pretty clear on what the problem is, but I'm not sure how to implement it. I was wondering:
1) In this situation, which load balancer would be best suited? The ones I've looked into so far are HAProxy, Linux Virtual Server, Ultramonkey and XR crossroads. LVS can support adding and removing servers without restart through ipvsadm, and crossroads has an (albeit hacky) way to support it. I haven't found any notes about this on HAProxy so I don't know if it's possible.
1b) Is there a "best" OS for a particular LB? Obviously Linux is going to be needed, but is there a flavour which is best suited to this?
2) Is it actually possible to set up an Active-Active configuration? I've seen lots of people saying they've done it, but no notes on how to do it.
3) Would it be better to connect each balancer to its own pool of servers (IE a different interface and physical LAN)? Would this make a difference?
Any ideas/ tips/ kicks to the brain are accepted. I've been doing a lot of reading and it's hard to know what is/ isn't worth looking into.