Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Imagine I have an n-tier architecture in an auto-scaled cloud environment with say:

  • a load balancer in a failover pair
  • reverse proxy tier
  • web app tier
  • db tier

Each tier needs to connect to the instances in the tier below.

What are the standard ways of connecting tiers to make them resilient to failure of nodes in each tier? i.e. how does each tier get the IP addresses of each node in the tier below?

For example if all reverse proxies should route traffic to all web app nodes, how could they be set up so that they don't send traffic to dead web app nodes, and so that when new web app nodes are brought online they can send traffic to it?

  • I could run an agent that would update all the configs to all the nodes, but it seems inefficient.
  • I could put an LB pair between each tier, so the tier above only needs to connect to the load balancers, but how do I handle the problem of the LBs dying? This just seems to shunt the problem of tier A needing to know the IPs of all nodes in tier B, to all nodes in tier A needing to know the IPs of all LBs between tiers A and B.

For some applications, they can implement retry logic if they contact a node in the tier below that doesn't respond, but is there any way that some middleware could direct traffic to only live nodes in the following tier?

If I was hosting on AWS I could use an ELB between tiers, but I want to know how I could achieve the same functionality myself.

I've read (briefly) about heartbeat and keepalived - are these relevant here? What are the virtual IPs they talk about and how are they managed? Are there still single points of failure using them?

share|improve this question
up vote 1 down vote accepted

an application load balancer like haproxy does this. for example, if it detects 5xx errors from a web server, it can mark the server as failed. also, if a server fails the three-way handshake, it can mark it as failed, plus try another server while the client continues to wait.

using keepalived and heartbeat, you can have a pair of haproxy servers. if one fails, the other takes over.

i use haproxy as an example here, but pretty much any application load balancer (a.k.a. layer 4/7 load balancers) have these characteristics.

share|improve this answer
So if I put haproxy between the reverse proxy and web app tiers, would I put the IPs of both haproxy instances into the reverse proxy configs? How do I handle haproxy failing (possibly catastrophically with the instance dying and losing the IP. When another comes up it'd have a new IP)? – andy Sep 24 '12 at 16:35
you need to look at keepalived. you assign a single IP address for keepalived to use, and it assigns the right to use it to one server. if that server dies, keepalived assigns it to the other server. – longneck Sep 24 '12 at 17:36
thanks, this was the part I was missing - handling failover of LBs using a virtual IP – andy Sep 25 '12 at 15:00

Your question is How do I deal with failures?
The answer is Redundancy, or more specifically
enter image description here

  • Create a set of nodes that can do the work you need done.
    • Make sure they have separate power and networking paths to your core.
  • If you need to tolerate a failure of a single node in a set, put the set behind a load balancer as you described.
  • If you need to tolerate the failure of your load balancer, give it a partner.
    • Same caveat about separate power and networking paths.
  • If you need to tolerate the failure of multiple nodes go for N+S redundancy
    (multiple spares ready to jump in and take over).

You can do this with Amazon ELB (if you're on EC2), the pf firewall (or pfsense) with a round-robin virtual IP, or various software load-balancing tools like haproxy (which are probably the best choice as they come with some decent failure-detection capabilities, though they do require additional hardware).
There are also dedicated commercial load balancer solutions like Cisco's content switches or content switching modules if you have the cash.

Don't forget to simulate failures in your test environment to make sure things fail over the way you expect.

share|improve this answer
Yes, I will probably use internal ELBs between my tiers but I wanted to understand how they work. It was the stuff about the virtual IPs I didn't know. Thanks anyway – andy Sep 25 '12 at 15:03
@andy yeah that part is important - otherwise when you lose one of your N machines you lose 1/N of the things behind it and start having a bad day :-) – voretaq7 Sep 25 '12 at 15:17

The LB should monitor the proxy tier and automatically remove hosts (i.e. redirect traffic to surviving nodes) that are gone.

The reverse-proxy should again use an LB that monitors the web apps. The web-apps should be able to take over sessions from other nodes.

The web-apps should connect via LB to the db-servers.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.