I offer my solution to the following problem, and ask you networking and server admin professionals to validate it or poke holes in it. I am interested in any obvious attack vectors or scalability issues you may see. Thanks!


  • HTTPS support, handled by each application server independently
  • near-linear horizontal scalability
  • bandwidth distributed across servers (response data does not all return through the LBs or proxies)
  • something like failover for application servers and load balancers
  • client-server affinity
  • Linux-friendly (solution not closed source)
  • bootstrap-friendly! (i.e. low initial cost)

The Scheme:

            PUBLIC NETWORK
      |      |        |     |
      v      v        v     v
    +---+  +---+     +--+  +--+
    |LB1|  |LB2| ... |S1|  |S2| ...
    +---+  +---+     +--+  +--+

Redundant load balancers (LB*, via something like DNS RR, or just failover): their only purpose is to offer clients the URI to some application server instance, which the client would then use perpetually for its requests. The distribution would be random or round robin, initially.

Application Server instances (S*) each independently handle requests directly from clients.

Stateless architecture lets individual servers go down. Clients request a new server from the load balancers if their assigned server fails.

New application servers could spin up, register with the load balancers, and be assigned to clients very quickly. All S* would have a subdomain DNS entry to share a wildcard certificate.

A naive implementation could be done entirely on one server with zero redundancy, and delegate responsibilities to expand as needed.

Immediate Concerns

Firewall and DDoS protection would have to be managed at each server, instead of centrally like you have with load balancing reverse proxies. Centralized configuration management is as far as I've thought into this.

This scheme does not take advantage of geographic location or server response time, as something like Anycast DNS would. It's a conscious trade-off for greater likelihood of server affinity, and can possibly be shoehorned in later.


On a high level it looks sound but there are some gaps in the scheme.

  • Explain how the client knows to fall back if a server goes down. (biggest problem I think)
  • Explain how your load balancers just provide a URI. Are these just web servers?
  • How do you handle stateful data such as session cookies which may impart implicit data from a previous server. Else, you use normal cookies?
  • How do you register with the load balancer?
  • How would storage work in this design? How does that scale?
  • How does a load balancer actually balance load in this scheme? Since all it offers are referrals there is no means for it to know when a session has ended on the server side.
  • How does a load balancer know if a server is down?
  • I imagined simple solutions. Clients can listen for specific 4xx/5xx errors to refresh from the LBs, something like heartbeat could monitor server health and update the LB configs automatically, the LBs are indeed a kind special-purpose web server, there is no application state to deal with (no sessions, like Amazon S3), and registering a new server with the LB is an admin task (update configs manually, or through some admin API; should not happen often). The "balancers" ensure initial spread, but S* resource usage polling could influence LB* distribution if needed. Spread may be good enough. – drfloob Mar 3 '14 at 21:51
  • If you're listening on the LB's via some form of poll/websocket its not truly horizontal seeing as all connections ultimately pool on the LB's anyway. May as well get the LB's to do proper load balancing in that case. – Matthew Ife Mar 3 '14 at 22:16
  • As for storage: if storage is necessary at all, isn't choosing the right distributed storage system pretty application-dependent? There are so many options (PostgreSQL, MongoDB, Riak, etc.). I think it may end up being the bottleneck for IO-heavy applications, but I don't know. Presuming the rest of the scheme isn't a terrible idea, I'd build out a few things and load test them to see what falls apart first. – drfloob Mar 3 '14 at 22:21
  • Re pooling on the LBs: having all the client-server data funnel through the LBs is a NO-GO for me. Server affinity is a goal, and I didn't find any open source Layer 4 balancers that did that. Client connections wouldn't pool at the LBs. The LBs still only git hit initially, and when a server dies. Resource usage would only influence server assignment in those two scenarios, so balancing would be loose but present. And I still have hope that simple random assignment or round robin would be sufficient for most cases most of the time. I really appreciate the feedback. – drfloob Mar 3 '14 at 22:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.