AWS ELB - downstream connections higher than workers count

Question

This might be something I fundamentally don't understand about AWS application elastic load balancer set up.

It's very simple. The load balancer is a public front end, forwarding all traffic to a single target group.

There are N containers in a target group. Each container running some number of workers, say X. Per Amazon documentation it is suggested to have containers with keep-alive enabled and set to a higher value than ELB inbound timeout.

But that essentially means that there is absolutely no point in creating more than N*X connections from ELB to targets, because requests coming on extra connections would just wait in the queue on the target, potentially forever.

From my experiments that is exactly what is happening. ELB opens as many connections to targets as it receives, and during spikes in traffic these extra requests fail.

Kind of defeats the purpose of ELB, does not it?

Ideally there would be an option that would limit how many connections to open per target and ELB would keep extras in a wait queue on its side, before forwarding to targets on their open connections (or by new targets launched by auto-scaling).

I've read all of Amazon ELB documentation and FAQs, and searched extensively here, but did not find any solution other than disabling keep-alives.

Am I missing something basic?

Answer 1

So my basic understanding is, that ELB is opening the connection from the front end application while there are no backend connections available. If that is correct, what you're missing is this:

ELB is supposed to open as many connections as the requests. Its the basic concept of ELB to load balance your requests to as many targets registered to it. So the flaw is not with your ELB obviously it should accept all traffic that is being directed at it. The point that you're missing here is that you should have enough targets registered with the ELB to make sure that ELB can load balance all the traffic that it receives without overloading the targets what it means in simple terms is that you need to set up a proper auto scaling policy to account for the spikes in traffic and making sure that enough backend targets are available to handle the load.

Hope this helps!