2

Our infrastructure consists of load balancers running HAProxy and Apache, which forward traffic to our app servers running just Apache. The past few days, we've been seeing connection floods which the load balancers happily pass along, but the connections quickly overwhelm our application servers. They become unresponsive, and our only mitigation tactic is to launch more application servers to sustain the flood. At first we couldn't identify why these servers would go down because there were no real spikes traffic on the load balancers, but after some investigation we see that the number of Apache connections is through the roof.

Attached are some graphs which seem to be the only indicators of the flood. I've lowered the HAProxy maxconn directive for each backend server to a more reasonable number (its previous default was 255), but I worry that new legitimate connections will be delayed until the flood subsides. The service will appear to be up for users with existing connections, but it will appear down to new connections and external monitoring services because HAProxy is rate-limiting without regard to host.

Is there anything else we can do on our end (HAProxy or Apache) to help us sustain the load, or should we look at filtering it outside our network? Since it's such a small bit of traffic, I feel like there is something more we can be doing, but I don't have much familiarity with all of the capabilities of HAProxy. I'm also interested in exploring if HAProxy can rate-limit based on IP, but I haven't found anything.

EDIT: We audited the access log for one of the load balancers, and the request counts for the top IPs are as follows:

1070 69.64.*.*
1227 1.9.*.*
1235 64.71.*.*
1376 69.64.*.*
1459 12.189.*.*
1572 1.9.*.*
1678 208.106.*.*
1982 5.15.*.*
2630 23.22.*.*
3300 76.125.*.* (our office)
3543 216.38.*.*

So, we could dynamically ban IPs that establish too many sessions within a small window, but we couldn't block based on total requests as that would catch us as well. Does this route make sense? Should we do this at the iptables level on the load balancers?

Any advice is much appreciated!

Thanks,

Chris

enter image description here enter image description here enter image description here enter image description here enter image description here enter image description here

1 Answer 1

1

first, if what you get are "just" connection floods, setting the maxconn on the servers will be enough because haproxy will only pass valid requests to the servers, not plain connections with no request. Also, having a lower maxconn makes your servers faster and in general serves your user faster (provided you don't go too low, keep at least 2-3 times the number of CPU cores on the apache machines).

If you get request floods, then you need better counter measures. Haproxy 1.5-dev has a number of these, it is able to blacklist some IP addresses which are doing too many requests, too high connection rates, too many concurrent connections, or experiencing too many errors.

There is a great explanation here http://blog.exceliance.fr/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/ on how to set this up.

Hoping this helps !

2
  • Hi Willy, thanks so much for the reply! I actually read that article while investigating last night. I set the maxconn directive for each backend server to 40 (It was the original 255 before!). These servers are AWS ec2.large instances, so they have 4 cores. I'm messing around with 1.5dev, but getting it into production will take at least a few days of testing. I saw all of the awesome features it has to protect floods, so I'd like to finish testing ASAP. Sep 26, 2012 at 13:41
  • Also, if HAProxy doesn't forward on connections with no requests, I'm afraid that's not what we're dealing with since the app servers have most definitely been getting the requests. So, I guess it makes sense to attempt to mitigate the floods. Sep 26, 2012 at 14:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .