Please do not reply with "this is not possible", as it's waste of time. I am developing cloud appliance and I have a valid reason to protect this layer against DDoS, and there are few companies doing the same, so please dont tell me that I dont have the point, as many companies looking to buy this solution and I dont see the problem with implementing it using stock Linux

My Linux kernel is crashing with oops on 10.000 connections due to lack of resources like CPU and RAM. I was wondering how to limit it safely that it doesn't create tcp/ip connections in netfilter connection tracking table or elsewhere when somebody is trying to open 100.000 connections from various hosts?

The network card is 1GBps and with maxed buffers, it can take lot's of connections however I would like to have it to only 5.000 at the same time and the rest being dropped except when there are free connection slots. At the kernel level, so it doesn't pollute netfilter or anything, and it's dropped as soon as possible. There are these factors:

  • Number of HAProxy connections is limited to only 5.000
  • Linux is crashing with 10.000 open connections
  • I want to withstand the 100.000 open connections every minute, so maybe netfilter can handle it, but no HAProxy.
  • The existing connections continue to operate

This is to make machine withstand DDoS attack without making oops, and as soon as attack goes off, the service recovers automatically the way that it continues to serve as normal at low rate.

This is about the physical layer of the server instance, not the switch. Assuming that switch is passing to me this much traffic I can handle, upstream provider does not always have possibility to adjust or to protect at all from this.

  • 2
    Seriously? You want to hack your kernel to limit connections? How does that possibly sound like better idea than using iptables, or contacting your upstream provider to drop the DDoS traffic? – HopelessN00b Aug 15 '12 at 21:39
  • I want to make server not oops on the HAProxy instance, but take the DDoS bravely and resume operation as normal after attack. – Andrew Smith Aug 15 '12 at 22:50

I don't agree that there isn't anything you can do. There are lots of things you can do and depending on the size of the attack and the size of the hardware you've got you stand a pretty good chance on defending yourself.

For SYN floods google a bit. You probably want a spanking new Linux kernel as there have been quite a few improvements lately. Go for 3.6 and enable syn cookies. There are a couple of other tunables you can tweak. Make sure to read up on it first as randomly tuning stuff is never a good idea and will cause problems.

If is is a HTTP flood, which is common these days you might want to consider Varnish. You might be able to identify the attacking requests by some pattern and kill them off in vcl_recv. You can deploy the shield module in order to kill of these connection as serving up an error page is a wasted effort. Be warned: This is not a quick fix and will require substantial effort on your end.

Good luck.

  • Thank you. This is so far very promising. Actually, I already tuned the latest kernel and it's very resilient to high number of connections and accepts high burst rates at the same time. Varnish so far proven to be very very reliable and passed heavy tests and performs better than Apache with mod_qos so far, but mod_qos is my an other option anyway. – Andrew Smith Oct 23 '12 at 20:30

There is no real way you can mitigate a DDOS from the host that is getting attacked.

By the time the traffic has reached your host it has already passed through your local network stack and consumed local resources. No amount of hackery or gyrations can change this.

You need to work with your upstream provider to prevent the traffic from reaching your machine in the first place. If they don't provide that service you can likely purchase it from them or you can find a provider that does.

  • I know it's like sci-fi - it's for next-generation haproxy platform - software resilience to attacks - so it doesn't oops – Andrew Smith Aug 15 '12 at 22:35
  • But wait... if they migrate to the cloud.... They just need to make sure the vendor includes thunder with the cloud... ;-) – Red Tux Aug 16 '12 at 4:15
  • Placing it at various locations with different infrastructure requires coherent and working resilience in the server OS. If you run just a single cluster in same kind of infrastructure this is not problem. – Andrew Smith Aug 16 '12 at 9:27
  • 1
    @AndrewSmith. You're missing the point. Even if you could magically modify HAProxy to gracefully handle this kind of traffic without puking, now your incoming network connection or device/s are saturated. You've just moved the target of your DDOS. You are chasing vaporware. Talk to your provider or hosting company about mitigation plans, everything else is just wishful thinking. – user62491 Aug 16 '12 at 16:50
  • No, I am not missing the point. I dont want to modify HAProxy. I dont want to move target of DDoS. I am not chasing vaporware. I want to do it on Linux and not on switch. I have to do it on both. It's simple solution - cut the network connections to HAProxy at some limit. – Andrew Smith Aug 16 '12 at 19:30

What kind of ddos attack are you suffering from? If it's a syn flood, you could enable syn cookies.

  • I suffer from every type of DDoS attack using multiple kind of services based on various protocols - HTTP, DNS etc – Andrew Smith Aug 16 '12 at 9:20

I'll second the requirement for management of DDoS upstream, but as an intermediate measure you might want to employ some kind of policing or shaping of connections at your own router or switch a hop or so before the servers in question. The best way to bounce back from the crash is not to crash in the first place. The router/firewall/switch isn't actually terminating the packets and is (hopefully) designed to run at a much higher rate.

  • I am looking only for Linux hardware layer of the network card / tcp/ip stack. – Andrew Smith Aug 16 '12 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.