Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run several hundred webservers behind loadbalancers, hosting many different sites with a plethora of applications (of which I have no control). About once every month, one of the sites gets hacked and a flood script is uploaded to attack some bank or political institution. In the past, these were always UDP floods which were effectively resolved by blocking outgoing UDP traffic on the individual webserver. Yesterday they started flooding a large US bank from our servers using many TCP connections to port 80. As these type of connections are perfectly valid for our applications, just blocking them is not an acceptable solution.

I am considering the following alternatives. Which one would you recommend? Have you implemented these, and how?

  • Limit on the webserver (iptables) outgoing TCP packets with source port != 80
  • Same but with queueing (tc)
  • Rate limit outgoing traffic per user per server. Quite an administrative burden, as there are potentially 1000's of different users per application server. Maybe this: how can I limit per user bandwidth?
  • Anything else?

Naturally, I'm also looking into ways to minimize the chance of hackers getting into one of our hosted sites, but as that mechanism will never be 100% waterproof, I want to severely limit the impact of an intrusion.

Update: I'm currently testing with these rules, that would have prevented this specific attack. How would you propose to make them more generic? Am I missing a known TCP DoS attack when I only rate limit on SYN packets?

iptables -A OUTPUT -p tcp --syn -m limit --limit 100/min -j ACCEPT
iptables -A OUTPUT -p tcp --syn -m limit --limit 1000/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A OUTPUT -p tcp --syn -j REJECT --reject-with tcp-reset

Cheers!

share|improve this question
add comment

2 Answers 2

up vote 3 down vote accepted

The best possible solutions for my opinion and the one that has worked very well for me is to limit the number of connections/packets for destination ip. Setting the limit to a reasonable rate will prevent the attacker to send a big amount of connections to the target. Setting the port and protocol is not a good idea because if the attacker is sending http flood today, tomorrow he will use different type of attack. so limiting connections per ip as general would be a solution for your problem.

I hope it helps :)

share|improve this answer
add comment

The stance I have always taken is that a webserver should not be making outbound TCP connections at all - only sending traffic as a server responding to inbound requests. (I also allow inbound TCP only for the webserver and SSH.) (Related to this I also believe a webserver is never the right host to send mail from.) This won't just prevent outbound attacks - it also adds a bit of difficulty to the attacks made on your systems (hackers can't obtain an xterm window or wget their toolkit to your host).

share|improve this answer
    
OK, so how does a web site go and fetch an RSS feed from another web site, then? –  Michael Hampton Oct 25 '12 at 16:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.