Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

After a detailed analysis, I collected these details.

I am under a UDP Flood which is more of application dependent. I run a Game-Server and an attacker is flooding me with "getstatus" query which makes the GameServer respond by making the replies to the query which cause output to the attacker's IP as high as 30mb/s and server lag.

Here are the packet details,

Packet starts with 4 bytes 0xff and then getstatus.
Theoretically, the packet is like "\xff\xff\xff\xffgetstatus "

Now that I've tried a lot of iptables variations like state and rate-limiting along side but those didn't work. Rate Limit works good but only when the Server is not started. As soon as the server starts, no iptables rule seems to block it.

Anyone else got more solutions? someone asked me to contact the provider and get it done at the Network/Router but that looks very odd and I believe they might not do it since that would also affect other clients.


Responding to all those answers, I'd say:

Firstly, its a VPS so they can't do it for me. Secondly, I don't care if something is coming in but since its application generated so there has to be a OS level solution to block the outgoing packets. At least the outgoing ones must be stopped.

Secondly, its not Ddos since just 400kb/s input generates 30mb/s output from my GameServer. That never happens in a D-dos. Asking the provider/hardware level solution should be used in that case but this one is different. And Yes, Banning his IP stops the flood of outgoing packets but he has many more IP-Addresses as he spoofs his original so I just need something to block him automatically.

Even tried a lot of Firewalls but as you know they are just front-ends to iptables so if something doesn't work on iptables, what would the firewalls do?

These were the rules I tried,

iptables -A INPUT -p udp -m state --state NEW -m recent --set --name DDOS --rsource
iptables -A INPUT -p udp -m state --state NEW -m recent --update --seconds 1 --hitcount 5 --name DDOS --rsource -j DROP

It works for the attacks on un-used ports but when the server is listening and responding to the incoming queries by the attacker, it never works.


Okay Tom.H, your rules were working when I modified them somehow like this:

iptables -A INPUT -p udp -m length --length 1:1024 -m recent --set --name XXXX --rsource
iptables -A INPUT -p udp -m string --string "xxxxxxxxxx" --algo bm --to 65535 -m recent --update --seconds 1 --hitcount 15 --name XXXX --rsource -j DROP

They worked for about 3 days very good where the string "xxxxxxxxx" would be rate-limited, blocked if someone flooded and also didn't affect the clients. But just today, I tried updating the chain to try to remove a previously blocked IP so for that I had to flush the chain and restore this rule ( iptables -X and iptables -F ), some clients were already connected to servers including me. So restoring the rules now would also block some of the clients string completely while some are not affected. So does this mean I need to restart the server or why else would this happen because the last time the rules were working, there was no one connected?

share|improve this question
    
What rate-limiting rules have you tried? –  Jeff Ferland Mar 15 '12 at 16:54
    
some qos rules could clear this up. and stop one ip/user taking up a unfair amount of resources. you can do this for both inbound and outbound traffic. –  The Unix Janitor Mar 15 '12 at 17:01
    
Tried something like this: iptables -A INPUT -p udp -m state --state NEW -m recent --set --name DDOS --rsource iptables -A INPUT -p udp -m state --state NEW -m recent --update --seconds 1 --hitcount 5 --name DDOS --rsource -j DROP It works for the attacks on non-used ports but when the server is listening and responding to the queries, it never works. –  Asad Moeen Mar 16 '12 at 11:21
    
@AsadMoeen it doesnt work when the server is started because youre using --state NEW. UDP is stateless so connection tracking on it is fuzzy. So it sees consecutive packets to an open port as being the same connection, and thus state is not new. –  Patrick Mar 16 '12 at 11:38
    
So what else should I use? –  Asad Moeen Mar 16 '12 at 11:40
show 30 more comments

4 Answers

up vote 8 down vote accepted

You're very nearly there, but it's possible that you've been cargo-culting someone else's work, possibly on ssh rate-limiting, without really understanding it. Please note that I'm not criticising you: building on other people's work is an excellent idea in the free software community; but you should understand why they've done what they've done, so you don't fail to use it correctly.

I set up a test rig, using nc (netcat) to flood UDP traffic from a machine called bill to a machine called risby with the following lines:

risby% nc -l -u 12345
bill% seq 1 10000000 | nc -u risby 12345

This produced a very-rapidly increasing list of numbers from risby's netcat, much like the command-flooding you've been having.

But when I created two new rules for risby's iptables which filtered only UDP traffic to port 12345 without regard for state, it worked fine:

iptables -I INPUT 1 -p udp --dport 12345 -m recent --set --name ddos
iptables -I INPUT 2 -p udp --dport 12345 -m recent --rcheck  --seconds 1 --hitcount 5 --name ddos -j DROP

When I re-ran the netcats, the first few packets from bill got through on risby, and the numbers climbed rapidly to about 1800, but then it stalled completely and no further traffic was received from bill.

Note that it's quite important that these rules come early in your iptables INPUT chain, which is why I've inserted them at lines 1 and 2 respectively.

Edit:

Increase the rate, and require it to be sustained for longer; perhaps --seconds 10 --hitcount 50? Eventually you'll reach a threshold where few legitimate clients are affected, but the DDoS is still substantially throttled. Note that friendly-fire is always a possibility in this kind of layer-3 throttling; my own ssh server limits new connections to two per 60s window, which makes repeated scps quite slow. But it's a price I'm willing to pay, and to do better requires layer-4 throttling, which means the application has to be throttling-aware. iptables can't help you there.

I note that --hitcount can take no value higher than the ip_pkt_list_tot parameter of the xt_recent kernel module, and if the value's exceeded an error is thrown at rule-creation time:

[root@risby scratch]# iptables -A INPUT -p udp -m recent --rcheck  --seconds 1 --hitcount 50 --name ddos -j DROP 
iptables: Invalid argument. Run `dmesg' for more information.

But this value can be set at up to 255 at module insertion time. Following the suggestions in this blog entry, it's possible to reload the module, setting the parameter explicitly:

[root@risby scratch]# rmmod xt_recent
[root@risby scratch]# modprobe xt_recent ip_pkt_list_tot=100
[root@risby scratch]# iptables -A INPUT -p udp -m recent --rcheck  --seconds 1 --hitcount 50 --name ddos -j DROP 
[root@risby scratch]# 

Note how the --hitcount 50 no longer causes errors. You may need to flush the INPUT chain (iptables -F INPUT) and any other chains that use the recent module before you can remove and reinsert the xt_recent module.

share|improve this answer
    
Where this works pretty fine to block the attacker but it also blocks the clients to the server although their rate is lesser than the flooder so we might need something else and yes we're almost there. And also hitcount>20 is not accepted by iptables so I can't increase that either in-case the Game-Server players have a higher rate. –  Asad Moeen Mar 16 '12 at 15:42
    
See my edit above. –  MadHatter Mar 16 '12 at 15:53
    
I think with the addition of the string match for the getstatus request, this additional more sophisticated rate limiting would do the trick. –  Tom H Mar 16 '12 at 16:09
    
Nicely combined! I suspect you're right. –  MadHatter Mar 16 '12 at 16:17
    
Gonna try that now! –  Asad Moeen Mar 16 '12 at 16:21
show 1 more comment

use tcpdump to take a packet dump of the traffic.

tcpdump -s0 -w somefile.tcp proto udp and port NN and host www.xxx.yyy.zzz

inspect the packets in wireshark for the byte string you want to match;

create an iptables rule with a string match to look for the application protocol string, to allow for a certain number of those packets per second and then drop the rest of those packets

iptables -I INPUT -p udp --dport NN -m string  --algo bm \
 --hex-string "|ff ff ff ff 67 65 74 73 74 61 74 75 73|" \
 -m limit --limit 1/second -j ACCEPT

iptables -I INPUT -p udp --dport NN -m string  --algo bm \
 --hex-string "|ff ff ff ff 67 65 74 73 74 61 74 75 73|" -j DROP 

its lucky this is udp, because its less resource intensive to do the matching on the netfilter module...

caveats are that you are going to block all getstatus requests, unless you can find some other filter for just this source, and that you will have to do a little bit of wikipedia work to work out the correct hex representation of your match string

share|improve this answer
    
That probably looks good and might work but the thing is that some fair clients also use the same string "getstatus" protocol so we might have to rate-limit it for working around. And yes that is the correct hex of the string. But I'm not sure if that would auto include anything from the end of the string because the application reads anything after "getstatus" also as getstatus so "getstatus." would also be getstatus. So how do we rate limit inside that? –  Asad Moeen Mar 16 '12 at 12:00
    
there is an article here; tty1.net/blog/2007-02-06-iptables-firewall_en.html which talks about applying rate filtering on top of particular matches. –  Tom H Mar 16 '12 at 12:17
    
I would try that. But wouldn't direct rate-limiting work? I just tried limiting all the packets on the application port to lets say 20 per second which blocked the attacker but also kind of rate-limited the clients as they started Lagging. –  Asad Moeen Mar 16 '12 at 12:30
    
This is the solution. Normal clients don't send any more then one-two getstatus packets a second, so they shouldn't be impacted by this. Also, once they are connected to your server they no longer send getstatus packets. It should be noted here that you are not the target of this DDoS. They are using your server as part of a DDoS reflection attack. –  devicenull Mar 16 '12 at 13:13
    
Yes but the rules did not work and my clients also started Lagging so I removed them. I don't know why but maybe the hitcount/seconds are not working. I also tried a rule like "iptables -A INPUT -p udp -m udp -m length --length 40 -j DROP" and amazingly this also did not block the attacker although I could clearly see in tcpdump that packet length is a constant 40. Why wouldn't it work? or maybe when connection is established, length wont work. –  Asad Moeen Mar 16 '12 at 13:38
show 3 more comments

Contact your provier and ask them to block the traffic at the router.
This will not affect other clients as they will take the destination of the packets (=your server) into account.

Every iptables or other local approach will not be a solution, since the packets need to be dealt with anyway so they will affect your server.

share|improve this answer
    
so your asking him to contact the ISP everytime a dos is * detected* , i don't know who your isp is, but i'd like to be on yours! He did not say the his server is cpu bound. Qos rules can actually help flatten out out resource bottlenecks if deployed correctly. –  The Unix Janitor Mar 15 '12 at 17:03
    
Its a VPS so they can't do it for me. Secondly, I don't care if something is coming in but since its application generated so there has to be a OS level solution to block the outgoing packets. At least the outgoing ones must be stopped. –  Asad Moeen Mar 15 '12 at 18:20
    
Asad, you did not quite get what i am saying, it is not only about outgoing packets, even if your application is not replying to that kind of requests anymore you are still facing a large amount of incoming traffic which will eventually be too much for your server/iptables/network stack to handle, that's the whole point of a DOS Attack. @user37899 you need to read up about DOS Attacks and Network Security, you are missing the point here. –  Niko S P Mar 15 '12 at 23:08
    
How large is 70kb/s compared to Gigabits of Ddos? Yes 70kb/s flood generates 2mb/s output from the server. –  Asad Moeen Mar 16 '12 at 10:15
    
Did you ever experience a real DOS? 70kb/s does not even remotely qualify as a serious threat. I guess i am thinking about a much larger scale than you are. –  Niko S P Mar 16 '12 at 10:43
show 2 more comments

Have you considered banning his IP (if he's not spoofing it)? What you are experiencing is known as Denial of Service. I suggest you try out OSSEC. It might help blocking the IP's the attacker is using.

share|improve this answer
    
if he is spoofing his ip, make sure you protocol needs a tcp connection first then only accept traffic with udp connection from the same ip address. won't work for people behind nat. :-(. If your have lots of clients behind nat, think off adding a per user session key to your protocol. –  The Unix Janitor Mar 15 '12 at 17:05
    
Its not Ddos since 400kb/s input generates 30mb/s output from my GameServer. That never happens in a Ddos. And Yes, Banning his IP stops the flood of outgoing packets but he has many more as he spoofs his original IP Address so I just need something to block him automatically. –  Asad Moeen Mar 15 '12 at 18:23
    
you need to change your protocol so he has to login in, or create a new account. also you need to add protection and throttling methods in your game code to stop resources hots and malicious requests. what game is this? can i play? –  The Unix Janitor Mar 16 '12 at 13:36
    
Soldier of Fortune 2 - 1.00 Now coding is a tough ask. –  Asad Moeen Mar 16 '12 at 13:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.