1

I run a popular tech website, and for weeks my site has been under DDOS attacks. The attacks happen at random times, and often about 10 times per day. Usually the attacks only last a few minutes, then it stops.

When my site is under attack, it become unreachable. The load of the server does not go up, services like MySQL, Nginx & FPM are not affected. It seems to be a SYN attack or something similar.

I run CentOS 5.6 (64bit), on a powerful machine. I have tried to harden SYSCTL a bit already, my settings can be found below. Also, I have setup iptables to block all ports except the ones I need. That script can also be found below.

I know this question has been asked many times before by people, but are there any further tips someone can give me to fight off these attacks? It is really becoming very annoying.

I am open for trying anything.

SYSCTL SETTINGS

net.ipv4.ip_forward                         = 0
net.ipv4.conf.default.rp_filter             = 1
net.ipv4.conf.default.accept_source_route   = 0
kernel.sysrq                                = 1
kernel.core_uses_pid                        = 1
net.ipv4.tcp_syncookies                     = 1
kernel.msgmnb                               = 65536
kernel.msgmax                               = 65536
kernel.shmmax                               = 68719476736
kernel.shmall                               = 4294967296
net.ipv4.conf.all.send_redirects            = 0
net.ipv4.conf.default.send_redirects        = 0
net.ipv4.tcp_max_syn_backlog                = 2048
net.ipv4.icmp_echo_ignore_broadcasts        = 1
net.ipv4.conf.all.accept_source_route       = 0
net.ipv4.conf.all.accept_redirects          = 0
net.ipv4.conf.all.secure_redirects          = 0
net.ipv4.conf.all.log_martians              = 1
net.ipv4.conf.default.accept_redirects      = 0
net.ipv4.conf.default.secure_redirects      = 0
net.ipv4.icmp_echo_ignore_broadcasts        = 1
net.ipv4.icmp_ignore_bogus_error_responses  = 1
net.ipv4.conf.default.rp_filter             = 1
net.ipv4.tcp_timestamps                     = 0
net.ipv4.conf.all.rp_filter                 = 1
net.ipv4.conf.default.rp_filter             = 1
net.ipv4.conf.eth0.rp_filter                = 1
net.ipv4.conf.lo.rp_filter                  = 1

IPTABLES SETTINGS

*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 21 -m state --state NEW -j ACCEPT
-A INPUT -p tcp --dport 22 --syn -m limit --limit 1/m --limit-burst 3 -j ACCEPT
-A INPUT -p tcp --dport 22 --syn -j DROP
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
COMMIT

MRTG graph of the last 24h. Spikes are attacks, and that is when the server becomes unreachable.

MRTG Graph

4
  • What is that a graph of?
    – womble
    Aug 8, 2011 at 8:06
  • The graph is my network traffic on eth0. Blue line is outgoing traffic, green line incoming traffic.
    – Mr.Boon
    Aug 8, 2011 at 8:11
  • What is the vertical scale on the green graph ?
    – user9517
    Aug 8, 2011 at 8:25
  • Full graph with more info: i.imgur.com/GmNKd.png (notice the huge spike on the far right)
    – Mr.Boon
    Aug 8, 2011 at 8:36

1 Answer 1

2

"It seems to be a SYN attack" -- on what evidence do you base that? Is your conntrack table filling up, or do you have ridiculous numbers of incomplete entries in your netstat output? Given that SYN cookies are usually pretty effective, I'd be more inclined to think it's a simple pipe-filling attack, for which the only solution is to get a bigger pipe, ask your provider to block the abusive traffic upstream (if it's simple traffic like a DNS reflection attack) or get a pipe-scrubbing service (if it's more complicated traffic), either from your upstream provider or a third-party like Arbor networks -- because the firewall on your machine is on the wrong side of the constrained network connection.

Based on the additional information about the nature of the graph, I think it's fairly likely that you're just getting flooded with traffic ("pipe-filling") -- the traffic accounting graphs provided by your provider should be able to confirm that, by showing your traffic rate flatlines at the level of the link capacity you're paying for from your provider.

8
  • That is what I was told by someone. But I might be wrong. I actually just wrote a script that is storing the number of SYN requests every minute. Next time there is an attack, I can see if that number has gone up a lot. Now it's around 50, which seems fine. How can I find out if it is a pipe-filling attack?
    – Mr.Boon
    Aug 8, 2011 at 8:13
  • Edited answer to suit.
    – womble
    Aug 8, 2011 at 8:19
  • The only thing my provider (singlehop) was able to offer me was to put my machine behind their Cisco Guard. This worked well in fighting off the attacks, but it caused another headache, as it would block IP's from normal users way too quickly. My home connection would get blocked after about 25 page views in a few minutes. Also Pingdom got blocked quickly, and bots like the Google Bot. This really wasn't a solution for me. I will look into external pipe-crubbing now. This is something brand new to me :)
    – Mr.Boon
    Aug 8, 2011 at 8:27
  • 2
    Welcome to the world of web-hosting.
    – womble
    Aug 8, 2011 at 8:28
  • Another attack just happened a minute ago. It only lasted a minute. pastebin.com/6du4L2XP that is the output of: netstat -nap | grep SYN. I logged as many as 800 active SYN requests during the attack. Fresh graph can be found here: i.imgur.com/GmNKd.png (notice the huge spike on the far right)
    – Mr.Boon
    Aug 8, 2011 at 8:36

Not the answer you're looking for? Browse other questions tagged .