Been up for 2 days please help! SYN flooding Error [closed]

Question

I'm having a problem been working on this for 3 weeks been up the past two days trying to fix this.

My server is a busy server

If I run netstat -plan | grep :80 | wc -l I get 22717.

In my logs I get

TCP: Possible SYN flooding on port 80. Sending cookies

during peak hours....

Server Info:

• OS: Ubuntu Linux 11.10
• CPU: Intel(R) Xeon(R) CPU X5550 @ 2.67GHz, 8 cores
• Ram: 12GB
• Kernel and CPU: Linux 3.0.0-14-server on x86_64
• Apache2 verison: 2.2.20 (Ubuntu)
• Php Version: 5.3.6-13ubuntu3.2
• Mysql Version: 5.1.58 (Has Been runing fine)

When I get TCP: Possible SYN flooding on port 80. Sending cookies in my logs I notice it will start denying me from visiting my site sometimes only happens during peak.

My /etc/sysctl.conf looks like this

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_reordering = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_max_syn_backlog = 2480
net.core.netdev_max_backlog = 1024
net.ipv4.tcp_max_tw_buckets = 360000
net.core.rmem_default = 65535
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.core.wmem_default = 65535
net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096 65535 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.optmem_max = 40960

I was wondering what I need to adjust so my server don't think the heavy traffic is SYN Flooding and denying traffic.

Output of netstat -tan | awk '$4 ~ /:80/ {print $6}' | sort | uniq -c | sort -n:

 1 LISTEN
 31 FIN_WAIT2
116 LAST_ACK
127 CLOSING
225 SYN_RECV
303 FIN_WAIT1
313 ESTABLISHED
22273 TIME_WAIT

I host very small scripts all around the world so the traffic I'm getting on my server is from all the other sites traffic to. The time_wait at peak I've seen it at 17000+. My server runs fine, but when the syn flooding messages are starting to be entered in my logs I'll start have intermittent connection problems. So when I visit my domain and if it does do the problem where it will not load ill refresh the page it will load but if I don't refresh it it wont load and return nothing.

Update:

Is there a kernel out there that does not have syn flooding protection. I want to turn this off since I cannot adjust any settings that will allow more connections to come through.

Answer 1

The command you used doesn't give you enough information. The two main areas where it goes wrong are:

1. It doesn't filter based on whether port 80 is incoming or outgoing.
2. It doesn't filter based on what state the connection is in.

TCP connections can be in any one of several different states. The one you will see a lot of if you are under a SYN flood attack is SYN_RECV. TCP states like ESTABLISHED and TIME_WAIT are fairly normal and don't normally indicate an attack.

This command:

netstat -tan | awk '$4 ~ /:80/ {print $6}' | sort | uniq -c | sort -n

Should give you more useful information. This version restricts the output to only requests that other people are making to your server and not requests you might be making to other people's servers and it also separates out all the different TCP states and aggregates them.

The next thing to investigate is whether this really is a SYN flood or not. Your comment that it only happens during peak hours hints that your server might be running out of some resource. Have a look at your performance graphs (Cacti, Munin, Zabbix, whatever you use) and see if one of the graphs is getting to 100% during your peak times. An attacker will not normally know when your peak times are.

You will also get a lot more information out of:

netstat -s

If it really is a SYN flood, syn cookies are the correct way to deal with it. The problem with a SYN flood is that it uses up resources on your machine keeping track of connections that will never be completed. Syn cookies encode the state of the connection in the response itself, meaning that you don't need to keep track of these half-open connections on your server any more.

SYN flooding may cause you to run out of some other resource (such as bandwidth) first. In that case, you will probably need to either ask your hosting provider to stop the flood or hire a DDoS mitigation company.

Answer 2

From your question it seems that you sure that it's a normal traffic and not an attack. If so, maybe outgoing traffic saturates your connection and replies to SYN just get lost because of that, leaving open connections at SYN ?

The only fact that kernel suspects it's an attack should not disrupt your connections.