Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm having a problem been working on this for 3 weeks been up the past two days trying to fix this.

My server is a busy server

If I run netstat -plan | grep :80 | wc -l I get 22717.

In my logs I get

TCP: Possible SYN flooding on port 80. Sending cookies

during peak hours....

Server Info:

  • OS: Ubuntu Linux 11.10
  • CPU: Intel(R) Xeon(R) CPU X5550 @ 2.67GHz, 8 cores
  • Ram: 12GB
  • Kernel and CPU: Linux 3.0.0-14-server on x86_64
  • Apache2 verison: 2.2.20 (Ubuntu)
  • Php Version: 5.3.6-13ubuntu3.2
  • Mysql Version: 5.1.58 (Has Been runing fine)

When I get TCP: Possible SYN flooding on port 80. Sending cookies in my logs I notice it will start denying me from visiting my site sometimes only happens during peak.

My /etc/sysctl.conf looks like this

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_reordering = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_max_syn_backlog = 2480
net.core.netdev_max_backlog = 1024
net.ipv4.tcp_max_tw_buckets = 360000
net.core.rmem_default = 65535
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.core.wmem_default = 65535
net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096 65535 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.optmem_max = 40960

I was wondering what I need to adjust so my server don't think the heavy traffic is SYN Flooding and denying traffic.

Output of netstat -tan | awk '$4 ~ /:80/ {print $6}' | sort | uniq -c | sort -n:

 1 LISTEN
 31 FIN_WAIT2
116 LAST_ACK
127 CLOSING
225 SYN_RECV
303 FIN_WAIT1
313 ESTABLISHED
22273 TIME_WAIT

I host very small scripts all around the world so the traffic I'm getting on my server is from all the other sites traffic to. The time_wait at peak I've seen it at 17000+. My server runs fine, but when the syn flooding messages are starting to be entered in my logs I'll start have intermittent connection problems. So when I visit my domain and if it does do the problem where it will not load ill refresh the page it will load but if I don't refresh it it wont load and return nothing.

Update:

Is there a kernel out there that does not have syn flooding protection. I want to turn this off since I cannot adjust any settings that will allow more connections to come through.

share|improve this question

closed as not a real question by John Gardeniers, Skyhawk, Chris S, Ward, Zoredache Dec 21 '11 at 7:20

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

5  
Posting the same question over and over again isn't generally going to help. I suggest you go back to your old question and edit it to add more detail. This will also bump it back to the front page, with the added bonus of hopefully being a better question. –  Mark Henderson Dec 11 '11 at 10:20
    
How do you KNOW that the server isn't actually being attacked? How many page requests does the server usually handle? How long does it take to process an average page request? –  pipTheGeek Dec 11 '11 at 10:40
    
What's your CPU load ? –  Sandman4 Dec 11 '11 at 10:50
    
My CPU load at peak is around 20 - 40% linux shows 0.10 - 0.40. i look under apache request currently being processed at peak is around 400. I never reach my max clients. I do notice on avg I have a lot of connections TIME_WAIT 17000 is there a way to close the connections out faster? –  chiller Dec 11 '11 at 11:05
    
What about network traffic, especially outgoing ? Is it possible that your connection is saturated ? –  Sandman4 Dec 11 '11 at 11:06

2 Answers 2

The command you used doesn't give you enough information. The two main areas where it goes wrong are:

  1. It doesn't filter based on whether port 80 is incoming or outgoing.
  2. It doesn't filter based on what state the connection is in.

TCP connections can be in any one of several different states. The one you will see a lot of if you are under a SYN flood attack is SYN_RECV. TCP states like ESTABLISHED and TIME_WAIT are fairly normal and don't normally indicate an attack.

This command:

netstat -tan | awk '$4 ~ /:80/ {print $6}' | sort | uniq -c | sort -n

Should give you more useful information. This version restricts the output to only requests that other people are making to your server and not requests you might be making to other people's servers and it also separates out all the different TCP states and aggregates them.

The next thing to investigate is whether this really is a SYN flood or not. Your comment that it only happens during peak hours hints that your server might be running out of some resource. Have a look at your performance graphs (Cacti, Munin, Zabbix, whatever you use) and see if one of the graphs is getting to 100% during your peak times. An attacker will not normally know when your peak times are.

You will also get a lot more information out of:

netstat -s

If it really is a SYN flood, syn cookies are the correct way to deal with it. The problem with a SYN flood is that it uses up resources on your machine keeping track of connections that will never be completed. Syn cookies encode the state of the connection in the response itself, meaning that you don't need to keep track of these half-open connections on your server any more.

SYN flooding may cause you to run out of some other resource (such as bandwidth) first. In that case, you will probably need to either ask your hosting provider to stop the flood or hire a DDoS mitigation company.

share|improve this answer
    
1 CLOSE_WAIT 1 LISTEN 19 FIN_WAIT2 28 CLOSING 37 LAST_ACK 106 FIN_WAIT1 153 SYN_RECV 191 ESTABLISHED 10361 TIME_WAIT It isn't peak time but the info you requested. –  chiller Dec 11 '11 at 11:11
    
You can probably reduce how often you see the error message by increasing the value of net.ipv4.tcp_max_syn_backlog. It seems that Linux only starts using syn-cookies once it has too many connection attempts to keep track of. The default is 1024 so it seems you have already increased this once. This is only dealing with the error message and not the error, however. As long as your server is still serving clients properly, that might be enough. –  Ladadadada Dec 11 '11 at 11:45
    
The problem is when the syn flooding errors start coming in I start to have a intermittent connection lost where my browser will return nothing. If I refresh the page It will load quickly. –  chiller Dec 11 '11 at 11:50
    
I will try adjusting net.ipv4.tcp_max_syn_backlog what would be recommended to adjust from 1024 to X? –  chiller Dec 11 '11 at 11:58
    
You want it to be higher than the highest number of simultaneous half-open connection attempts you regularly get. For now, I would try 4096 and double it if you still get the error message. I'm not sure where to look for how much RAM this uses up so if you find it, please add it here. –  Ladadadada Dec 11 '11 at 12:40

From your question it seems that you sure that it's a normal traffic and not an attack. If so, maybe outgoing traffic saturates your connection and replies to SYN just get lost because of that, leaving open connections at SYN ?

The only fact that kernel suspects it's an attack should not disrupt your connections.

share|improve this answer
    
My webpage will not load sometimes when the kernel thinks it's getting syn flooding. When I visit my webpage and it does not want to load ill refresh the page it will load quickly. –  chiller Dec 11 '11 at 11:16
    
My point is that possibly kernel TCP flood message is not the cause but another consequence of the real problem. –  Sandman4 Dec 11 '11 at 11:48
    
I understand but I'm only getting that message in the log when I have problems with my webpage so I'm assuming it has something to do with my problems. I was wondering if there something I can adjust to allow more connections so the kernel TCP flood message would go away or reduce it. –  chiller Dec 11 '11 at 11:54
    
maybe "echo 0 > /proc/sys/net/ipv4/tcp_syncookies" –  Sandman4 Dec 11 '11 at 12:12
    

Not the answer you're looking for? Browse other questions tagged or ask your own question.