Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've read tons of posts about this, but none of them have an actual answer.

We have 3 machines, one (not in our care) balance loads to the other two. The first one is working correctly with a FreeBSD, the other one was recently formatted and is now using Ubuntu Server.

The second machine is currently leaving all connections on SYN_RECV for no apparent reason. Both machines have no firewall.

dmesg shows the Possible SYN Flood attack but we know it is not the case.

What could be wrong? Is there some kernel configuration I must do? Does Ubuntu have some known issue with this?

Thanks

Edit: I found a rule on the PF of the BSD machine, I'm not sure but it should be related to this problem.

pass in log on bce1 proto tcp from <nois> to any port = http flags S/SA keep state (max 2000, source-track rule, max-src-states 120, max-src-conn 80, adaptive.start 1200, adaptive.end 2400)

It basically keepts state of SYN SYNACK flag packages... does anyone know how to translate this to IPTABLES ?

share|improve this question
    
What is the actual problem? Is something not working? You list a lot of information from troubleshooting, but not why you started troubleshooting in the first place. Why do you think something is wrong exactly? –  David Schwartz Nov 10 '11 at 16:40
    
basically services stop working. nginx/unicorn stop working because they cannot complete requests. It is a webserver that can't serve anything. –  Draiken Nov 10 '11 at 16:58
    
That's a basic reflexive accept rule. You shouldn't need it unless you have a rule to deny essential traffic. What firewall rules do you have on this box? –  David Schwartz Nov 10 '11 at 20:36
    
No firewall on the ubuntu... –  Draiken Nov 11 '11 at 10:17

2 Answers 2

Basically, what's happening is that the Linux box is receiving SYNs (attempting to open up a TCP connection) and then it's not getting ACKs (completely the process). The half-open connections accumulate until the server decides it's under attack.

There are a few possible causes:

First, the load balancer might be configured to do this. It might pass a SYN on to both boxes and only allow one connection to complete to the client, since it only needs one. The balancer might assume that ignoring the reply is harmless, since packets drop all the time. However, in this case, there's really no reason you should be having an actual problem. Make sure you have syncookies enabled, just to be sure it wouldn't cause a problem if this is the case.

Second, the Linux box's ACK in response to the SYN might not be getting through the Linux box's firewall. This would require a pretty badly broken firewall configuration, so it's not likely.

The less likely possibilities are:

Third, the load balancer may reject the Linux box's ACK for some reason.

Fourth, the load balancer may fail to transmit an ACK back to the Linux box.

Fifth, the Linux box's firewall may drop the incoming ACK.

share|improve this answer
    
syncookies are on, and I'm going to investigate these other possibilities... this nightmare seems to never end. –  Draiken Nov 11 '11 at 10:20

From what you say it sounds more than likely that the loadbalancer checks the returned SYN/ACK from the backend server to see if it is up, and only then communicating said SYN/ACK back to the client; this means in addition to loadbalancing it is also performing failover duties.

Apparently, FreeBSD handles this differently than Ubuntu does, and the Ubuntu box sees this as a SYN flood attack.

Sniffing the traffic from the clients and between the load balancer and the backends will tell you exactly what is happening.

share|improve this answer
    
We only have access to the two machines, the load balancer is not in our reach :/ any other ideas on how I could find out what is this problem? –  Draiken Nov 10 '11 at 16:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.