Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Lately, we've become aware of a TCP connection issue that is mostly limited to mac and Linux users who browse our websites.

From the user perspective, it presents itself as a really long connection time to our websites (>11 seconds).

We've managed to track down the technical signature of this problem, but can't figure out why it is happening or how to fix it.

Basically, what is happening is that the client's machine is sending the SYN packet to establish the TCP connection and the web server receives it, but does not respond with the SYN/ACK packet. After the client has sent many SYN packets, the server finally responds with a SYN/ACK packet and everything is fine for the remainder of the connection.

And, of course, the kicker to the problem: it is intermittent and does not happen all the time (though it does happen between 10-30% of the time)

We are using Fedora 12 Linux as the OS and Nginx as the web server.

Screenshot of wireshark analysis

Screenshot of wireshark analysis

Update:

Turning off windows scaling on the client stopped the issue from happening. Now I just need a server side resolution (we can't make all the clients do this) :)

Final Update:

The solution was to turn off tcp windows scaling and tcp timestamps on our servers that are accessible to the public.

share|improve this question
1  
I think we will need to see some tcpdump of it happening. –  coredump Feb 15 '11 at 23:24
    
Do you have any acls or rules based on reverse DNS? You may need to look at more then just the connection between the client and the server. Perhaps a DNS lookup is timing out? –  Zoredache Feb 15 '11 at 23:28
    
@coredump: here's a screen shot of the wireshark analysis that shows the issue i.imgur.com/Bnzrm.png (couldn't figure out how to export just the stream....) –  codemonkey Feb 15 '11 at 23:49
    
@Zoredache: no, we don't have any acls or rules based on reverse DNS. This is a public facing webserver and we allow everyone to access it –  codemonkey Feb 15 '11 at 23:51
    
Just a hunch, but are you doing any kind of incoming connection rate-limiting on the server? Say, with iptables? –  Steven Monday Feb 15 '11 at 23:58

6 Answers 6

Just wondering, but why for the SYN packet (frame #539; the one that was accepted), the WS and TSV fields are missing in the "Info" column?

WS is TCP Windows Scaling feature and TSV is Timestamp value. Both of them are found under tcp.options field and Wireshark still should show them if they are present. Maybe Client TCP/IP stack resent different SYN packet on 8th attempt and that was the reason why it was suddenly acknowledged?

Could you provide us with frame 539 internal values? Does the SYN/ACK always comes for a SYN packet that does not have WS enabled?

share|improve this answer
    
@Ansis: here's some screen shots for frame 539 details (had to do it in two parts): i.imgur.com/D84GC.png & i.imgur.com/4riq3.png –  codemonkey Feb 16 '11 at 0:42
    
@codemonkey: Your 8th SYN packet seems to be different than the first seven SYN packets. Does the server respond with SYN/ACK to the client's SYN only when tcp.options field is of size 8 bytes (The first seven SYN packets probably have tcp.options of size 20 bytes.)? Can you disable TCP window scaling at the client side to see if the problem disappears? Seems like a problem with TCP/IP stack on the server side or misconfigured firewall somewhere... –  Ansis Atteka Feb 16 '11 at 0:52
    
@Ansis: yeah, I've been looking at that since you pointed it out and all the other SYN packets are 24 bytes. I will try disabling window scaling on the client and check back in with the results in the morning. –  codemonkey Feb 16 '11 at 1:10
    
@Ansis: turning off windows scaling on the client stopped the issue from happening. Thanks! However, now I need to figure out how to fix this on the server side (since we can't make all our clients disable windows scaling) :) The server in question does have net.ipv4.tcp_windows_scaling = 1 –  codemonkey Feb 16 '11 at 15:08
    
@Codemonkey: I agree that disabling WS on all clients is not a solution, but we at least have tracked the issue to WS/Packet Size issues. To further find the cause we should look into how your firewall is configured. Can you establish TCP connections with WS to different TCP ports? From different source IPs? –  Ansis Atteka Feb 16 '11 at 18:12

We just ran into the exact same problem (really took quite a while to pin it to server not sending syn-ack).

"The solution was to turn off tcp windows scaling and tcp timestamps on our servers that are accessible to the public."

share|improve this answer

We had this exact same problem. Just disabling TCP timestamps solved the problem.

sysctl -w net.ipv4.tcp_timestamps=0

To make this change permanent, make an entry in /etc/sysctl.conf.

Be very careful about disabling the TCP Window Scale option. This option is important for providing maximum performance over the internet. Someone with a 10 megabit/sec connection will have a suboptimal transfer if the round trip time (basically same as ping) is more than 55 ms.

We really noticed this problem when there were multiple devices behind the same NAT. I suspect that the server might have been confused seeing timestamps from Android devices and OSX machines at the same time since they put completely different values in the timestamp fields.

share|improve this answer
1  
In case someone else ends up here through the same rabbit hole that I just went down: Before turning off TCP timestamps or window scaling, which may have severe performance consequences on a high-traffic link, check to see if tcp_tw_recycle is your problem: stackoverflow.com/questions/8893888/… –  nephtes Oct 6 at 20:43

To carry on what Ansis has stated, I've seen issues like this when the firewall doesn't support TCP Windows Scaling. What make/model firewall is between these two hosts?

share|improve this answer
    
The firewall is a Fedora 13 box using iptables. net.ipv4.tcp_windows_scaling is set to 1 on this machine also –  codemonkey Feb 16 '11 at 15:12

This is the behavior of a listening TCP socket when its backlog is full.

Ngnix allows the backlog argument to listen to be set in the configuration: http://wiki.nginx.org/HttpCoreModule#listen

listen 80 backlog=num

Try setting num to something larger than the default, like 1024.

I provide no guarantee that a full listen queue is actually your problem, but this is a good first thing to check.

share|improve this answer
    
thanks for the tip. I will try it out. We've set the backlog at the OS level, but not explicitly in Nginx config. I'll update with the result. –  codemonkey Feb 16 '11 at 0:11
    
it didn't change the behavior at all. Guess, it's not the problem? or the only problem... –  codemonkey Feb 16 '11 at 0:25

I just discovered that Linux TCP clients change their SYN packet after 3 tries, and remove the Window Scaling option. I guess the kernel developers figured that this is a common cause of connection failure in the Internet

It explains why these clients manage to connect after 11 seconds ( the window-less TCP SYN happens after 9 seconds in my brief test with default settings )

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.