53

Lately, we've become aware of a TCP connection issue that is mostly limited to mac and Linux users who browse our websites.

From the user perspective, it presents itself as a really long connection time to our websites (>11 seconds).

We've managed to track down the technical signature of this problem, but can't figure out why it is happening or how to fix it.

Basically, what is happening is that the client's machine is sending the SYN packet to establish the TCP connection and the web server receives it, but does not respond with the SYN/ACK packet. After the client has sent many SYN packets, the server finally responds with a SYN/ACK packet and everything is fine for the remainder of the connection.

And, of course, the kicker to the problem: it is intermittent and does not happen all the time (though it does happen between 10-30% of the time)

We are using Fedora 12 Linux as the OS and Nginx as the web server.

Screenshot of wireshark analysis

Screenshot of wireshark analysis

Update:

Turning off window scaling on the client stopped the issue from happening. Now I just need a server side resolution (we can't make all the clients do this) :)

Final Update:

The solution was to turn off both TCP window scaling and TCP timestamps on our servers that are accessible to the public.

12
  • 1
    I think we will need to see some tcpdump of it happening.
    – coredump
    Feb 15, 2011 at 23:24
  • Do you have any acls or rules based on reverse DNS? You may need to look at more then just the connection between the client and the server. Perhaps a DNS lookup is timing out?
    – Zoredache
    Feb 15, 2011 at 23:28
  • @coredump: here's a screen shot of the wireshark analysis that shows the issue i.imgur.com/Bnzrm.png (couldn't figure out how to export just the stream....)
    – codemonkey
    Feb 15, 2011 at 23:49
  • @Zoredache: no, we don't have any acls or rules based on reverse DNS. This is a public facing webserver and we allow everyone to access it
    – codemonkey
    Feb 15, 2011 at 23:51
  • Just a hunch, but are you doing any kind of incoming connection rate-limiting on the server? Say, with iptables? Feb 15, 2011 at 23:58

9 Answers 9

17

We had this exact same problem. Just disabling TCP timestamps solved the problem.

sysctl -w net.ipv4.tcp_timestamps=0

To make this change permanent, make an entry in /etc/sysctl.conf.

Be very careful about disabling the TCP Window Scale option. This option is important for providing maximum performance over the internet. Someone with a 10 megabit/sec connection will have a suboptimal transfer if the round trip time (basically same as ping) is more than 55 ms.

We really noticed this problem when there were multiple devices behind the same NAT. I suspect that the server might have been confused seeing timestamps from Android devices and OSX machines at the same time since they put completely different values in the timestamp fields.

2
  • 6
    In case someone else ends up here through the same rabbit hole that I just went down: Before turning off TCP timestamps or window scaling, which may have severe performance consequences on a high-traffic link, check to see if tcp_tw_recycle is your problem: stackoverflow.com/questions/8893888/…
    – nephtes
    Oct 6, 2014 at 20:43
  • First check that server and client have the correct time. May 12, 2022 at 5:21
16

In my case the following command has fixed the problem with missing SYN/ACK replies from the Linux server:

sysctl -w net.ipv4.tcp_tw_recycle=0

I think this is more appropriate than disabling TCP timestamps, since TCP timestamps can be useful for high performance (PAWS, window scaling, etc).

The documentation on the tcp_tw_recycle explicitly states that it is not recommended to enable it, as many NAT routers preserve timestamps and this can cause PAWS (Protection Against Wrapping Sequence) to activate, as timestamps from the same source IP address can become inconsistent.

   tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4)
          Enable fast recycling of TIME_WAIT sockets.  Enabling this
          option is not recommended for devices communicating with the
          general Internet or using NAT (Network Address Translation).
          Since some NAT gateways pass through IP timestamp values, one
          IP can appear to have non-increasing timestamps.  See RFC 1323
          (PAWS), RFC 6191.
4
  • 1
    good explanation at here: vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux On the server side, do not enable net.ipv4.tcp_tw_recycle unless you are pretty sure you will never have NAT devices in the mix.
    – Gnought
    Apr 6, 2017 at 7:02
  • 2
    In my case, net.ipv4.tcp_tw_recycle is the real reason. Thanks.
    – bluearrow
    Dec 26, 2017 at 3:10
  • 2
    tcp_tw_recycle has been removed in recent kernels. Is there another solution alike? @nephtes implies disabling timestamp hurts performance.
    – MappaM
    Mar 6, 2019 at 14:28
  • 1
    Since tcp_tw_recycle has been removed, the problem should not occur again as it only happened with a non-default value of tcp_tw_recycle.
    – lav
    Mar 7, 2019 at 9:01
5

Just wondering, but why for the SYN packet (frame #539; the one that was accepted), the WS and TSV fields are missing in the "Info" column?

WS is TCP Window Scaling and TSV is Timestamp Value. Both of them are found under tcp.options field and Wireshark still should show them if they are present. Maybe Client TCP/IP stack resent different SYN packet on 8th attempt and that was the reason why it was suddenly acknowledged?

Could you provide us with frame 539 internal values? Does the SYN/ACK always comes for a SYN packet that does not have WS enabled?

8
  • @Ansis: here's some screen shots for frame 539 details (had to do it in two parts): i.imgur.com/D84GC.png & i.imgur.com/4riq3.png
    – codemonkey
    Feb 16, 2011 at 0:42
  • @codemonkey: Your 8th SYN packet seems to be different than the first seven SYN packets. Does the server respond with SYN/ACK to the client's SYN only when tcp.options field is of size 8 bytes (The first seven SYN packets probably have tcp.options of size 20 bytes.)? Can you disable TCP window scaling at the client side to see if the problem disappears? Seems like a problem with TCP/IP stack on the server side or misconfigured firewall somewhere... Feb 16, 2011 at 0:52
  • @Ansis: yeah, I've been looking at that since you pointed it out and all the other SYN packets are 24 bytes. I will try disabling window scaling on the client and check back in with the results in the morning.
    – codemonkey
    Feb 16, 2011 at 1:10
  • @Ansis: turning off windows scaling on the client stopped the issue from happening. Thanks! However, now I need to figure out how to fix this on the server side (since we can't make all our clients disable windows scaling) :) The server in question does have net.ipv4.tcp_windows_scaling = 1
    – codemonkey
    Feb 16, 2011 at 15:08
  • @Codemonkey: I agree that disabling WS on all clients is not a solution, but we at least have tracked the issue to WS/Packet Size issues. To further find the cause we should look into how your firewall is configured. Can you establish TCP connections with WS to different TCP ports? From different source IPs? Feb 16, 2011 at 18:12
4

We just ran into the exact same problem (really took quite a while to pin it to server not sending syn-ack).

"The solution was to turn off tcp windows scaling and tcp timestamps on our servers that are accessible to the public."

3

The missing SYN/ACK could be caused by too low limits of your SYNFLOOD protection on firewall. It depends on how many connections to your server user creates. Using spdy would reduce the number of connections and could help in situation where turning net.ipv4.tcp_timestamps off does not help.

2

To carry on what Ansis has stated, I've seen issues like this when the firewall doesn't support TCP Windows Scaling. What make/model firewall is between these two hosts?

1
  • The firewall is a Fedora 13 box using iptables. net.ipv4.tcp_windows_scaling is set to 1 on this machine also
    – codemonkey
    Feb 16, 2011 at 15:12
1

This is the behavior of a listening TCP socket when its backlog is full.

Ngnix allows the backlog argument to listen to be set in the configuration: http://wiki.nginx.org/HttpCoreModule#listen

listen 80 backlog=num

Try setting num to something larger than the default, like 1024.

I provide no guarantee that a full listen queue is actually your problem, but this is a good first thing to check.

3
  • thanks for the tip. I will try it out. We've set the backlog at the OS level, but not explicitly in Nginx config. I'll update with the result.
    – codemonkey
    Feb 16, 2011 at 0:11
  • it didn't change the behavior at all. Guess, it's not the problem? or the only problem...
    – codemonkey
    Feb 16, 2011 at 0:25
  • 1
    application level backlog parameter controls size of queue for completed tcp connections i.e. 3-way handshake finished, i.e. syn-ack received - so it doesn't match OP situation
    – ygrek
    Oct 15, 2015 at 21:59
1

I just discovered that Linux TCP clients change their SYN packet after 3 tries, and remove the Window Scaling option. I guess the kernel developers figured that this is a common cause of connection failure in the Internet

It explains why these clients manage to connect after 11 seconds ( the window-less TCP SYN happens after 9 seconds in my brief test with default settings )

0

I had a similar problem, but in my case it was the TCP checksum that was wrongly computed. The client was behind a veth and running ethtool -K veth0 rx off tx off did the trick.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .