2

Various workstations behind the client side Network Address Translation (NAT) firewall are sending timestamp information on the TCP packet to our server. The packets from many workstations arrive with out-of-sequence time-stamps from behind the NAT device. When they arrive at our server - in some cases on the same port - the server cannot differentiate these packets from others arriving on the same port and from the same client IP (the NAT device).

The server interprets the packets with out of sequence time-stamps as belonging to a connection that has already been completed, and the packet is then ignored - but in this case it should not be. Theses are legitimate packets from workstations behind the NAT device. Dropping packets with old time stamp values is a by-design feature of TCP called PAWS (http://www.ietf.org/rfc/rfc1323.txt, Protect Against Wrapped Sequences) - the server simply assumes the packet is old and it has already dealt with the connection.

To work around for this we disabled the timestamp settings on our servers. BUT - what is the best practice for this situation? Should all servers have timestamp support disabled? Or should all NAT devices remove or rewrite timestamp values? Or?

2 Answers 2

4

The source port is an additional identifying feature to the TCP connection, and no two connections behind the NAT device should be assigned the same source port - clients will never interfere with each other's connections unless they get given the same source port by a bad NAT device.

PAWS shouldn't be discarding needed packets, just duplicates from resending - out of order delivery doesn't update the timestamp floor. The time value is copied from the last in-sequence segment; a packet with a higher sequence number (ie, one that is needed) should always have a higher timestamp than one with a lower sequence number.

If it's in proper sequence but has a lower timestamp, then it's the TCP client misbehaving - and if by some oddity PAWS is throwing away good, in-sequence packets, then the client should be re-sending with a new timestamp, recovering from any issues that were caused by the discard.

What behavior are you seeing that's led you to this issue?

4
  • Thanks for the response - based on what you said we should not have seen this problem. But we do see - using wireshark - that our servers are dropping client connections originating from behind this specific NAT device when packets arrive that appear identical in every way - except with older timestamp values. The issue seems specific to this customer - others do not appear affected. This server fault post describes the situation we see too: serverfault.com/questions/235965/…
    – user93453
    Sep 2, 2011 at 13:40
  • @user93453 It's actually dropping the connections completely, or just some individual packets? For established connections, or the handshake like in the linked question? Is it sending a reset packet? Check for interface drops or errors in ifconfig? Sep 2, 2011 at 18:36
  • Sorry yes- just the individual packets are being ignored. A reset packet is not being sent. What we see is exactly as should occur according to the PAWS spec 1323. It seems to us that timestamps should likely be disabled on high volume servers to avoid this. This particular NAT device has 1000's of people behind it all of whom hit our service heavily, and therefore the problem may be made more apparent. Also we've learned that most Windows OS's have time stamps off by default. Any other insights are greatly appreciated. I would love to know the best practice.
    – user93453
    Sep 6, 2011 at 14:03
  • perhaps this client simply has not allocated enough internal IPs with their NAT, and they are re-using them frequently?
    – user93453
    Sep 6, 2011 at 14:37
1

In my case the following command fixed the problem with missing SYN/ACK replies from Linux server:

sysctl -w net.ipv4.tcp_tw_recycle=0

I think it is more correct than disabling TCP timestamps, as TCP timestamps are useful after all (PAWS, window scaling, etc).

The documentation on the tcp_tw_recycle explicitly states that it is not recommended to enable it, as many NAT routers preserve timestamps and thus PAWS kicks in, as timestamps from the same IP are not consistent.

   tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4)
          Enable fast recycling of TIME_WAIT sockets.  Enabling this
          option is not recommended for devices communicating with the
          general Internet or using NAT (Network Address Translation).
          Since some NAT gateways pass through IP timestamp values, one
          IP can appear to have non-increasing timestamps.  See RFC 1323
          (PAWS), RFC 6191.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .