Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Various workstations behind the client side Network Address Translation (NAT) firewall are sending timestamp information on the TCP packet to our server. The packets from many workstations arrive with out-of-sequence time-stamps from behind the NAT device. When they arrive at our server - in some cases on the same port - the server cannot differentiate these packets from others arriving on the same port and from the same client IP (the NAT device).

The server interprets the packets with out of sequence time-stamps as belonging to a connection that has already been completed, and the packet is then ignored - but in this case it should not be. Theses are legitimate packets from workstations behind the NAT device. Dropping packets with old time stamp values is a by-design feature of TCP called PAWS (http://www.ietf.org/rfc/rfc1323.txt, Protect Against Wrapped Sequences) - the server simply assumes the packet is old and it has already dealt with the connection.

To work around for this we disabled the timestamp settings on our servers. BUT - what is the best practice for this situation? Should all servers have timestamp support disabled? Or should all NAT devices remove or rewrite timestamp values? Or?

share|improve this question

1 Answer 1

The source port is an additional identifying feature to the TCP connection, and no two connections behind the NAT device should be assigned the same source port - clients will never interfere with each other's connections unless they get given the same source port by a bad NAT device.

PAWS shouldn't be discarding needed packets, just duplicates from resending - out of order delivery doesn't update the timestamp floor. The time value is copied from the last in-sequence segment; a packet with a higher sequence number (ie, one that is needed) should always have a higher timestamp than one with a lower sequence number.

If it's in proper sequence but has a lower timestamp, then it's the TCP client misbehaving - and if by some oddity PAWS is throwing away good, in-sequence packets, then the client should be re-sending with a new timestamp, recovering from any issues that were caused by the discard.

What behavior are you seeing that's led you to this issue?

share|improve this answer
    
Thanks for the response - based on what you said we should not have seen this problem. But we do see - using wireshark - that our servers are dropping client connections originating from behind this specific NAT device when packets arrive that appear identical in every way - except with older timestamp values. The issue seems specific to this customer - others do not appear affected. This server fault post describes the situation we see too: serverfault.com/questions/235965/… –  user93453 Sep 2 '11 at 13:40
    
@user93453 It's actually dropping the connections completely, or just some individual packets? For established connections, or the handshake like in the linked question? Is it sending a reset packet? Check for interface drops or errors in ifconfig? –  Shane Madden Sep 2 '11 at 18:36
    
Sorry yes- just the individual packets are being ignored. A reset packet is not being sent. What we see is exactly as should occur according to the PAWS spec 1323. It seems to us that timestamps should likely be disabled on high volume servers to avoid this. This particular NAT device has 1000's of people behind it all of whom hit our service heavily, and therefore the problem may be made more apparent. Also we've learned that most Windows OS's have time stamps off by default. Any other insights are greatly appreciated. I would love to know the best practice. –  user93453 Sep 6 '11 at 14:03
    
perhaps this client simply has not allocated enough internal IPs with their NAT, and they are re-using them frequently? –  user93453 Sep 6 '11 at 14:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.