I'm experiencing an issue on our RHEL7 boxes that are talking to a central server sitting behind a Cisco ASA firewall.
These machines boot and start up an agent that establishes and keeps alive a connection to the central server. The central server then periodically sends traffic down that tunnel to the client. At the 5 minute mark, we were seeing the agents disconnect and were unable to send traffic over the connection unless we restarted the connection, after which point it worked fine.
Further research showed that it wasn't just this application/agent, and in fact we could replicate it with 'nc', without fail. We did packet captures and found that at the 5 minute mark, the ASA was dropping the ACK packets being sent from client server. The central server would see no packets coming and keep trying to retransmit. The client would get the retransmits, send an ACK, ASA would drop it - rinse/repeat.
While investigating the captures, we found that at the 5 min mark, the client server is resetting the TSVal on the ACK packet to some low number.
You can see in the below packet capture that packet 101, the client sends an ACK with a TSVal of 4294962488. The server then pushes more data ( packet 102 ), but on packet 103, the client now responds with an ACK with TSVal set to 196.
No. Time Timestamp Source Destination Protocol Length Info
96 2017-07-11 15:16:04.717785 22:16:04.717785 10.158.35.162 10.153.195.227 TCP 101 4506 → 38208 [PSH, ACK] Seq=3089107029 Ack=2031069343 Win=29056 Len=35 TSval=1400815609 TSecr=4294947477
97 2017-07-11 15:16:04.717802 22:16:04.717802 10.153.195.227 10.158.35.162 TCP 66 38208 → 4506 [ACK] Seq=2031069343 Ack=3089107064 Win=29312 Len=0 TSval=4294952481 TSecr=1400815609
98 2017-07-11 15:16:09.721130 22:16:09.721130 10.158.35.162 10.153.195.227 TCP 101 4506 → 38208 [PSH, ACK] Seq=3089107064 Ack=2031069343 Win=29056 Len=35 TSval=1400820612 TSecr=4294952481
99 2017-07-11 15:16:09.721152 22:16:09.721152 10.153.195.227 10.158.35.162 TCP 66 38208 → 4506 [ACK] Seq=2031069343 Ack=3089107099 Win=29312 Len=0 TSval=4294957485 TSecr=1400820612
100 2017-07-11 15:16:14.724742 22:16:14.724742 10.158.35.162 10.153.195.227 TCP 101 4506 → 38208 [PSH, ACK] Seq=3089107099 Ack=2031069343 Win=29056 Len=35 TSval=1400825616 TSecr=4294957485
101 2017-07-11 15:16:14.724757 22:16:14.724757 10.153.195.227 10.158.35.162 TCP 66 38208 → 4506 [ACK] Seq=2031069343 Ack=3089107134 Win=29312 Len=0 TSval=4294962488 TSecr=1400825616
102 2017-07-11 15:16:19.728187 22:16:19.728187 10.158.35.162 10.153.195.227 TCP 101 4506 → 38208 [PSH, ACK] Seq=3089107134 Ack=2031069343 Win=29056 Len=35 TSval=1400830619 TSecr=4294962488
103 2017-07-11 15:16:19.728207 22:16:19.728207 10.153.195.227 10.158.35.162 TCP 66 38208 → 4506 [ACK] Seq=2031069343 Ack=3089107169 Win=29312 Len=0 TSval=196 TSecr=1400830619
104 2017-07-11 15:16:19.728556 22:16:19.728556 10.158.35.162 10.153.195.227 TCP 66 [TCP Dup ACK 2#1] 4506 → 38208 [PSH, ACK] Seq=3089107169 Ack=2031069343 Win=29056 Len=0 TSval=1400830619 TSecr=4294962488
105 2017-07-11 15:16:19.928307 22:16:19.928307 10.158.35.162 10.153.195.227 TCP 101 [TCP Spurious Retransmission] 4506 → 38208 [PSH, ACK] Seq=3089107134 Ack=2031069343 Win=29056 Len=35 TSval=1400830820 TSecr=4294962488
106 2017-07-11 15:16:19.928319 22:16:19.928319 10.153.195.227 10.158.35.162 TCP 78 [TCP Dup ACK 103#1] 38208 → 4506 [ACK] Seq=2031069343 Ack=3089107169 Win=29312 Len=0 TSval=396 TSecr=1400830820 SLE=3089107134 SRE=3089107169
107 2017-07-11 15:16:19.928702 22:16:19.928702 10.158.35.162 10.153.195.227 TCP 66 [TCP Dup ACK 2#2] 4506 → 38208 [PSH, ACK] Seq=3089107169 Ack=2031069343 Win=29056 Len=0 TSval=1400830820 TSecr=4294962488
The ASA then considers this malformed and drops it, since the TSVal shouldn't decrement.
We experimented with disabling tcp_timestamps via /proc/sys/net/ipv4/tcp_timestamps. This works and the issue goes away, however disabling timestamps may have other consequences with other application traffic.
Besides - the question still begs, why would the TSVal reset to such a low number after 5 minutes of the host being online? This only happens on our RHEL7u2 systems - our RHEL6 boxes do not experience this issue.
Any thoughts/help would be appreciated.