TCP TSVal reset at 5 min after reboot on active connections

Question

I'm experiencing an issue on our RHEL7 boxes that are talking to a central server sitting behind a Cisco ASA firewall.

These machines boot and start up an agent that establishes and keeps alive a connection to the central server. The central server then periodically sends traffic down that tunnel to the client. At the 5 minute mark, we were seeing the agents disconnect and were unable to send traffic over the connection unless we restarted the connection, after which point it worked fine.

Further research showed that it wasn't just this application/agent, and in fact we could replicate it with 'nc', without fail. We did packet captures and found that at the 5 minute mark, the ASA was dropping the ACK packets being sent from client server. The central server would see no packets coming and keep trying to retransmit. The client would get the retransmits, send an ACK, ASA would drop it - rinse/repeat.

While investigating the captures, we found that at the 5 min mark, the client server is resetting the TSVal on the ACK packet to some low number.

You can see in the below packet capture that packet 101, the client sends an ACK with a TSVal of 4294962488. The server then pushes more data ( packet 102 ), but on packet 103, the client now responds with an ACK with TSVal set to 196.

No.     Time                          Timestamp          Source                Destination           Protocol Length Info
     96 2017-07-11 15:16:04.717785    22:16:04.717785    10.158.35.162         10.153.195.227        TCP      101    4506 → 38208 [PSH, ACK] Seq=3089107029 Ack=2031069343 Win=29056 Len=35 TSval=1400815609 TSecr=4294947477
     97 2017-07-11 15:16:04.717802    22:16:04.717802    10.153.195.227        10.158.35.162         TCP      66     38208 → 4506 [ACK] Seq=2031069343 Ack=3089107064 Win=29312 Len=0 TSval=4294952481 TSecr=1400815609
     98 2017-07-11 15:16:09.721130    22:16:09.721130    10.158.35.162         10.153.195.227        TCP      101    4506 → 38208 [PSH, ACK] Seq=3089107064 Ack=2031069343 Win=29056 Len=35 TSval=1400820612 TSecr=4294952481
     99 2017-07-11 15:16:09.721152    22:16:09.721152    10.153.195.227        10.158.35.162         TCP      66     38208 → 4506 [ACK] Seq=2031069343 Ack=3089107099 Win=29312 Len=0 TSval=4294957485 TSecr=1400820612
    100 2017-07-11 15:16:14.724742    22:16:14.724742    10.158.35.162         10.153.195.227        TCP      101    4506 → 38208 [PSH, ACK] Seq=3089107099 Ack=2031069343 Win=29056 Len=35 TSval=1400825616 TSecr=4294957485
    101 2017-07-11 15:16:14.724757    22:16:14.724757    10.153.195.227        10.158.35.162         TCP      66     38208 → 4506 [ACK] Seq=2031069343 Ack=3089107134 Win=29312 Len=0 TSval=4294962488 TSecr=1400825616
    102 2017-07-11 15:16:19.728187    22:16:19.728187    10.158.35.162         10.153.195.227        TCP      101    4506 → 38208 [PSH, ACK] Seq=3089107134 Ack=2031069343 Win=29056 Len=35 TSval=1400830619 TSecr=4294962488
    103 2017-07-11 15:16:19.728207    22:16:19.728207    10.153.195.227        10.158.35.162         TCP      66     38208 → 4506 [ACK] Seq=2031069343 Ack=3089107169 Win=29312 Len=0 TSval=196 TSecr=1400830619
    104 2017-07-11 15:16:19.728556    22:16:19.728556    10.158.35.162         10.153.195.227        TCP      66     [TCP Dup ACK 2#1] 4506 → 38208 [PSH, ACK] Seq=3089107169 Ack=2031069343 Win=29056 Len=0 TSval=1400830619 TSecr=4294962488
    105 2017-07-11 15:16:19.928307    22:16:19.928307    10.158.35.162         10.153.195.227        TCP      101    [TCP Spurious Retransmission] 4506 → 38208 [PSH, ACK] Seq=3089107134 Ack=2031069343 Win=29056 Len=35 TSval=1400830820 TSecr=4294962488
    106 2017-07-11 15:16:19.928319    22:16:19.928319    10.153.195.227        10.158.35.162         TCP      78     [TCP Dup ACK 103#1] 38208 → 4506 [ACK] Seq=2031069343 Ack=3089107169 Win=29312 Len=0 TSval=396 TSecr=1400830820 SLE=3089107134 SRE=3089107169
    107 2017-07-11 15:16:19.928702    22:16:19.928702    10.158.35.162         10.153.195.227        TCP      66     [TCP Dup ACK 2#2] 4506 → 38208 [PSH, ACK] Seq=3089107169 Ack=2031069343 Win=29056 Len=0 TSval=1400830820 TSecr=4294962488

The ASA then considers this malformed and drops it, since the TSVal shouldn't decrement.

We experimented with disabling tcp_timestamps via /proc/sys/net/ipv4/tcp_timestamps. This works and the issue goes away, however disabling timestamps may have other consequences with other application traffic.

Besides - the question still begs, why would the TSVal reset to such a low number after 5 minutes of the host being online? This only happens on our RHEL7u2 systems - our RHEL6 boxes do not experience this issue.

Any thoughts/help would be appreciated.

Answer 1

I don’t think the firewall should be dropping these, it should be looking at both sequence number and timestamp together... what I’m guessing happened is that RHEL 7 is now randomizing the timestamps for security reasons, but perhaps the firewall is rather old?”