Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

While troubleshooting another thing, I've noticed an odd pattern of TCP closes.

Packets: http://www.cloudshark.org/captures/7590ec4e6bef

What's happening is that the last few packets of the close sequence are getting retransmitted for some odd reason. The pattern is in the cloudshark link, but for posterity here is a summary:

  1. Source -> Syn
  2. Dest -> Ack
  3. Source -> SynAck
  4. Data
  5. Data
  6. Source -> Fin/Ack
  7. Dest -> Psh/Ack (6)
  8. Dest -> Fin/Ack
  9. Source -> Ack (7)
  10. Source -> Ack (8)
  11. [At this point the connection should be closed on both sides. But it isn't.]
  12. [+200ms] Dest -> Fin/Ack
  13. Source -> Ack (8 & 12)

Something on the Destination system is waiting 200ms before reissuing the Fin/Ack packet. This is pretty consistent across multiple transactions. More damning, this pattern replicates on both sides of the transaction: it shows up in captures on both hosts. It isn't as simple as the Fin/Ack packet getting dropped somewhere and getting retransmitted. Or maybe it is getting dropped, but on a level above where tcpdump operates.

The 200ms delay makes me think TCP Delayed Ack is involved here, but I'm having trouble figuring out why it would.

Is above tcpdump even a thing?

Is this a normal connection pattern for RHEL6 systems?

share|improve this question

1 Answer 1

up vote 5 down vote accepted

Note that the PSH/ACK packet in frame #2 of your capture is carrying 37 bytes of data. It is not a response to the FIN request from 10.232.16.133. The response follows in frame #3, the ACK to that is in frame #5.

What seems to be happening now is that this last ACK does not make it to the destination, so the FIN/ACK sent in frame #3 is retransmitted in frame #6. The ~200 ms delay you are seeing here is not the delayed ack but more likely the retransmission timeout. You should be able to verify this by looking at the packet capture from the other side of the connection where you never should be seeing the last ACK arriving.

If you see this behavior consistently on all connections (and possibly in both directions), an explanation crossing my mind is that some state-keeping packet filter in the path is swallowing the last ACK.

share|improve this answer
    
Linux firewalls often consider the last ACK invalid. –  Michael Hampton Feb 20 at 23:31
    
Oo, useful. The thing that made go buwhuh? is that the packet trace looks the same on both sides. On the host via TCPDUMP I see the retransmit and ack, on the perimeter network device via proprietary methods I also see the ack, retransmit and ack. The host here isn't using iptables, it's using a TrendMicro centrally managed firewall. I want to blame it. –  Blue Warrior NFB Feb 21 at 1:28
    
@BlueWarriorNFB well, switch it off (in a lab) and see if this changes matters. If the packet filter is host-based, you would likely see the last ACK packet in the receiver trace, as blocking/filtering happens up on the packet's way to the TCP stack. –  the-wabbit Feb 21 at 6:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.