2

On a customer server running Apache 2.2 on Windows server 2012 we're noticing that from time to time, some requests to the server never finish. Using wireshark I've found a bunch of duplicate ACKs get sent to the server as soon as it starts answering the request and after a few seconds a couple of retransmissions are received from the server.

The network setup is really basic with a server and some clients connected to a switch using UTP cables.

I'm not really sure what to make of this. I'm considering asking them to try different cabling, switch and/or NIC's, but would like a second opinion on that.

 1 0.000000000 192.168.1.103 -> 192.168.2.100 TCP 66 52011?81 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
 2 0.000742000 192.168.2.100 -> 192.168.1.103 TCP 66 81?52011 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
 3 0.000782000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=1 Ack=1 Win=65536 Len=0
 4 0.001646000 192.168.1.103 -> 192.168.2.100 HTTP 606 GET /symfony/web/app.php/legacy/mutaties/afroepen.php?afroepid=16250 HTTP/1.1
 5 0.002353000 192.168.2.100 -> 192.168.1.103 TCP 60 81?52011 [ACK] Seq=1 Ack=553 Win=65536 Len=0
 6 0.747171000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
 7 0.747172000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
 8 0.747246000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=553 Ack=2921 Win=65536 Len=0
 9 0.747504000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
10 0.747507000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
11 0.747562000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [ACK] Seq=553 Ack=5841 Win=65536 Len=0
12 0.748241000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
13 0.748242000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
14 0.748243000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
15 0.748244000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
16 0.748319000 192.168.1.103 -> 192.168.2.100 TCP 66 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=10221
17 0.748338000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#1] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=11681
18 0.748593000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
19 0.748594000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
20 0.748595000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
21 0.748624000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#2] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=13141
22 0.748643000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#3] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=14601
23 0.748654000 192.168.1.103 -> 192.168.2.100 TCP 66 [TCP Dup ACK 16#4] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=8761 SRE=16061
24 0.748965000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
25 0.748966000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
26 0.748967000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
27 0.748999000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#5] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=18981 SLE=8761 SRE=16061
28 0.749014000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#6] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=20441 SLE=8761 SRE=16061
29 0.749249000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
30 0.749250000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
31 0.749275000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#7] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=21901 SLE=8761 SRE=16061
32 0.749289000 192.168.1.103 -> 192.168.2.100 TCP 74 [TCP Dup ACK 16#8] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=17521 SRE=23361 SLE=8761 SRE=16061
33 0.749578000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Fast Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
34 0.749581000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
35 0.749907000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
36 0.749909000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
37 0.749910000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=16061 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
38 0.749939000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#9] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=26281 SLE=17521 SRE=23361 SLE=8761 SRE=16061
39 0.749958000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#10] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=27741 SLE=17521 SRE=23361 SLE=8761 SRE=16061
40 0.750240000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
41 0.750241000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
42 0.750269000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#11] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=29201 SLE=17521 SRE=23361 SLE=8761 SRE=16061
43 0.750289000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#12] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=30661 SLE=17521 SRE=23361 SLE=8761 SRE=16061
44 0.750622000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
45 0.750624000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
46 0.750649000 192.168.1.103 -> 192.168.2.100 TCP 82 [TCP Dup ACK 16#13] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
47 0.751016000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=23361 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
48 0.751017000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
49 0.751059000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#14] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=35041 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
50 0.751549000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
51 0.751551000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
52 0.751567000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#15] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=36501 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
53 0.751582000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#16] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=37961 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
54 0.752392000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
55 0.752393000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
56 0.752394000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=32121 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
57 0.752396000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
58 0.752406000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#17] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361 SLE=8761 SRE=16061
59 0.752433000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#18] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=42341 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
60 0.753551000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
61 0.753553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
62 0.753571000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#19] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=43801 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
63 0.753588000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#20] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=45261 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
64 0.754550000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
65 0.754553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Out-Of-Order] 81?52011 [PSH, ACK] Seq=39421 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
66 0.754553000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP segment of a reassembled PDU]
67 0.754572000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#21] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=46721 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
68 0.754610000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#22] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=48181 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
69 1.055597000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
70 1.664583000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
71 2.867842000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
72 5.274379000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
73 10.071711000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]
74 19.685995000 192.168.2.100 -> 192.168.1.103 TCP 60 81?52011 [RST, ACK] Seq=8761 Ack=553 Win=0 Len=0
75 19.686030000 192.168.1.103 -> 192.168.2.100 TCP 90 [TCP Dup ACK 16#23] 52011?81 [ACK] Seq=553 Ack=7301 Win=65536 Len=0 SLE=40881 SRE=48181 SLE=33581 SRE=39421 SLE=24821 SRE=32121 SLE=17521 SRE=23361
76 45.752325000 192.168.1.103 -> 192.168.2.100 TCP 55 [TCP Keep-Alive] [TCP Window Full] 52011?81 [ACK] Seq=552 Ack=7301 Win=65536 Len=1
77 58.686133000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
78 58.985830000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
79 59.585853000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
80 60.785894000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
81 63.185944000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
82 67.988211000 192.168.1.103 -> 192.168.2.100 TCP 54 [TCP Retransmission] 52011?81 [FIN, ACK] Seq=553 Ack=7301 Win=65536 Len=0
83 77.582500000 192.168.1.103 -> 192.168.2.100 TCP 54 52011?81 [RST, ACK] Seq=554 Ack=7301 Win=0 Len=0

4 Answers 4

2

That log screams, to me, of either a cabling problem or a duplex mismatch between a device and the switch. I'd consider forcing the speed / duplex on the NICs on two devices (the server and a client) to 100Base-TX half-duplex and working up from there.

Shoddy patch cables, bad NIC drivers, or a failing switch could cause these behaviors, too. It's important to try to isolate the problem by taking a systematic approach-- changing one thing at a time and testing before making further changes.

1

I was seeing similar issue when connection was initiated from a Windows 2012 Server R2 Core guest on VirtualBox 5.1 over bridged network with Centos 6 as VirtualBox host. While i am still investigating the problem, disabling ECN (https://en.wikipedia.org/wiki/Explicit_Congestion_Notification) on Windows machine solved my issue. Your trace shows that ECN is disabled so it is not the answer but I thought it might help others.

EDIT: Disabling GRO (Generic Receive Offload) also work make it work:

ethtool -K eth0 gro off
0

I would say you only lost one packet there. The client is doing selective acknowledgement and getting all the data except one packet. Wireshark says it is a duplicated ack, but it's not. Finally the server decides to send the lost packet back at frame 69 (0.3s after finishing sending the HTTP response), tries 4 times and decides to kill the connection due to time-out (client unresponsive). The client finally decides to acknowledge it got the lost packet, way after expected.

Client side looks weird to me. Rule out any 3rd party drivers on the network stack (e.g. antivirus or other security software). Also try to rule out any devices in between that could be doing packet inspection. Although based on the time deltas I think the capture was taken on the client so that pretty much rules out the client packets being modified on its way by a third party device.

0

the retransmisson of the missing frame is already in Frame 33 - you can tell by the seq=7301.

33 0.749578000 192.168.2.100 -> 192.168.1.103 TCP 1514 [TCP Fast Retransmission] 81?52011 [PSH, ACK] Seq=7301 Ack=553 Win=65536 Len=1460

Also the duplicate Ack is present - cause there are multiple Ack with Ack=7301 cause this is the missing frame.

Just the client isn't recognising this frame.

I would guess that the trace has been taken on the server - seeing outgoing packets - but they may not be received at the client side - (indicated by the dup ack) packets.

A trace of the situation at the client side would probably show that these frames never arived. So i would say - the already mentioned cabling or duplex settings would be a good guess for the cause.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .