After the suggestions of the dear commenters, I have looked to see whether an MTU problem could be the cause.
The following was found when trying to connect from "Site A" to "Site B" from a Fedora system. On a Windows system everything is working perfectly fine -- wireshark indicates that outgoing packets' length never exceeds 1158 byte, so the problem is not triggered there.
In brief, if I read this correctly:
- There is an initial successful exchange of small packets.
- A packet with length 1900 is sent. I suppose the network card will
break this up because the MTU for the local network is 1500.
- A router in the ISP network with address 10.10.80.7 tells us to "please
fragment the packet to MTU 1492".
- Wilco! A packet with length 1492 is sent.
- A router in the ISP network with address 10.10.80.7 tells us to
"please fragment the packet to MTU 1492".
- Things go downhill from here.
It looks like I will have to open a ticket with the ISP (which is POST Telecom Luxembourg btw, in case someone googles for similar problems).
It also suggests a remediation. Force the MTU to SITE_A to 1000:
ip route add $SITE_A_IP via $GATEWAY_IP dev $ETHDEV mtu lock 1000
Indeed, this fixes the problem.
Reference info
Use ping
to test MTU behaviour:
ping -c $COUNT -M $MTUDS -s $PPLSZ $HOST
where
COUNT=1
: "One ping only"
MTUDS=do
: MTU discovery strategy is "prohibit fragmentation, even local one" i.e. set the 'DF' (don't fragment) bit (why is this 'do'? dunno). USE THIS.
MTUDS=want
: MTU discovery strategy is "do PMTU discovery, fragment locally when packet size is large" i.e. set the 'DF' bit and fragment locally
MTUDS=dont
: MTU discovery strategy is "don't set the 'DF' bit", i.e. fragment as needed
PPLSZ=1464
: ICMP ping packet payload size in byte.
Use tcpdump
to monitor all ICMP packets and packets from and to "Site A":
tcpdump -vvv -n -nn icmp or '(' host $SITE_A_IP ')'
This is a bit hard to read though.
Watch what the kernel thinks about the MTU to "Site A".
watch ip route get to $SITE_A_IP
Note that a lower MTU than the default will get cached with a TTL of 600 seconds after the first failed ping.
Scenario
Suppose the maximum IP packet size in byte (i.e. the size of the Ethernet payload) is 1492 (this is the case on Amazon EC2), then an interesting ping payload size would be 1465, because the 28 byte used for the IP and ICMP header information would give 1493, one byte pas the maximum.
Then ping -c 1 -M want -s 1465 $HOST_IP
does the following:
On the first ping you get "Frag needed and DF set (mtu = 1492) 100% packet loss". tcpdump
shows echo request part 1 (length 1493) going out and a router of the target network sending back an "ICMP unreachable" with the request to fragment down to MTU 1492. A cached entry with MTU=1492 appears in the kernel route cache.
On subsequent pings you get "1 packets transmitted, 1 received". tcpdump
shows echo request part 1 (length 1492) and echo request part 2 (length 21, offset 1472) and the corresponding echo reply (length 1493).
Or you can use traceroute
# traceroute --mtu SITE_A 1500
Packet size 1500. Traceroute tells us that route 10.10.80.7 has MTU 1492
traceroute to SITE_A (SITE_A_IP), 30 hops max, 1500 byte packets
1 gateway (192.168.10.1) 0.550 ms 0.536 ms 0.393 ms
2 192.168.178.1 (192.168.178.1) 1.458 ms 1.485 ms 1.344 ms
3 10.10.80.7 (10.10.80.7) 4.889 ms F=1492 2.968 ms 4.854 ms
4 10.10.80.7 (10.10.80.7) 4.955 ms !F-1492 3.559 ms !F-1492 5.022 ms !F-1492
Try with 1492: same problem!
traceroute to SITE_A (SITE_A_IP), 30 hops max, 1492 byte packets
1 gateway (192.168.10.1) 0.635 ms 0.554 ms 0.483 ms
2 192.168.178.1 (192.168.178.1) 1.510 ms 1.504 ms 1.311 ms
3 10.10.80.7 (10.10.80.7) 48.305 ms 17.436 ms 5.496 ms
4 10.10.80.7 (10.10.80.7) 5.963 ms !F-1492 6.865 ms !F-1492 4.887 ms !F-1492
Try with 1491: same problem!
traceroute to SITE_A (SITE_A_IP), 30 hops max, 1491 byte packets
1 gateway (192.168.10.1) 0.594 ms 0.650 ms 0.492 ms
2 192.168.178.1 (192.168.178.1) 1.716 ms 1.782 ms 1.580 ms
3 10.10.80.7 (10.10.80.7) 7.327 ms 7.385 ms 4.775 ms
4 10.10.80.7 (10.10.80.7) 5.210 ms !F-1492 5.624 ms !F-1492 4.841 ms !F-1492
Try with 1490: we get through. There is bound to be some off-by-one error in there.
traceroute to SITE_A (SITE_A_IP), 30 hops max, 1490 byte packets
1 gateway (192.168.10.1) 0.616 ms 0.688 ms 0.484 ms
2 192.168.178.1 (192.168.178.1) 1.712 ms 1.853 ms 1.611 ms
3 10.10.80.7 (10.10.80.7) 6.248 ms 7.008 ms 4.995 ms
4 SITE_A_IP.dyn.luxdsl.pt.lu (SITE_A_IP) 12.441 ms !X 9.641 ms !X 9.576 ms !X
Further info of interest: