Strange problem. I have server "ALPHA" in Dallas and server "BETA" in Hong Kong, about 195ms ping times apart from each other. ALPHA is connected at 1gbps full, and BETA at 10mbps full. Occasionally when they communicate to each other over TCP, the connection hangs. Eventually ALPHA gives up and closes the connection, but BETA still sees the connection as established and waits for a long time until it eventually times out.


On BETA, wget -O /dev/null ALPHA:50001/1mb.test

--2013-05-19 02:45:54--  http://ALPHA:50001/1mb.test Resolving dfw... ip.address.redacted Connecting to ALPHA|ip.address.redacted|:50001... connected. HTTP request sent, awaiting response... 200 OK Length: 1000000 (977K)
[application/octet-stream] Saving to: `/dev/null'

and it freezes at 0 bytes received. Note that it did get a 200 OK from the server, so they did seem to handshake successfully. To eliminate wget as part of the problem, I did a telnet:

telnet ALPHA 50001
Trying ip.address.redacted...
Connected to ALPHA.
Escape character is '^]'.
GET /1mb.test HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 19 May 2013 22:43:48 GMT
Content-Type: application/octet-stream
Content-Length: 1000000
Last-Modified: Sat, 21 Jan 2012 21:47:29 GMT
Connection: close
ETag: "4f1b3271-f4240"
Accept-Ranges: bytes

It also freezes at this point.

Both servers do have iptables rules of -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Here is a tcpdump that was running on ALPHA during the above wget freeze. http://pastebin.com/axX7ap6T

This problem happens only occasionally. Sometimes it works just fine. Actually, if I run about 3-5 wget's, usually everything starts working on about the fourth or fifth try.

BETA can communicate with GAMMA, DELTA, and my other servers without problem. Only seems to happen when talking to ALPHA. ICMP pings show no packet losses, but even if there was packet loss, it would slow it down, not disconnect. Both machines have the same CentOS 6.4 64-bit OS, kernel 2.6.32-358.6.2.el6.x86_64. Nothing useful logged in /var/log files. ALPHA is listening on 50001 using nginx-1.4.1-1.el6.ngx.x86_64. The nginx access log file shows a 200 (request OK) 43440 bytes delivered during the frozen connection.

The problem does not seem to be nginx related, since I have had an ssh connection also freeze in the middle of communicating on the same day. Other servers connect to that nginx and never had any problems.

In summary, this appears to be what is happening in this particular communication:

ALPHA -> BETA "SYN Acknowledgement"
ALPHA -> BETA "Established"
BETA -> ALPHA "GET /file HTTP/1.0"
ALPHA -> BETA "200 OK, here's a bunch of data"
BETA freezes, gets no data.

Anyone seen a situation like this?

  • Maybe an MTU/MSS problem? The small initial packets get through, the big ones (data transfer) get killed somewhere? May 19, 2013 at 23:17
  • MTU is 1500 on both (the default). I just set BETA's MTU to 576, and not seeing the problem right now, and will see if that helps. Still, it's strange though. May 19, 2013 at 23:34
  • 1
    Try with higher values first, like ~1400. 576 is just to small. May 19, 2013 at 23:37
  • Ok, will try 1400 for a while and see what happens. May 19, 2013 at 23:39

3 Answers 3


Maybe an MTU/MSS problem? The small initial packets get through, the big ones (data transfer) get killed somewhere?

A work-around is to reduce the MTU (on both sides) a bit, from 1500 to 1450 e.g. Or to use the Netfilter target TCPMSS just for the affected connections (or differently for separate connections).

But the real solution is to get path MTU discovery working again. So check whether ICMP packets (fragmentation needed: Type 3, Code 4) are blocked. Use tcpdump -i $ifname -n icmp to see whether such packets arrive.

  • When would those ICMP packets arrive, during the tcp request? I just ran a few tests with MTU 1400 on BETA and saw no tcpdump icmp data on either host. I do have an iptables rule that allows ALL ICMP between these two machines: -A TRUSTED -p icmp -j ACCEPT on both sides restricted to privileged IPs. BETA blocks all ICMP from other IPs. May 20, 2013 at 1:20
  • Just did the same thing with MTU 1500 on BETA, still no tcpdump icmp on either side. As soon as I changed BETA's MTU back to 1500, the wget instantly froze like before. Back to 1400 MTU, and it works. Glad we're on the right track! Maybe some ICMP blockage somewhere is causing this problem. I've never seen an issue like this before. May 20, 2013 at 1:31

There is indeed defragmentation occurring. That's what IPTABLES does to inspect packets. It must reassemble them in order to validate content.

In my case, I would see periodic ping dropouts and web browser stalls while I was web browsing on a CentOS 7 system, or my PuttY sessions from windows would mysteriously hang -- permanently.

I used a variant of method Mick so kindly provided to determine the MTU size. The problem with that method is that if you've set the MTU size small, you get a false reading on the fragmentation. So set the size large (jumbo frame/9000) and then begin the ping probes.

After setting MTU to 1472 (what I determined was the correct value) the ping problem when browsing went away. Hopefully when I next ssh from windows, that will work too.


My first guess is that it's a TCP Window scaling issue:


Disable it on both ends and see if it the problem goes away. Another option (as Hauke suggested) is there's defragmention occurring.

You can use ping to check for packet de-fragmentation, a simple HOWTO is here:


Edit: missed a word.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .