Sign up ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Strange problem. I have server "ALPHA" in Dallas and server "BETA" in Hong Kong, about 195ms ping times apart from each other. ALPHA is connected at 1gbps full, and BETA at 10mbps full. Occasionally when they communicate to each other over TCP, the connection hangs. Eventually ALPHA gives up and closes the connection, but BETA still sees the connection as established and waits for a long time until it eventually times out.


On BETA, wget -O /dev/null ALPHA:50001/1mb.test

--2013-05-19 02:45:54--  http://ALPHA:50001/1mb.test Resolving dfw... ip.address.redacted Connecting to ALPHA|ip.address.redacted|:50001... connected. HTTP request sent, awaiting response... 200 OK Length: 1000000 (977K)
[application/octet-stream] Saving to: `/dev/null'

and it freezes at 0 bytes received. Note that it did get a 200 OK from the server, so they did seem to handshake successfully. To eliminate wget as part of the problem, I did a telnet:

telnet ALPHA 50001
Trying ip.address.redacted...
Connected to ALPHA.
Escape character is '^]'.
GET /1mb.test HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 19 May 2013 22:43:48 GMT
Content-Type: application/octet-stream
Content-Length: 1000000
Last-Modified: Sat, 21 Jan 2012 21:47:29 GMT
Connection: close
ETag: "4f1b3271-f4240"
Accept-Ranges: bytes

It also freezes at this point.

Both servers do have iptables rules of -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Here is a tcpdump that was running on ALPHA during the above wget freeze.

This problem happens only occasionally. Sometimes it works just fine. Actually, if I run about 3-5 wget's, usually everything starts working on about the fourth or fifth try.

BETA can communicate with GAMMA, DELTA, and my other servers without problem. Only seems to happen when talking to ALPHA. ICMP pings show no packet losses, but even if there was packet loss, it would slow it down, not disconnect. Both machines have the same CentOS 6.4 64-bit OS, kernel 2.6.32-358.6.2.el6.x86_64. Nothing useful logged in /var/log files. ALPHA is listening on 50001 using nginx-1.4.1-1.el6.ngx.x86_64. The nginx access log file shows a 200 (request OK) 43440 bytes delivered during the frozen connection.

The problem does not seem to be nginx related, since I have had an ssh connection also freeze in the middle of communicating on the same day. Other servers connect to that nginx and never had any problems.

In summary, this appears to be what is happening in this particular communication:

ALPHA -> BETA "SYN Acknowledgement"
ALPHA -> BETA "Established"
BETA -> ALPHA "GET /file HTTP/1.0"
ALPHA -> BETA "200 OK, here's a bunch of data"
BETA freezes, gets no data.

Anyone seen a situation like this?

share|improve this question
Maybe an MTU/MSS problem? The small initial packets get through, the big ones (data transfer) get killed somewhere? –  Hauke Laging May 19 '13 at 23:17
MTU is 1500 on both (the default). I just set BETA's MTU to 576, and not seeing the problem right now, and will see if that helps. Still, it's strange though. –  Crash Override May 19 '13 at 23:34
Try with higher values first, like ~1400. 576 is just to small. –  Sandor Marton May 19 '13 at 23:37
Ok, will try 1400 for a while and see what happens. –  Crash Override May 19 '13 at 23:39

3 Answers 3

up vote 6 down vote accepted

Maybe an MTU/MSS problem? The small initial packets get through, the big ones (data transfer) get killed somewhere?

A work-around is to reduce the MTU (on both sides) a bit, from 1500 to 1450 e.g. Or to use the Netfilter target TCPMSS just for the affected connections (or differently for separate connections).

But the real solution is to get path MTU discovery working again. So check whether ICMP packets (fragmentation needed: Type 3, Code 4) are blocked. Use tcpdump -i $ifname -n icmp to see whether such packets arrive.

share|improve this answer
When would those ICMP packets arrive, during the tcp request? I just ran a few tests with MTU 1400 on BETA and saw no tcpdump icmp data on either host. I do have an iptables rule that allows ALL ICMP between these two machines: -A TRUSTED -p icmp -j ACCEPT on both sides restricted to privileged IPs. BETA blocks all ICMP from other IPs. –  Crash Override May 20 '13 at 1:20
Just did the same thing with MTU 1500 on BETA, still no tcpdump icmp on either side. As soon as I changed BETA's MTU back to 1500, the wget instantly froze like before. Back to 1400 MTU, and it works. Glad we're on the right track! Maybe some ICMP blockage somewhere is causing this problem. I've never seen an issue like this before. –  Crash Override May 20 '13 at 1:31

There is indeed defragmentation occurring. That's what IPTABLES does to inspect packets. It must reassemble them in order to validate content.

In my case, I would see periodic ping dropouts and web browser stalls while I was web browsing on a CentOS 7 system, or my PuttY sessions from windows would mysteriously hang -- permanently.

I used a variant of method Mick so kindly provided to determine the MTU size. The problem with that method is that if you've set the MTU size small, you get a false reading on the fragmentation. So set the size large (jumbo frame/9000) and then begin the ping probes.

After setting MTU to 1472 (what I determined was the correct value) the ping problem when browsing went away. Hopefully when I next ssh from windows, that will work too.

share|improve this answer

My first guess is that it's a TCP Window scaling issue:

Disable it on both ends and see if it the problem goes away. Another option (as Hauke suggested) is there's defragmention occurring.

You can use ping to check for packet de-fragmentation, a simple HOWTO is here:

Edit: missed a word.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.