Strange problem. I have server "ALPHA" in Dallas and server "BETA" in Hong Kong, about 195ms ping times apart from each other. ALPHA is connected at 1gbps full, and BETA at 10mbps full. Occasionally when they communicate to each other over TCP, the connection hangs. Eventually ALPHA gives up and closes the connection, but BETA still sees the connection as established and waits for a long time until it eventually times out.
wget -O /dev/null ALPHA:50001/1mb.test
--2013-05-19 02:45:54-- http://ALPHA:50001/1mb.test Resolving dfw... ip.address.redacted Connecting to ALPHA|ip.address.redacted|:50001... connected. HTTP request sent, awaiting response... 200 OK Length: 1000000 (977K) [application/octet-stream] Saving to: `/dev/null'
and it freezes at 0 bytes received. Note that it did get a 200 OK from the server, so they did seem to handshake successfully. To eliminate wget as part of the problem, I did a telnet:
telnet ALPHA 50001 Trying ip.address.redacted... Connected to ALPHA. Escape character is '^]'. GET /1mb.test HTTP/1.0 HTTP/1.1 200 OK Server: nginx Date: Sun, 19 May 2013 22:43:48 GMT Content-Type: application/octet-stream Content-Length: 1000000 Last-Modified: Sat, 21 Jan 2012 21:47:29 GMT Connection: close ETag: "4f1b3271-f4240" Accept-Ranges: bytes
It also freezes at this point.
Both servers do have iptables rules of
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
Here is a tcpdump that was running on ALPHA during the above wget freeze. http://pastebin.com/axX7ap6T
This problem happens only occasionally. Sometimes it works just fine. Actually, if I run about 3-5 wget's, usually everything starts working on about the fourth or fifth try.
BETA can communicate with GAMMA, DELTA, and my other servers without problem. Only seems to happen when talking to ALPHA. ICMP pings show no packet losses, but even if there was packet loss, it would slow it down, not disconnect. Both machines have the same CentOS 6.4 64-bit OS, kernel 2.6.32-358.6.2.el6.x86_64. Nothing useful logged in /var/log files. ALPHA is listening on 50001 using nginx-1.4.1-1.el6.ngx.x86_64. The nginx access log file shows a 200 (request OK) 43440 bytes delivered during the frozen connection.
The problem does not seem to be nginx related, since I have had an ssh connection also freeze in the middle of communicating on the same day. Other servers connect to that nginx and never had any problems.
In summary, this appears to be what is happening in this particular communication:
BETA -> ALPHA "SYN!" ALPHA -> BETA "SYN Acknowledgement" BETA -> ALPHA "ACK" ALPHA -> BETA "Established" BETA -> ALPHA "GET /file HTTP/1.0" ALPHA -> BETA "200 OK, here's a bunch of data" BETA freezes, gets no data.
Anyone seen a situation like this?