246 reputation
bio website DataExpedition.com
visits member for 3 years
seen Dec 28 '14 at 13:07

comment connection hanging
If possible, try hard-coding the known-good IP address in the script and see if that helps. If that fixes the problem, then its a DNS issue. Most likely either a bad name server in /etc/resolv.conf or improper caching somewhere.
comment How long does it take to fsck a volume?
That is certainly a possibility, although I would think relocation on that scale would produce some I/O errors. Based on the very slow baseline of 80 megabits per second, I was assuming the test was run on an active system. So... are there I/O errors in the system log, how were the hdparm tests performed, and were the results in "megabits" or "megabytes" per second?
comment Exhausting Linux machine TCP socket limit (~70k)?
TIME_WAIT is a factor simply because any port in that state is unavailable. If there is a spike in traffic, the number of TIME_WAITs could become so high that there are not enough free ports available for new connections. That would cause a burst of already in use errors. But because your connection times are short, samples with netstat may not capture that moment. Given all the circumstances, spreading across more nodes/addresses may be the most practical solution.
comment Exhausting Linux machine TCP socket limit (~70k)?
Wow, that's a lot of TIME_WAITs for a 4 second timeout. I see you have net.ipv4.tcp_tw_reuse set. You could also try net.ipv4.tcp_tw_recycle. serverfault.com/questions/342741/… stackoverflow.com/questions/6426253/… However, if you have access to the source code I suggest examining how the TCP sessions are being closed. A TIME_WAIT state usually occurs when a connection is closed while data is still in transit or not cleanly closed.
comment Exhausting Linux machine TCP socket limit (~70k)?
Another possibility is that the connections are not dropping cleanly, causing ports to be left in a TIME_WAIT state. How quickly are these connections cycling? In other words, how long does a typical TCP session last?