Top tag
Next privilege 300 Rep.
Create new tags
1 5
~32k people reached

  • 0 posts edited
  • 0 helpful flags
  • 15 votes cast
comment How often does asymmetric latency actually happen?
Yes, without a trusted clock, you can only estimate the one-way latency from the round-trip-latency. If someone is uploading a file at the same time as streaming a video, that estimate may be too high. If they are downloading a file while streaming a video, it may be too low. (BTW, a true Broadcast Television situation is a different scenario: no round-trip network exists.)
comment How often does asymmetric latency actually happen?
Anything where the flow of data is time sensitive and essentially one-way. In live video, for example, the delay between the live event and when the viewer sees it needs to be minimized. But without clock synchronicity, you can only measure the total round-trip time. Congestion control in data transfer also cares about which direction of the path has rising latency: if its the side you are sending on then you are experiencing congestion, but if its the other side then you are not. The only way to measure one-way latency is to have synchronized and trusted clocks on both ends.
comment connection hanging
If possible, try hard-coding the known-good IP address in the script and see if that helps. If that fixes the problem, then its a DNS issue. Most likely either a bad name server in /etc/resolv.conf or improper caching somewhere.
comment How long does it take to fsck a volume?
That is certainly a possibility, although I would think relocation on that scale would produce some I/O errors. Based on the very slow baseline of 80 megabits per second, I was assuming the test was run on an active system. So... are there I/O errors in the system log, how were the hdparm tests performed, and were the results in "megabits" or "megabytes" per second?
comment Exhausting Linux machine TCP socket limit (~70k)?
TIME_WAIT is a factor simply because any port in that state is unavailable. If there is a spike in traffic, the number of TIME_WAITs could become so high that there are not enough free ports available for new connections. That would cause a burst of already in use errors. But because your connection times are short, samples with netstat may not capture that moment. Given all the circumstances, spreading across more nodes/addresses may be the most practical solution.
comment Exhausting Linux machine TCP socket limit (~70k)?
Wow, that's a lot of TIME_WAITs for a 4 second timeout. I see you have net.ipv4.tcp_tw_reuse set. You could also try net.ipv4.tcp_tw_recycle.…… However, if you have access to the source code I suggest examining how the TCP sessions are being closed. A TIME_WAIT state usually occurs when a connection is closed while data is still in transit or not cleanly closed.
comment Exhausting Linux machine TCP socket limit (~70k)?
Another possibility is that the connections are not dropping cleanly, causing ports to be left in a TIME_WAIT state. How quickly are these connections cycling? In other words, how long does a typical TCP session last?