I’m trying to improve my TCP throughput over a “high delay network”, with some packet drops, between Linux machines.

I set tcp_mem, tcp_wmem and tcp_rmem to “8192 20530752 20530752”.
I set rmem_max, wmem_max, rmem_default and wmem_default to “20530752”.
I set netdev_max_backlog and txqueuelen to 50000.
I set tcp_congestion_control to “scalable”.

I’m using “nist” (cnistnet) to simulate a delay of 50ms (25ms for each direction) and drop of 0.5% (0.25% for each direction), and the BW I reach is about 7.48mbps.
Here is a table of the results (using iperf to measure the rates):

     |   0ms    |   50ms
0%   | 710mbps  | 276mbps
0.5% | 181mbps  | 7.48mbps

I didn’t expect the latency to have so much effect on the throughput (not with these big TCP windows). I also didn’t expect the drops to have so much effect. Especially not with the “scalable” algorithm, as it’s “congestion window” is supposed to recovery very quickly from packet drops.

I used tcpdump & sar (part of sysstat) to try to see what’s going on. I didn’t see anything suspicious in sar’s reports. And in tcp dump I see:
“rexmt data pkts” = 88 (out of 8183 sent)
“rexmt data bytes” = 127,424 bytes (out of 44,649,104 bytes sent)
“avg owin” = 135,964 bytes.
“avg retr time” = 53.5 ms.

Is this the best TCP can do? Is there no other algorithm which can achieve better throughput in this scenario? Is my scenario out of the ordinary (I think that 50ms delay with 0.5 drop are normal WAN properties)?

  • 0.5% packet loss is not small, it is HUGH. Ridiculously hugh. Non working hardware level of loss. – TomTom Oct 2 '11 at 13:20
  • @TomTom However, it is reasonable for flaky RF network connections. – sysadmin1138 Oct 2 '11 at 13:36
  • Nope, still in the area you call support and get them fix their crap setup. – TomTom Oct 2 '11 at 15:28
  • You didn't say if TCP window scaling was enabled (not much point allocating lots of memory for the buffers otherwise) and working end-to-end. – symcbean Oct 3 '11 at 11:05
  • Window scaling was enabled and proven working in a different scenario (with 0% packet drop) – Gil Barash Oct 4 '11 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.