I have a server with a 10GbE connection to a switch, and 10 clients each with a 1GbE connection to the same switch.

Running nuttcp in parallel on each of the clients, I can push 10 TCP streams of data to the server simultaneously at close to wire speed (i.e. just shy of 100 megabytes per second from all 10 clients simultaneously).

However, when I reverse the direction and send data from the server to the clients -- i.e., 10 TCP streams, one going to each client -- the TCP retransmissions skyrocket and the performance drops to 30, 20, or even 10 megabytes per second per client. I want to get these numbers up, because this traffic pattern is representative of certain applications I care about.

I have verified that my server is capable of saturating a 10GbE link by performing the same experiment over a 10GbE connection to a similar server. I have verified that there are no errors on any of my ports.

Finally, when I forcibly clamp (limit) the receiver's TCP window size, I can get the bandwidth somewhat higher (30-40 megabytes/sec); and if I clamp it extremely low, I can get the retransmissions to zero (with the bandwidth ludicrously low).

Thus I am reasonably confident I am overrunning the buffers in my switch, resulting in packet loss due to congestion. However, I thought TCP's congestion control was supposed to deal with this nicely, eventually stabilizing at something above 50% of wire speed.

So my first question is very simple: Which TCP congestion control algorithm would be best for my situation? There are a ton of them available, but they mostly seem to be targeted at lossy networks or high-bandwidth high-latency networks or wireless networks... None of which apply to my situation.

Second question: Is there anything else I can try?

  • 1
    It would be helpful to know what model of switch. Different switches handle queuing in different ways, and would help narrow down a solution. – scottm32768 Mar 20 '13 at 1:50
  • 2
    Also different switches have different buffer sizes, so knowing the switch model would help eliminate hardware issues from your problem. – cpt_fink Mar 20 '13 at 4:49
  • 1
    Also, the NIC models, drivers, Linux version, kernel, distribution, etc. My answers for a Myricom or Solarflare NIC with a Cisco 4900M would be different than a Dell Powerconnect switch and Intel NICs. – ewwhite Mar 20 '13 at 5:32
  1. You would want an algorithm where the window size is not drastically reduced when there is a packet drop. It's the drastic drop in window size that results in the sudden drop in throughput with TCP traffic.

  2. If your switch and your server support flow control, try enabling flow control. How well this works depends almost entirely on the Switch's silicon and firmware. Basically, the switch will detect egress congestion on the port that is connected to a client, determine where the packets came from, and send flow control frames out the ingress port (i.e. back to the server). If the server understands flow control frames, it will reduce the transmission speed. If it all works well you will get optimal throughput with virtually zero packet drops occurring on the switch's egress buffer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.