Aggregated throughput of multiple connections (Linux)

Question

Why do multiple connections from the same host achieve better throughput than multiple connections from different hosts?

I suspect is something kernel-related (Linux) but a confirmation would help a lot.

More details I have 1 receiver process, let's call it R. It accepts incoming connections and receives data using select().

I have 3 sender processes S1, S2, S3. They connect to R and send data at a fixed rate, i.e. 200Mbit per second each.

If S1, S2 and S3 are on the same machine, I get better results than having each one of them on a different machine. (R is in both cases on some other machine)

Example: R on host0, S1, S2, S3 on host2, R receives at 600Mbit/s

R on host0, S1 on host1, S2 on host2, S3 on host3, R receives at 480Mbit/s

This looks counter-intuitive for me, I expected the opposite since in the second case senders don't have to share the network card and the processor (Not that expect processor or network card to be bottlenecks...)

[The hosts above are nodes in a linux cluster with a dedicated full-duplex Gigabit switch. They are running 2.6.24-24-generic (latest Ubuntu i guess)]

Answer 1

I suspect this is related to how the switch manages contention; with 1 sender there is no contention, with 3 senders all pushing as fast as they can the switch might start dropping packets causing TCP backoff and retransmission.

I suppose you could test this by putting 3 cards in the reciever and connecting them directly to the senders (don't think you need crossover cables for gigabit)

Answer 2

I believe this is a classic pipeline efficiency issue.

In the case of multiple sender hosts, they will all start sending ethernet frames through the wire. These frames will reach the switch back-plane, and get queued in the port attached to host0. Host0 will receive those as fast as it can, but the senders will quickly flood the switch with many more frames per second than the host0 switch port can deliver. They are trying to push 3 gigabit/s through a port that can only read 1 gigabit/s.

Usually, the switch would be able to buffer these frames if they pile up in the port0 queue. But they can't do miracles. So the switch will start dropping frames in the floor. The consequence is lots of TCP packets will get lost. You can verify this by checking an increase of this counter in the sender hosts:

netstat -s|grep 'segments retrans'

The retransmissions will just make the congestion worse, as you can imagine - even with a backoff algorithm

If there's just one host, it will transmit at 1 Gigabit/s. And the receiver will probably get all ethernet frames and it should work fine. I'd also suggest doing the one-to-one test using UDP, and you'll get even better results. One nice tool for this task is 'iperf'.

Good luck!

Answer 3

For the sake of discussion, I will assume TCP/IP is used for the communications. If this is wrong please comment and let me know.

You can probably take a packet capture, and get the full details, but this is likely moving the point of contention from a Single coordinated TCP/IP stack on a single machine, to multiple seperate TCP/IP stacks on seperate machines, and the Gigabit switch could be an el-cheapo that really can't cut it.

In you're original scenario, you have one server sending to another server, using the same IP stack. Most servers NICs these day's have TCP offload, and alot of stuff they have in hardware, so they can coordinate the IP stack pretty well, with what frames get sent when, and do proper interleaving. This probably makes the standard NIC available better than most switches you can buy. However, by moving this to seperate servers, you've move the point of contention to the switch, which is really only aware up to layer 2 (send a frame to x mac address), when it needs to interleave several high throughput connections to a single port, it could simply have a really bad chip and memory that causes inconsistent latencies (known as jitter), overruns, or even dropped frames. This is because before, all the switch had to do was copy frames from port 1 to port 2 and vice versa, but now it's got alot of work to do since it's got lots of frames coming in from 3 sources that it must try to interleave withing microseconds.

However, I should mention some of the more basic items that could be of issue here as well:

If you have a slightly bad cable, or got a small pinch and are getting some added impedance on the wire, this could be affecting throughput, always check for lost packets / frames, and definitely check you're interfaces for errors. I don't know how well it caries over to linux, but on FreeBSD netstat -I will show the number of input / output errors.

Also, make sure you don't have any auto-negotiate errors, that one of the interfaces negotiated to half-duplex. This isn't very common at Gigabit speeds, but can be very common when mixing 100Mb with Gigabit and Solaris is particularly bad at negotiating.