While doing some benchmarking on a network app I discovered strange behavior that involves 100MBit and 1GBit ethernet clients connected through a switch (at-gs900/8e). I have one server (Gbit) and multiple clients, the network app uses TCP/IPv4. The clients receive big data streams over a single connection each.
As long as only clients with GBit speed receive data from the server all is fine, the uplink of the server is saturated even by just one client; when multiple clients receive data, the bandwidth is split up evenly among them. But as soon as one 100MBit client also receives data from the server, the output drops to pretty much exactly 100MBit/s per client, even for the GBit ones. So with only the 100MBit client receiving data I get 100MBit/s output, with an additional GBit client I get 200MBit/s and so on... My first idea was like "maybe the server's NIC switches back to 100MBit/s FD", but that clearly doesn't make any sense as I get 100MBit/s output per client, not in total.
As a side note, all clients were continuously connected to the server during the test (idle TCP connections). Can anyone enlighten me what is going on here, and if this is expected behavior? I really hope not, as it would make switches quite stupid devices. I hope to get some different models to test during the next days...
Update: I found out that this is caused by ethernet flow control, namely so called pause frames. I did not figure out who is creating these frames, the 100MBit client or the switch itself, but it is still weird to me that it won't throttle the server to 100MBit/s in total, instead of 100MBit/s per client... So the "solution" on the server is to disable flow control, which at least in my scenario is fine, as TCP deals with congestion on its own:
ethtool -A eth0 autoneg off rx off tx off will result in 100MBit/s for the 100Mbit/s client and 900MBit/s for a second, GBit-capable client. :-)