I am witnessing very counter-intuitive effect in my network setup.
I have a network setup with three devices (let's call them server, laptop and nettop, all running Linux, details below) connected to a gigabit ethernet switch.
I measure network throughput (a) by transferring a file from the server to the laptop using
scp and (b) using
iperf -s on the server and
iperf -c on the laptop.
When I perform the throughput test while all three devices are up and running, I get an ok-ish performance of roughly 40 MB/s (SSH) or 280 Mbits/s (iperf).
However, as soon as I turn off the nettop which isn't involved in the measurement process at all, I get a lousy performance of roughly 3.5 MB/s (SSH) or 58 Mbits/s (iperf).
If it was the other way round, I would try to find out if the nettop is interfering with the transfer somehow. But the fact that performance drops by an order of magnitude when I turn it off leaves me mystified.
This behavior is perfectly reproducible. As soon as the nettop gets pingable, the performance rises immediately. As soon as it is off, the performance drops.
I tried cutting power to the switch for a minute and shuffling around the ethernet ports on the switch with no effect.
So my question is: What could possibly cause the effect I am witnessing? Where could I look in order to narrow down the cause of the problem?
Details on the Setup
- Foxconn nT-330i, Intel Atom 330
- Built in Atheros AR8131 Gigabit Ethernet Controller
- Ubuntu 10.04 (lucid), Kernel 2.6.32-29, amd64
- Thinkpad R61, Intel Core 2 Duo T8300
- Built in Broadcom BCM5787M Gigabit Ethernet Controller
- Ubuntu 11.10 (oneiric), Kernel 3.0.0-16, amd64
- AMD Athlon 64 X2
- Intel 82574L Gigabit Ethernet Controller (e1000e)
- Debian 6.0.4 (squeeze), Kernel 2.6.32-5, amd64
- Level One GSW-0806 8-Port Gigabit Switch
The server has two additional network interfaces, one hooked up to a cable modem, the other one to a wifi router.
The server runs a DHCP server which the nettop and laptop acquire their addresses from. It also runs numerous other services, including KVM running two VMs.
All devices are in the
10.2.3.0/24 network. Server has
10.2.3.42. The Server interface connected to the Wifi has
10.2.4.1/24. Neither the nettop's nor the laptop's wifi adaptor is enabled.
ethtool shows the following adaptor settings. These outputs are unaffected by turning off and on the nettop.
On the server:
# ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: on Supports Wake-on: pumbag Wake-on: g Current message level: 0x00000001 (1) Link detected: yes
On the laptop:
# ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: g Wake-on: g Current message level: 0x000000ff (255) drv probe link timer ifdown ifup rx_err tx_err Link detected: yes