Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am witnessing very counter-intuitive effect in my network setup.

I have a network setup with three devices (let's call them server, laptop and nettop, all running Linux, details below) connected to a gigabit ethernet switch.

I measure network throughput (a) by transferring a file from the server to the laptop using scp and (b) using iperf -s on the server and iperf -c on the laptop.

When I perform the throughput test while all three devices are up and running, I get an ok-ish performance of roughly 40 MB/s (SSH) or 280 Mbits/s (iperf).

However, as soon as I turn off the nettop which isn't involved in the measurement process at all, I get a lousy performance of roughly 3.5 MB/s (SSH) or 58 Mbits/s (iperf).

If it was the other way round, I would try to find out if the nettop is interfering with the transfer somehow. But the fact that performance drops by an order of magnitude when I turn it off leaves me mystified.

This behavior is perfectly reproducible. As soon as the nettop gets pingable, the performance rises immediately. As soon as it is off, the performance drops.

I tried cutting power to the switch for a minute and shuffling around the ethernet ports on the switch with no effect.

So my question is: What could possibly cause the effect I am witnessing? Where could I look in order to narrow down the cause of the problem?

Details on the Setup

Nettop

  • Foxconn nT-330i, Intel Atom 330
  • Built in Atheros AR8131 Gigabit Ethernet Controller
  • Ubuntu 10.04 (lucid), Kernel 2.6.32-29, amd64

Laptop

  • Thinkpad R61, Intel Core 2 Duo T8300
  • Built in Broadcom BCM5787M Gigabit Ethernet Controller
  • Ubuntu 11.10 (oneiric), Kernel 3.0.0-16, amd64

Server

  • AMD Athlon 64 X2
  • Intel 82574L Gigabit Ethernet Controller (e1000e)
  • Debian 6.0.4 (squeeze), Kernel 2.6.32-5, amd64

Switch

  • Level One GSW-0806 8-Port Gigabit Switch

The server has two additional network interfaces, one hooked up to a cable modem, the other one to a wifi router.

The server runs a DHCP server which the nettop and laptop acquire their addresses from. It also runs numerous other services, including KVM running two VMs.

IP-Addressing

All devices are in the 10.2.3.0/24 network. Server has 10.2.3.1, laptop 10.2.3.37, nettop 10.2.3.42. The Server interface connected to the Wifi has 10.2.4.1/24. Neither the nettop's nor the laptop's wifi adaptor is enabled.

ethtool output

ethtool shows the following adaptor settings. These outputs are unaffected by turning off and on the nettop.

On the server:

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: on
    Supports Wake-on: pumbag
    Wake-on: g
    Current message level: 0x00000001 (1)
    Link detected: yes

On the laptop:

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Half 1000baseT/Full 
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Half 1000baseT/Full 
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: g
    Current message level: 0x000000ff (255)
                   drv probe link timer ifdown ifup rx_err tx_err
    Link detected: yes
share|improve this question
    
What does your IP addressing look like? I think its no coincidence that you seem to drop to 802.11G speeds. –  SpacemanSpiff Apr 17 '12 at 21:19
    
Added the IP-Addressing to the question. Do you need any more details? –  hc_ Apr 17 '12 at 21:23
    
Could you take the dhcp server out of the equation and assign static IP addresses to nettop and laptop? –  aseq Apr 17 '12 at 21:28
6  
All I can think of is a bad switch. You seem to be using an off-brand model, and they often do weird things. I would try a known good switch, or experiment with different cables in different ports. You might also do a wireshark or tcpdump on the nettop looking for anything interesting. –  mfarver Apr 17 '12 at 21:29
2  
Remove the switch and connect the laptop directly to the server. –  Craig Apr 17 '12 at 22:13
show 12 more comments

2 Answers

However, as soon as I turn off the nettop which isn't involved in the measurement process at all, I get a lousy performance of roughly 3.5 MB/s (SSH) or 58 Mbits/s (iperf).

This strongly indicates that the switch (which is a no-brand so-so specimen) drops the entire backplane to 100mbit when you disconnect the nettop.

I agree it is worse than weird - it's unacceptable.

But there you have it.

share|improve this answer
add comment
up vote 0 down vote accepted

It's exactly as mfarver suggested: My switch is broken. I replaced it with a new one (a Cisco SG-200-08 this time) and now everything works like a charm, regardless of which additional devices are connected and which aren't.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.