Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two bonded Gigabyte NICs on a CentOS 6 server, the ethtool shows the speed is 1000Mb/s, but when I do the speed test from a PC to this server on the same subnet, the transfer speed is less than 110Mb/s. If I do the test from one PC to another PC, the speed is up to 900Mb/s. Why does the bonding nic speed is so slow? Should I at least get 1000Mb/s? Thanks.

Settings for eth1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes

Settings for eth2: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes

===Message log===

Apr 3 15:43:16 HOSTNAME kernel: bond0: link status definitely up for interface eth1, 1000 Mbps full duplex.

Apr 3 15:43:16 HOSTNAME kernel: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

Apr 3 15:43:16 HOSTNAME kernel: bond0: link status definitely up for interface eth2, 1000 Mbps full duplex.

share|improve this question
1  
How exactly are these devices interconnected? –  David Schwartz Apr 3 '13 at 23:47
    
All the devices are connected to the same switch. –  garconcn Apr 3 '13 at 23:48
1  
your speed is limited by the slowest device in your link between your machine and the server. The word server means (to me at least) this device lives in a segregated section of the network, protected by one or more firewalls and perhaps some routers. In this mix, it is not surprising to find a network interface, speed of which is less than gigabit. –  MelBurslan Apr 4 '13 at 0:05
1  
The PC and server are all connected to the same switch, there's no route in this network. The PC and server link speeds are all 1Gb in the switch. –  garconcn Apr 4 '13 at 0:10
1  
How are you testing? What command are you using to test? Try testing with iperf, and include the results in your question. –  Zoredache Apr 4 '13 at 0:29

2 Answers 2

This probably has your answer: Bonding Nics with Linux to get better performance

Essentially your transmitting data balanced on 2 NICs, hits the switch and the switch forwards both streams of packets only down 1 of the receiving interfaces. That receiving interface is only 1Gbps so that is all you get.

share|improve this answer
    
Thanks for the answer, but my speed is only 110Mb/s, not even close to one NIC speed 1000Mb/s. –  garconcn Apr 4 '13 at 0:14
    
Are you mixing up your megabits and megabytes? Gigabit is 1000 Megabits/s which equates to approximately 120 megabytes/s. Getting 110 megabytes/s is a pretty good amount including overheads and would make sense. –  Nick Apr 4 '13 at 0:21
    
I didn't mix up with the Mb and MB. My PC Ethernet is 1Gb, the transfer speed is only 110Mb/s or 10% utilization according to the Windows Resource monitor. –  garconcn Apr 4 '13 at 0:29

Just so you know, I typically use netperf with a switch that outputs the results in MB instead of Mb.

There should be no difference between a speed test on an interface that is bonded and one that isn't. For me, if the range isn't between 95-120 MB/s then something is wrong. After throughput testing in dozens of different environments the average I see for testing 1GbE is 109 MB/s.

As to why speed tests can be slow, when I run into systems using LACP with an incorrectly configured aggregation group I typically get 15-20 MB/s.

Another reason a speed test can be much slower than expected is the a socket buffer size that is too small. In my experience, 64k seems to be the sweet spot for 1GbE and 128k works best for 10GbE.

I'm not sure what the iperf equivalent is but using netperf the syntax for a test on 1GbE would be:

netperf -H <hostIP> -f M -l 30 -- -s 65536

-H is for the target IP/hostname -f M is to display output in MB -l 30 is a length of 30 seconds (Change timeout for longer/shorter tests) -- means options will follow -s is for the socket size.

I specified the options because netperf tends to have poorly documented man pages.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.