Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am currently using an Intel PRO/1000 PT LP Quad Port Server NIC with Ubuntu 12.04 LTS Server 64 Bit. I installed Gnome so I have a GUI.

I downloaded bmon 2.0.1 to monitor the network, and right now according to /proc/net/bonding/bond0 the bond is up and working. I also believe the switch is configured correctly.

The network setup has two links from the Ubuntu host going to a switch - these links are teamed. Two separate links to the switch go to two hosts which copy a file from the Ubuntu teamed NIC host.

When I copy two files (one to each single-link host), I get near-wire-speed for only one of the bonded pair, and the other one does nothing! This is despite the fact that the switch is doing MAC hashing and the files are going to separate places. I am copying the files off of an SSD to two RAM disks, so I know those are not the bottleneck. I should easily be able to saturate a single 1 gig line.

I am using the latest (2.0.0) e1000e driver from Intel, and while I realize Ubuntu is not officially supported by Intel, I would like to get this working in Ubuntu.

Thanks!

share|improve this question
    
Are the files going to destinations inside the subnet, or out? What bonding mode are your NICs in? –  Shane Madden Jun 20 '12 at 16:58
    
NICs are in 802.3ad bonding mode and the traffic is within the same subnet –  user124712 Jun 20 '12 at 17:34

1 Answer 1

A hash algorithm is in use to deterministically distribute traffic across the two links in your channel. This algorithm might be using src/dst MAC address or L2/L3/L4 information in various permutations. For purposes of this discussion let's assume that it's MAC based and that to achieve a 50/50 split we can simply take the final digit in the destination's MAC address and if it's an even number (0,2,4,6,8,A,C,E) it goes on link 1 and if it's odd (1,3,5,7,9,B,D,F) it goes on link 2. If the destination hosts in question have MAC addresses ending in 00 and 02 then they will both (correctly) be placed on the first link.

The actual algorithms in use aren't exactly like the pidgin version above, but the effect is the same - over a sufficiently large population of addresses the distribution will approach 50/50. On much smaller samples, however, there's also a 50% chance that all of your traffic will be driven into a single link.

The best suggestion I'd have is to try different hashing algorithms. In practical terms your odds get better if more information is figured into the hash - so L3 addresses, L4 port numbers, etc all improve the probability that your traffic will be more evenly distributed.

As an interim troubleshooting step you could also try changing the MAC address of one of the receiving hosts to observe if you can influence the hash appropriately.

share|improve this answer
    
It was in fact the hashing algorithm. For the specific scenario we have using layer2+3 worked in the /etc/modules config file –  user124712 Jun 21 '12 at 23:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.