I'm standing before choice of NICs and switch for storage server and virtualization hosts and I can't decide between 10G and quad gigabit with bonding-rr. 10G is noticeably more expensive solution and we don't really need 10G, 4G is definitely enough...

However I'm a bit worried about how NFS scales with bonding-rr. Did anyone use NFS with such setup? Is it viable alternative to single higher bandwidth link? I'm aware of some good practices for this kind of bonding (4 separate vlans for each set of links to reduce collisions) and I took a look at some stories describing such setups in production but they typically compare it to theoretical bandwidth or single gigabit performance - not 10G.

Does anyone know some other topics and issues I should take into consideration with mentioned options?


You definitely should not consider balance-rr as a bonding mode for anything other than direct point-to-point links for a single protocol to go over. The primary issue with balance-rr is that you'll get excessive jitter and out-of-order delivery, which is pretty damning for services like NFS and CIFS. Even with a point-to-point topology and one protocol, bonding the maximum supported amount of 1G links in balance-rr won't be anywhere near as reliable as a single 10G link. balance-rr is a desperate last-resort kludge for more bandwidth in a limited system.

LACP will balance connections well, but is only effective for multiple endpoints. This isn't an issue for most file and block servers, since those are typically deployed for the purpose of providing storage to multiple endpoints. Most switches max out at 8 port LACP bonds, and balancing traffic over 8 ports is far from perfect.

10G Ethernet is fairly expensive to transition to. Have you looked at 10G or 40G infiniband? It's worlds cheaper, you can run all your hosts along a homogeneous redundant fabric, and you get built in RDMA.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.