3

I'm standing before choice of NICs and switch for storage server and virtualization hosts and I can't decide between 10G and quad gigabit with bonding-rr. 10G is noticeably more expensive solution and we don't really need 10G, 4G is definitely enough...

However I'm a bit worried about how NFS scales with bonding-rr. Did anyone use NFS with such setup? Is it viable alternative to single higher bandwidth link? I'm aware of some good practices for this kind of bonding (4 separate vlans for each set of links to reduce collisions) and I took a look at some stories describing such setups in production but they typically compare it to theoretical bandwidth or single gigabit performance - not 10G.

Does anyone know some other topics and issues I should take into consideration with mentioned options?

Improve this question

1 Answer 1

Reset to default
3

You definitely should not consider balance-rr as a bonding mode for anything other than direct point-to-point links for a single protocol to go over. The primary issue with balance-rr is that you'll get excessive jitter and out-of-order delivery, which is pretty damning for services like NFS and CIFS. Even with a point-to-point topology and one protocol, bonding the maximum supported amount of 1G links in balance-rr won't be anywhere near as reliable as a single 10G link. balance-rr is a desperate last-resort kludge for more bandwidth in a limited system.

LACP will balance connections well, but is only effective for multiple endpoints. This isn't an issue for most file and block servers, since those are typically deployed for the purpose of providing storage to multiple endpoints. Most switches max out at 8 port LACP bonds, and balancing traffic over 8 ports is far from perfect.

10G Ethernet is fairly expensive to transition to. Have you looked at 10G or 40G infiniband? It's worlds cheaper, you can run all your hosts along a homogeneous redundant fabric, and you get built in RDMA.

Improve this answer
2
  • 1
    According to kernel.org/doc/Documentation/networking/bonding.txt there are some remedies to the out of packet reordering? Is there a "best" bonding mode for NFS from single client?
    – rogerdpack
    Jun 11, 2021 at 18:36
  • 1
    The only way you'll get increased performance with a single client via bonding is to use balance-rr. This increased performance comes with caveats, which that manual mentions, along with a very specific mitigation strategy of using network devices on both ends which have interrupt coalescing disabled (so they interrupt the CPU every time they receive a packet). This has a pretty large impact on latency and CPU usage, and even for 1Gib connections can bring an immodest system to its knees if one were to bond multiple links in this way. It's just not worth it. Use a bigger pipe.
    – Spooler
    Jun 14, 2021 at 16:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .