7

I'm trying to maximize the NFS/TCP bandwidth between a Linux client and a single NFS server over a 10Gbps link. Although each NFS mount has its own device (major/minor), everything comes down to a single TCP connection and this is a bottleneck.

Using iperf and multiple TCP connections, I can reach ~8Gbps, but NFS is stuck on a single TCP stream and limited to ~2Gbps (no tcp tuning yet).

On Solaris, there's the system setting rpcmod:clnt_max_conns which allow to set how many concurrent TCP connection the system is able to use to a single NFS server.

Is there a Linux equivalent or a way to do so ?

As a workaround, I know I could set many IP on the NFS server, and then load-balance the mountpoints on them but that wouldn't be optimal.

9
  • 1
    Have you looked into the throughput from your disks? 8gbps is A LOT to ask of an i/o stack. Sep 26, 2013 at 8:10
  • That's not issue, the NAS server is high performance storage appliance definitely able to sustain this.
    – Benoît
    Sep 26, 2013 at 10:43
  • What distro are you using on the server? How much RAM? number of cores? How many disks? type of disks? average iops per disk that you are seeing? operations in queue on the disks? time in queue per op on the disk? version of nfs? number of clients connecting to the server? There is a lot that goes into perf tuning and it is important to step back and look at the big picture first, and then narrow down. If you have done this, it helps to post numbers and stats you have already gathered. Even the clients may have something to do with it (eg, buffer size, mount options, dirty pages, etc).
    – MaQleod
    Sep 27, 2013 at 3:06
  • I am not looking to fine-grained tune the system. I can see that a TCP connection doesn't go above 250MB/s but using netcat with 8 concurrent TCP streams, I can reach 800MB/s. That's why I am looking for a way to run multiple TCP streams to the same NFS server. And because the NFS servers are proprietary, it's going to be very hard to tune them, so I'd rather use a simple trick like multiple connections.
    – Benoît
    Sep 27, 2013 at 16:09
  • Multiple connections is going to be the only way to do what you're looking for. A single stream should max out around 350MB/s per thread on a fine-tuned nfs server. In any case, if we don't know what distro or nfs version you're using, it is hard to give a good recommendation.
    – MaQleod
    Sep 27, 2013 at 19:26

3 Answers 3

1

You could switch back to UDP and use jumbo packets (4k datagrams for instance). Having your whole network support jumbo frames might help a lot, so that you don't have IP packets reassembly on each side.

1
  • Good suggestion. Ideally, jumbo frames should be at about 9K. This will maximize its network throughput.
    – shodanshok
    Apr 14, 2016 at 16:26
1

Starting from linux kernel 5.3 a new mount option "nconnect=X" is available, where X specifies the number of TCP connections to the server to use.

0

If you're using NFS v4.1 or greater you may be able to avail of pNFS (parallel NFS). As per the following release info from Red Hat, it's available in RHEL 6.4, but I haven't seen any updates indicating it's availability in other Linux distros.

https://www.redhat.com/about/news/press-archive/2013/2/red-hat-announces-general-availability-of-next-minor-release-of-red-hat-enterprise-linux-6

pNFS is defined in RFC5661 from 2010. You may want to investigate whether or not your storage appliance fully supports pNFS.

1
  • Unfortunately, I have to stick with NFSv3.
    – Benoît
    Sep 27, 2013 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.