1

I have experienced some issues with less-than-expected throughput with NFS. After examining a number of options (endless googling), including raid setup, network interfaces etc, I wanted to test a basic setup, exporting a ramdisk mounted on /media/ramdisk and accessing it through localhost.

The tests with the ramdisk shows the same performance issue pattern as with my SSD RAID0 NFS export, with a massive performance hit while accessing via NFS, and both the SSD raid and the ramdisk maxes out at approx. 13K IOPS read, and 4K IOPS write.

Server:

Ubuntu 18.04, fully updated. VM with 5 GB ram, and 8 processors. (Host cpu: Intel(R) Core(TM) i7-4800)

Content of /etc/exports:

/media/ramdisk       *(rw,async,no_subtree_check,fsid=1)

Mounted via:

sudo mount 127.0.0.1:/media/ramdisk /mnt/ramdisk -o async

Output of mount command:

127.0.0.1:/media/ramdisk on /mnt/ramdisk type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

Using fio to test the speed of the ramdisk without NFS with the following command:

sudo fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test1 --filename=/media/ramdisk/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

I get:

read: IOPS=283k, BW=1107MiB/s (1160MB/s)(3070MiB/2774msec)
write: IOPS=94.7k, BW=370MiB/s (388MB/s)(1026MiB/2774msec)
cpu          : usr=21.89%, sys=78.07%, ctx=5, majf=0, minf=7

Very fast, as expected

Testing via NFS with the following command:

sudo fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test2 --filename=/mnt/ramdisk/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

I get:

read: IOPS=12.7k, BW=49.6MiB/s (51.0MB/s)(3070MiB/61951msec)
write: IOPS=4239, BW=16.6MiB/s (17.4MB/s)(1026MiB/61951msec)
cpu          : usr=5.24%, sys=21.16%, ctx=786004, majf=0, minf=5

I am monitoring the system with atop during the test, and neither CPU nor network are particularly stressed during the test.

Using dd directly on the ramdisk with:

sudo dd if=/media/ramdisk/test of=/dev/null bs=1M iflag=nocache status=progress

I get:

4294967296 bytes (4.3 GB, 4.0 GiB) copied, 0.691044 s, 6.2 GB/s

Doing the same via NFS (/mnt/ramdisk/test), I get:

4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.58302 s, 1.7 GB/s

I have increased RPCNFSDCOUNT to 64

Running iperf on localhost, I get

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  64.6 GBytes  55.5 Gbits/sec

So apparently network throughput is good.

So what can be holding back the NFS performance?

3
sudo fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test1 --filename=/media/ramdisk/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Oh no :-(, you're using --iodepth=64 with the libaio ioengine and direct=0 so this command likely won't behave as you're expecting! See the warning in the fio help/manual about failing to use direct=1 when using libaio ioengine:

libaio

Linux native asynchronous I/O. Note that Linux may only support queued behavior with non-buffered I/O (set direct=1 or buffered=0). This engine defines engine specific options. [emphasis added]

so libaio is likely acting as if iodepth=1... (see https://github.com/axboe/fio/issues/512#issuecomment-356604533 for a more detailed explanation)

So what can be holding back the NFS performance?

I'm afraid the fio command job you created is holding you back - maybe you need to use different options and/or a different I/O engine?

PS: You are very unlikely to be pushing enough I/O such that --gtod_reduce=1 will make any difference to you...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.