Situation: We have an Ubuntu server that hosts three VMs using KVM. All guests as well as the host need to access the same files in a certain subfolder of /var . Thus, the subfolder is exported via NFS. Our problem is that the guest can read from/write to the directory with only half the speed of the host. The export table looks like this

alice@host:~$ cat /etc/exports

where the host has IP and the VMs{1..3}. /home/videos is a symlink to that certain subfolder in /var. In particular, it's /var/videos/genvids.

This is the relevant line from the VM's fstab: /mnt/nfs nfs auto,noatime,rsize=4096,wsize=4096  0 0

The hard disk has a sustained data rate of ~155 MB/s which is verified by outputs of hdparm -tT as well as dd:

alice@host:~$ dd if=/home/videos/4987_1359358478.mp4 of=/dev/null bs=1024k count=300
300+0 records in
300+0 records out
314572800 bytes (315 MB) copied, 2.04579 s, 154 MB/s

From within a VM things look differently:

bob@guest:~$ dd if=/mnt/nfs/4959_3184629068.mp4 of=/dev/null bs=1024k count=300
300+0 records in
300+0 records out
314572800 bytes (315 MB) copied, 4.60858 s, 68.3 MB/

Fitting the blocksize to the file system's page size had no satisfying effect:

bob@guest:~$ dd if=/mnt/nfs/4925_1385624470.mp4 of=/dev/null bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 5.77247 s, 71.0 MB/s

I consulted various pages on NFS performance, most relevant the NFS FAQs Part B, and the respective Performance Tuning Howto. Most of the hints do not apply. The others did not improve the results. There are threads here that deal with disk performance and KVM. However they do not cover the NFS aspect. This thread does, but network speed seems not the limiting factor in our case.

To give a complete picture this is the content of the exports etab with symlinks resolved and all active export options shown:

alice@host:~$ cat /var/lib/nfs/etab

What also bothers me in this context - and what I do not understand - is the nfsd's procfile output:

alice@host:~$ cat /proc/net/rpc/nfsd
th 8 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

For the third column and beyond I would have expected values other than zeros after reading from the disk within the VMs. However, nfsstat tells me that there were indeed read operations:

alice@host:~$ nfsstat
Server nfs v3:
null         getattr      ...
9     0%     15106     3% ...
read         write        ...
411971   95% 118       0% ...

So, the topic is quite complex and I'd like to know where else to look or whether there is an easy solution for this.

  • Do the guests and the host have similar NFS performance to devices NOT on the host? What kind of performance do you get over NFS to the KVM host from another physical machine? – TheFiddlerWins Sep 12 '13 at 14:58
  • how are the VMs set up, do you use virtio? On the NFS host, how is the exported partition mounted? – dyasny Sep 12 '13 at 21:15
  • @TheFiddlerWins This is an experimental lab installation. Currently, there are no external devices connected via NFS. – betastrahler Sep 13 '13 at 8:19
  • @dyasny The VM's network devices use virtio. The exported partition is mounted as plain xfs file system within the host: /dev/sdb1 /var xfs defaults 0 2 – betastrahler Sep 13 '13 at 8:22
  • 1) I meant the VM's disk controllers - are those virtio? 2) I have noticed some serious slowdowns when the exported partition doesn't have noatime and nodiratime. Haven't tried XFS yet, but it usually helps with ext – dyasny Sep 13 '13 at 14:10

As it turns out the problem was easier to resolve than expected. Tuning the rsize and wsize option in the VM's fstab did the trick. The respective line is now /mnt/nfs nfs auto,noatime,rsize=32768,wsize=32768  0 0

For me this was not obvious since I had expected best performance if values for rsize and wsize meet the disk's block size (4096) and are not greater than the NIC's MTU (9000). Apparently, this assumption was wrong.

It is notable that the exact sustained disk data rate depends on the very file: For two similar files of size 9 GB I observed rates between 155 MB/s (file 1) and 140 MB/s (file 2). So a reduced data rate with one file may still result in full data rate with another file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.