Few days ago I noticed something rather odd (at least for me). I ran rsync copying the same data and deleting it afterwards to NFS mount, called
/nfs_mount/TEST is hosted/exported from
nfs_server-eth1. The MTU on both network interfaces is 9000, the switch in between supports jumbo frames as well. If I do
rsync -av dir /nfs_mount/TEST/ I get network transfer speed X MBps. If I do
rsync -av dir nfs_server-eth1:/nfs_mount/TEST/ I get network transfer speed at least 2X MBps. My NFS mount options are
Bottom line: both transfers go over the same network subnet, same wires, same interfaces, read the same data, write to the same directory, etc. Only difference one is via NFSv3, the other one over rsync.
The client is Ubuntu 10.04, the server Ubuntu 9.10.
How come rsync is that much faster? How to make NFS match that speed?
Edit: please note I use rsync to write to NFS share or to SSH into the NFS server and write locally there. Both times I do
rsync -av, starting with clear destination directory. Tomorrow I will try with plain copy.
Edit2 (additional info): File size ranges from 1KB-15MB. The files are already compressed, I tried to compress them further with no success. I made
tar.gz file from that
dir. Here is the pattern:
rsync -av dir /nfs_mount/TEST/= slowest transfer;
rsync -av dir nfs_server-eth1:/nfs_mount/TEST/= fastest rsync with jumbo frames enabled; without jumbo frames is a bit slower, but still significantly faster than the one directly to NFS;
rsync -av dir.tar.gz nfs_server-eth1:/nfs_mount/TEST/= about the same as its non-tar.gz equivalent;
cp -r dir /nfs_mount/TEST/= slightly faster than
rsync -av dir /nfs_mount/TEST/but still significantly slower than
rsync -av dir nfs_server-eth1:/nfs_mount/TEST/.
scp -r dir /nfs_mount/TEST/= fastest overall, slightly overcomes
rsync -av dir nfs_server-eth1:/nfs_mount/TEST/;
scp -r dir.tar.gz /nfs_mount/TEST/= about the same as its non-tar.gz equivalent;
Conclusion, based on this results:
For this test there is not significant difference if using tar.gz large file or many small ones. Jumbo frames on or off also makes almost no difference.
scp are faster than their respective
rsync -av equivalents. Writing directly to exported NFS share is significantly slower (at least 2 times) than writing to the same directory over SSH, regardless of the method used.
rsync are not relevant in this case. I decided to try
scp just to see if they show the same pattern and they do - 2X difference.
As I use
cp in both cases, I can't understand what prevents NFS to reach the transfer speed of the same commands over SSH.
How come writing to NFS share is 2X slower than writing to the same place over SSH?
Edit3 (NFS server /etc/exports options):
rw,no_root_squash,no_subtree_check,sync. The client's /proc/mounts shows:
Thank you all!