6
votes

I have a home network with a 1 GBit/sec switch as the central point, and all computers with 1 GBit/sec NIC's. However, I get at most 200 MBit/sec copy speeds over NFS shares. It seems I should be able to get much more. What can I realistically expect and how do I tune my systems and NFS server to get higher speeds ?

All systems involved are Linux, by the way (mix of CentOS and Fedora).

2
  • 1
    How fast are your source disks ? Does hdparm -t show they are capable of doing more than 20mb/sec sustained ? May 3, 2009 at 6:21
  • 1
    It would be helpful to see a relevant line from /etc/fstab so we can see the options you're using to mount the NFS share. May 4, 2009 at 17:21

7 Answers 7

8
votes

You're getting beaten by the overhead. You want to look into jumbo frames and also expand your nfs read/write data size. There's lots of guides to NFS performance tuning on the 'net, like this one that doesn't look too out of date.

Also realize that you're never going to be able to do better than disk speed, so make sure that's not a bottleneck.

1
  • Most modern NFS stacks automatically optimize the read/write data size and will likely do a better job than you will. 'cat /proc/mounts' for the current values. Sep 12, 2012 at 9:56
5
votes

You should defiantly look into jumbo frames. Even cheap 5 port GbE hubs can do it these days, however you have to enable jumbo frames on both ends of the connection. To enable jumbo frames, do this

ifconfig eth0 mtu 9000

Here are the mount options that I use when automounting home directories from our filer

rw,intr,soft,nfsvers=3.tcp,nolock,noatime,rsize=32768,wsize=32768

Please update the question and let us know if this improves the situation

1
  • Note: not just the endpoints of the connection, but all devices along the path (and preferably in the entire LAN segment) must have jumbo-frames enabled, switches, servers, clients, etc., for them to work. Also, jumbo-frames are no magical solution, but they do help. Sep 12, 2012 at 9:50
3
votes

Jumbo frames can certainly help, but don't expect it to magically give you 100% increase in throughput. Check out my post here in another question that discusses a way to test network throughput without bringing disk i/o into the picture. This will give you a baseline to see if there are perhaps other factors not related to the network.

If you do plan on enabling Jumbo Frames, as some here have suggested, please be aware of some limitations and requirements.

  • All devices along the path must support and have jumbo frames enabled. This includes:
    • Both the sending and receiving nic
    • Both switch ports the devices are connected to
    • All L2 switch ports along the path from sender to receiver (switch-to-switch connections)
    • All L3 interfaces along the path, if applicable
  • Make absolutely sure that all devices in the same vlan support and are configured for jumbo frames.

Reason for the second bullet: Consider two PCs setup in the same vlan, one configured for 9000 byte frames, and the other for 1500 byte frames. The PC1, configured for standard frames, will be able to send packets successfully to PC2. However, the reply that PC2 send will all be 9000 byte frames. Since PC1 is configured for standard frames, it will be unable to process the jumbo frames, and they will be dropped. To fix this issue, you'll need to insert an L3 device (router) between the two PCs so they are able to communicate.

1
vote

Bottleneck can be your network or your hard drives. First of all check which are, then optimize.

1
vote

Have you got the ability to switch on NIC Interrupt Coalescence, Receive-Side Scaling or any form of TCP-offload (checksum or whole-packed)? They will make a huge difference if you can enable them, which depends on your NICs and IP stack.

1
vote

For NFS specific information, be aware that newer implementation of NFS support not only UDP, but also TCP. Default is still usually UDP. Oddly enough, on very fast networks, TCP can be faster due to windowing. Try it both ways and compare.

For other tips, just search the site. The general form of this topic has already been beat to death eleventy-billion times. :-)

0
votes

Don't forget to ensure that you have jumbo frames enabled, and make sure that any network switches you use also support jumbo frames and have then enabled. You may want to run RAID striping for speed on the NFS share.

Not the answer you're looking for? Browse other questions tagged .