I'm adding new centralized storage on a linux server on a private GBE LAN. There are about a dozen client machines which in the past have done NFS v3 mounts over TCP. At times the LAN gets congested and dropped packets result in long timeouts on NFS. Is there something else I should be looking at? Would I be happier with NFS v4, SMB or iSCSI, for example?

I'm not looking at AFS or its derivatives as I don't want to locally cache files. For my applications I need to run the server in sync mode to ensure that everything written gets saved to disk. The LAN is behind a firewall and all of the clients are under my control so I am not worried about security aspects. The only thing not on the table is fibre channel as I can't afford adding a SAN. I currently have timeo=30 and retrans=default (3). The system works ok most of the time, it's just that throughput decreases due to the errors when the LAN gets busy.

  • What type of networking hardware? I would steer clear of iSCSI for this purpose. – SpacemanSpiff Feb 28 '13 at 2:47
  • 48-port Netgear switch. Why not iSCSI? – stark Feb 28 '13 at 2:59
  • A dedicated switch of the same class, just for doing storage access might not be a bad investment. I would avoid iSCSI because of the complexities of encapsulating your data in LUNs – SpacemanSpiff Feb 28 '13 at 3:11

If it's Windows end to end, SMB3 is really really good at this. Microsoft has done a good job. So good, they're even allowing HyperV machines to run on mounted drives as of Server 2012.

However, I don't think you are Windows end-to-end.

iSCSI is a bad choice here since storage stacks in operating systems don't take well to random delays in packet arrival. This is one of the biggest reasons why they say go with dedicated networks for your iSCSI network. It sounds like you have one, but if you're getting a lot of congestion I'd steer clear of iSCSI even with dedicated.

NFS4 is much better than NFS3 for this, since they've actually configured the protocol to tune TCP settings. Try it out, you'll likely find those packet drop/delay problems are much reduced.

SMB is not something I'd recommend for non-Windows systems at this point, I'd go with NFSv4. Support is there, but personally I'm not convinced the protocol stacks are robust enough for heavy hitting. The one difference is if your filer is a Windows server; in that case it's a toss-up if I'd recommend NFSv4 over SMB.

  • Thanks. When I get a chance to try NFS4 I'll post back here. – stark Feb 28 '13 at 3:34
  • NFS4 seems like a win – stark May 9 '13 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.