25

Can NFS be reasonably used on production servers as a means of connecting a compute server to a storage server, assuming the connection is over a LAN 1Gbe or 10Gbe connection?

There's obviously some network overhead and NFS seems particularly slower with writes if you have sync mode enabled. Otherwise it seems reasonably lightweight and able to scale from what I can tell, but I have little experience with it personally. Am I wrong?

The problem is I have a server right now that acts as both the storage and web server but I'm going to end up needing to split the two likely in the future, and considering some requests need to pass through the web application layer for authentication before initializing the file transfer, it gets kind of tricky with this software. A network fs mount is the simplest option I just.. don't know if that's a good one.

I also plan to try and utilize local caching with NFS which should improve performance a good bit, but I'm not sure if that's enough.

As far as alternatives, there's only iSCSI that I'm aware of as a real competitor, and most people seem to recommend NFS over any of the other lesser known ones.

  • 8
    If you'll be maintaining the NFS storage yourself (and not some high availability NFS-as-a-service) and uptime is important, you may want to look into how to handle downtime for the NFS storage. Doing simple failover between two NFS servers can be tricky. Although more complicated, we moved to GlusterFS to better handle storage failures. – sbrattla Aug 16 at 18:55
  • 5
    Do you have any reason to believe that NFS wouldn't be reaonable, based on facts or expertise? – womble Aug 17 at 0:44
  • 4
    So long as you keep in mind that NFS is a best-effort storage stack - it's fine. Don't rely on it for anything a) high-volume, b) mission-critical. Can it handle high-volume and/or mission-critical services? Yep. And it likely will. Right up until it doesn't. – warren Aug 17 at 19:05
  • Just as a short follow up, I've set up our new server infrastructure using an NFS mount as described above. Had some issues getting jumbo frames enabled, but things still seem to be performing quite well. I have a 256GB fscache partition on our NVMe drive to help performance and all seems to be working as expected. So, pushing out around 2TB of data transfer per day right now over a 1Gbps line without any noticeable drop in performance. I'm certain fscache is definitely helping with that, but otherwise I'm relieved everything is working well and appreciate everyone here's help! – Makoto Sep 30 at 11:19
29

NFS is fine, barring some specific other criteria are met, namely:

  • The systems involved are both able to use NFS natively. Windows doesn't count here, it kind of works, but it's got a lot of quirks and is often a pain to work with when dealing with NFS in a cross-platform environment (and if it's just Windows, use SMB3, it eliminates most of the other issues with NFS). Note that on the client side, this means kernel-level support, because a user-level implementation either has to deal with the efficiency issues inherent in using something like FUSE, or it has to be linked directly into the application that needs to access the share.
  • You've properly verified how the NFS client handles an NFS server restart. This includes both the OS itself (which should be fine in most cases), and the software that will be accessing the share. In particular, special care is needed on some client platforms when the software using the share holds files open for extended periods of time, as not all NFS client implementations gracefully handle server restarts by explicitly remounting and revalidating locks and file handles like they should (which leads to all kinds of issues for the client software). Note that you should recheck this any time any part of the stack is either upgraded or reconfigured.
  • You're willing to set up proper user/group ID mapping. This is big, because without it you either need to mirror the UID/GID mappings between the systems (doable, but I'd be wary of setting up SSO against an internal network for an internet facing system) or you end up with potentially serious security implications (namely, what you see on one system for permissions does not match what you see on others).
  • You're operating over a secured network link, or are willing to properly set up authentication for the share. Without auth, anybody on the link can access it (and a malicious client can easily side-step the basic UNIX discretionary access controls).

Assuming you meet all those criteria, and you have a reasonably fast network, you should be fine. Also, if you can run jumbo frames, do so, they help a lot for any network filesystem or networked block storage.

| improve this answer | |
  • 2
    I'd like to add that the Windows NFS client has completely broken a couple of my NFS shares over the last year. – Persistence Aug 18 at 8:29
  • NFS on Windows is licensed and very poorly maintained code. There's a reason why MSFT doesn't support NFS in production for majority of their "selling" scenarios like SQL Server, Exchange, Hyper-V and so on. – RiGiD5 Aug 28 at 18:32
17

NFS is absolutely OK and is preferred over iSCSI due to the fact NFS is much easier to manage, share and backup.

| improve this answer | |
  • 3
    That's too strong a statement to be unconditionally true. Some use cases NFS is completely unsuited for but iSCSI does fine -- if you need true POSIX semantics for an application you're running, for example. NFS is sorta, kinda POSIX-y; whereas if you can run your own filesystem, you can have exactly the semantics you need. (Of course, needing to use multi-master-safe block-layer filesystems is a serious constraint, but those that exist have been built to work with heavy-duty applications like Oracle RAC; see f/e GFS2) – Charles Duffy Aug 17 at 21:07
  • 1
    OP asked an exact question and got his answer. This isn’t a discussion forum and OP didn’t ask “NFS Vs Block” in general. – BaronSamedi1958 Aug 18 at 6:08
  • "A production web server" is nothing remotely like an exact specification. We don't know what datastores that web server is using and what their requirements on underlying storage semantics are. We don't know whether there's an intent to support active/active database backends, etc. – Charles Duffy Aug 18 at 15:37
  • 1
    I’m not here to argue. There’s always reddit. – BaronSamedi1958 Aug 18 at 15:54
  • 2
    Thank you for the feedback, I should have provided more information in my original post. When I say linking to a storage server, I mean for storing static files (user uploaded content) - media files, images, user avatars, things of that nature. Definitely not running MySQL or any other database server over NFS, that will be handled on the compute server on its native NVMe storage device. Additionally small static assets like images will be cached over CloudFlare's CDN as needed, so really only larger files (like videos) will be actively served directly from the server itself. – Makoto Aug 18 at 23:45
12

We've been using NFS for years to attach our SAN to our VMware ESXi servers, running hundreds of VMs on it. No trouble at all.

The bottleneck is rather the storage system than the network protocol.

The network connection should be fast enough of course, meaning 10Gb Ethernet or fibre. We don't even bother with a separate storage network anymore.

| improve this answer | |
  • Disk I/O is usually what ends up killing performance, especially when people try to do performance heavy tasks against systems not really designed well for performance. – MaQleod Aug 19 at 0:12
  • 1
    @MaQleod and latency. There's always someone who thinks cross-country NFS will work fine. – Wayne Conrad Aug 19 at 4:39
6

iSCSI might be a bit faster...

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/storage_protocol_comparison-white-paper.pdf

https://www.hyper-v.io/whos-got-bigger-balls-testing-nfs-vs-iscsi-performance-part-3-test-results/

...but NFS like any other network redirector (SMB3, AFS/AFP etc) allows concurrent multi-access which is tricky with iSCSI or other block protocols.

https://forums.starwindsoftware.com/viewtopic.php?f=5&t=1392

| improve this answer | |
  • 3
    NFS has the advantage that you can share it with more than one Frontend server at a time (without strange cluster filesystems) – eckes Aug 18 at 0:55
  • 1
    This is what my forums link is all about... – NISMO1968 Aug 20 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.