Secure Network Filesystems for Linux: What are people doing?

Question

NFSv3 is widespread, but the default security model is... quaint. CIFS can use Kerberos authentication, but without POSIX semantics it's a non-starter. AFS never did encrypt traffic on the wire and is krb4 — and basically a dead project. Fancy new experimental filesystems either never materialize or are focused on speed (and if you're lucky, data reliability) — for example, Lustre uses the same client-trust model as NFSv3. For home use, sshfs is nifty, but that sure doesn't scale.

And then of course there's NFSv4, with sec=krb5p. Great in theory, but after ten years, it seems to be troublingly unused in the real world. The Linux client has just now had the experimental tag removed. And if you look at EMC Celerra, Isilon, etc., it's all NFSv3. (Celerra supports NFSv4, but it's really buried in the documentation. Isilon apparently worked at adding the RPCGSS support to FreeBSD, so maybe it's coming, but it's not there now. ) I can't even tag this post as "nfsv4" because I'm new here and that'd be a new tag.

So, really. What are you all doing?

Answer 1

Since it's a specific question (What are you all doing), let's answer it: nothing. Most administrators and users just don't worry about NFS security, so everybody uses NFSv3. It's typically a controlled environment (in the sense that only well-known machines can attach to the network in the first place). If somebody gets caught abusing the infrastructure, they get fired or to jail.

For data that you really don't want anybody to be able to read, you encrypt them explicitly, e.g. Firefox password databases, ssh keys, or pgp keys. You do that because you know the admin could read them on the file server, so network file system security would not be of any help, anyway.

Answer 2

You seem to be asking two questions here:

What are we actually using? and What does this?

What I'm actually using is CIFS, in my use-cases POSIX is less important so I haven't had any problems. NFS3 is used in areas where security isn't important, such as my SLES install server. And finally, sshfs/gvfs for simple user-land sharing. Wireline encryption is not deemed needed, so that isn't a meaningful factor for us.

As for the other question, there seems to be six main requirements for what you're looking for:

1. Encrypts traffic on the wire.
2. Encrypts authentication.
3. Posix semantics.
4. Strong enforcement of server-based ACLs.
5. Is not userland.
6. Is actually used.

I suspect points 5 and 6 will be the killers here, but here goes (also, this is the point where a table would be really handy, but markdown/StackExchange doesn't support it).

NFSv3 + IPSec

1. Encrypted on the wire, pass
2. No encrypted authentication, fail
3. Posix semantics, pass
4. No strong enforcement of server-based ACLs, fail
5. Is not userland, pass
6. Is actually used, pass

NFSv4 + Krb + IPSec

1. Encrypted on the wire, pass
2. Encrypted authentication, pass
3. Posix semantics, pass
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is not actually used, fail

CIFS

1. Not encrypted on the wire, fail
2. Encrypted authentication
3. Posix semantics, pass (Samba & Kernel now, Windows has had a Posix layer since the NT days)
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is actually used, pass

CIFS + IPSec

1. Encrypted on the wire, pass
2. Encrypted authentication
3. Posix semantics, pass (Samba & Kernel now)
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is not actually used, fail

SSHFS

1. Encrypted on the wire, pass
2. Encrypted authentication, pass
3. Posix semantics, pass
4. Strong enforcement of server-based ACLs, pass
5. Is userland, fail
6. Is actually used, pass

AFP/NetATalk

1. Encrypted on the wire, fail
2. Encrypted authentication, pass
3. Posix semantics, pass
4. Strong enforcement of server-based ACLs, pass
5. Is not userland, pass
6. Is actually used, fail

And I'm not touching the distributed file-systems out there. There simply isn't one single thing that does it all. Some come close (CIFS) and some are already there but no one uses them (NFS4 + IPSec, CIFS+IPSec). For some reason a secure network filesystem is something that has been subjected to a lot of compromises over the years.

Answer 3

I've been using openafs in production for years, with both Linux and Windows clients. It works great, has an active development community, and has gotten much easier to install and administer over the last few years as the various linux distros have included packaging for it. It has its warts, but I've found that they are offset by more administrative flexibility, the ability to have clients and servers separated by slow links, the ease of offsite backups, and other positive AFSisms.

One thing I like in particular is running production internet-facing web servers on openafs, with the ACLs locked down. Without a kerberos ticket there is no process on the machine -- even one running as root -- that can write to the filesystem. I can't count how many times we've noticed attacks utterly fail because of that simple measure.

There are some pretty large openafs users -- the largest commercial user I know of is Morgan Stanley.

Answer 4

How about OpenAFS which is still alive and a VPN under it because its only encryption at the moment is DES.

Answer 5

I see that a lot of people in this thread are talking about data hiding, i.e. attacks not being able to snoop on your data. A equally important to think about data integrity and authenticity. Are those nfs packet really from your nfs server? Did a nfs packet get changed in transit?

Answer 6

Well, to me it sounds like one of those Distributed Filesystems would be for you. I wouldn't quite want to recommend OpenAFS, as it is old, doesn't support IPv6 yet,..

I'm quite happy with GlusterFS myself. Gluster is pretty mature, performs okay and has a good feature set. However, as recently discussed in IRC, Gluster does not support IPv6 in a stable manner either. This feature will be scheduled for 3.6 or 3.7.

There is also a project called HekaFS, which builds on Gluster, which adds more advanced Authentification features and SSL. It is imo extremely well documented and very well designed.

What may alos be of interest to you is XtreemFS, which is designed for global grid computing, so it comes with SSL and stuff by default. My choice for usage fell on Gluster though, since the community seems more active and it way better documented.

Both are posix compliant of course.

Answer 7

I use NFS. But server to server NFS is done over a dedicated network backbone so encryption is not needed and authentication is sort of pointless. Just set each export to only share a select directory to a server based on IP.

Answer 8

With all due respect, you are completely looking at this problem the wrong way and you should back away from the console for a few hours.

Pretty much all storage io is unencrypted because it doesn't matter at that layer of the abstraction stack. Doubt it? Put a tap on your brocade fibre switch and you will see that fibre channel,just like iscsi and nfs, is all an unencrypted mess - by design. Solving that is a medium problem, not a storage protocol issue. For instance, want secure and encrypted nfs? Created a lan that's encrypted point to point between the nfs client and server using ipsec/ssl/tls or a pure hardware solution.