Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Right now I have a FreeBSD hosts with ZFS and NFSv4. It is replicated to another FreeBSD box for backup purposes.

The ZFS features that are important to me are

  • software RAID6
  • snapshots, or some other way of replication to another host
  • quota
  • ACL
  • replace a failed disk without taking the host offline


Could the same or similar setup be done with XFS or GlusterFS on CentOS 6?


The hardware is

  • Supermicro CSE-847E16-R1400LPB chassis, 36 HS bays
  • Supermicro H8DG6-F AMD Dual G34 mainboard
  • AMD Opteron 6320, 2.8GHz 8-core, 8MB L2 cache, 6400MT
  • 64GB ram, and 128GB swap

Each host have 36*3TB space in RAIDZ2, so 100TB usable and 50TB used.

It seams that it is the Ubuntu clients that can crash the host on heavy reads. For now there are ~5 NFS clients. No read caching.

No NFSv4 tuning besides enabling Jumbo Frames

echo 'kern.ipc.nmbclusters="32768"' >> /boot/loader.conf

echo 'kern.ipc.maxsockbuf=16777216' >> /etc/sysctl.conf
echo 'net.inet.tcp.sendspace=262144' >> /etc/sysctl.conf
echo 'net.inet.tcp.recvspace=262144' >> /etc/sysctl.conf
echo 'net.inet.tcp.rfc1323=1' >> /etc/sysctl.conf
echo 'net.inet.tcp.sendbuf_max=16777216' >> /etc/sysctl.conf
echo 'net.inet.tcp.recvbuf_max=16777216' >> /etc/sysctl.conf
share|improve this question
Why do you want to move away from ZFS? – Joshua Miller Oct 8 '13 at 15:52
ZFS is perfect, but NFSv4 on FreeBSD crashes the host on heavy reads. – Sandra Oct 8 '13 at 16:02
Why not do it on the ultimate ZFS host: Solaris 11.1. If this is non-production it wouldn't cost you, otherwise you'll have to cough up $1K for your particular config. I believe that will give you a very stable (and fast) ZFS host and ditto for NFSv4. – unixhacker2010 Oct 18 '13 at 12:18
up vote 11 down vote accepted

♡ Hey there...

I read this question as really being a problem with the FreeBSD NFS stack...

ZFS works very well on the supported platforms. So much so, that I've moved most of my ZFS systems running Solaris and NexentaStor to Linux (RHEL/CentOS), thanks to the ZFS on Linux project. If you're using ZFS now, going to anything else is a step backwards.

I'm curious about the following, though:

  • How much data are you storing?
  • How many NFS clients do you have?
  • Have you performed any NFS tuning on your existing servers?
  • Are you using any form of L2ARC read caching on the existing setup? How much RAM do you have?
  • What is the hardware configuration of your servers?

Regardless of the answers to the above, you have a few options...

  • Fix or debug your FreeBSD issue. NFS shouldn't crash servers. It may be worth getting to the root-cause of this problem if you have a lot of time invested in this setup.
  • Convert to ZFS on another platform. NexentaStor, Linux, Solaris, OpenIndiana are pretty solid on the NFS side.

All-in, some combination of XFS and a cluster filesystem can so some of the same things as ZFS, but it's not a direct comparison. I don't think you should abandon ZFS yet.

share|improve this answer
I have now updated the OP. Is it correct understood that you use ZFSonLinux for production? If, how does it handle hot swap of failed disks? Have you had any problems due to ZFSonLinux? When reading about Openindiana, I can't see there is much of a community and package updates. Is that correct? – Sandra Oct 9 '13 at 0:24
I've been using ZFS in production for a year. See this post from last week about my experiences. It's been very stable. My only problem came when I needed some specific tuning in order to run ZFS on Linux with Fusion-io SSD cards. And yes, disk hot-swap works as long as your server's backplane supports it. – ewwhite Oct 9 '13 at 0:44
What about combining ZFS with GlusterFS? – CMCDragonkai Jun 20 '14 at 5:07

I would go for a mix of technologies.

You can also opt for the fairly new Btrfs if you like adventures.

share|improve this answer
I wouldn't be recommending BtrFS for anything these days... – ewwhite Oct 8 '13 at 18:20
@ewwhite Any pointers why? – johannes Oct 8 '13 at 18:58
@johannes btrFS just doesn't have momentum... It's the BetaMax of next-generation filesystems. – ewwhite Oct 8 '13 at 19:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.