First some background: I work at a company that runs a PHP-webapplication. We have a storage backend mounted over NFS on several webservers. Today we have the issue if one webserver writes a file over NFS, sometimes the file does not appear at other mounted clients until a few minutes later. It is also not redundant so we cannot perform any "invisible" maintenance.

I've been looking at migrating to a GlusterFS solution (two or three replicated bricks/machines for redundancy). Now, using XFS as the storage filesystem "behind" Gluster works very well, performance wise. Gluster also does not seem to have the sync problem mentioned above.

However, I would like to use ZFS as the backend filesystem, the reasons being;

  • Cheap compression (currently storing 1.5TB uncompressed)
  • Very easy to expand the storage-volume "live" (one command, compared the LVM mess)
  • Snapshotting, bit-rot protection and all the other ZFS glory.

In my demo-setup of the solution I have three servers with Replicated Gluster with a ZFS backend pool at a separate disk on each server. I'm using CentOS 6.5 with ZFS on Linux (0.6.2) + GlusterFS 3.4. I have also tried with Ubuntu 13.10. Everything is in VMware ESX.

To test this setup I have mounted the volume over Gluster, and then running BlogBench (http://www.pureftpd.org/project/blogbench) to simulate load. The issue I'm having is that at the end of the test, the ZFS storage seems to get stuck in a deadlock. All three machines have "zfs_iput_taskq" running at 90-100% CPU, and the test freezes. If I abort the test, the deadlock does not go away, only option seems to be hard reboot.

I have tried:

  • Disabled atime
  • Disabled scheduler (noop)
  • Different compression/no compression
  • Blogbench directly on ZFS works fine
  • Blogbench on Gluster + XFS as backend works fine

Ideas? Should I just drop ZFS and go with something else? alternatives?

Regards Oscar

  • How much RAM do you have on the ZFS nodes? Also, how many disks per node? Are you using zvols at all? – ewwhite Mar 21 '14 at 9:30
  • 3GB per node, one disk - since it is a labb-environment, each nodes ZFS-disk is only 5GB (so ram is supposedly enough). Should I try to crank up the ram anyways? – MannerMan Mar 21 '14 at 9:36

ZFS on Linux needs a bit of basic tuning in order to operate well under load. There's a bit of a struggle between the ZFS ARC and the Linux virtual memory subsystem.

For your CentOS systems, try the following:

Create an /etc/modprobe.d/zfs.conf configuration file. This is read during the module load/boot.

Add something like:

options zfs zfs_arc_max=40000000000
options zfs zfs_vdev_max_pending=24

Where zfs_arc_max is roughly 40% of your RAM in bytes (Edit: try zfs_arc_max=1200000000). The compiled-in default for zfs_vdev_max_pending is 8 or 10, depending on version. The value should be high (48) for SSD or low-latency drives. Maybe 12-24 for SAS. Otherwise, leave at default.

You'll want to also have some floor values in /etc/sysctl.conf

vm.swappiness = 10
vm.min_free_kbytes = 512000

Finally, with CentOS, you may want to install tuned and tuned-utils and set your profile to virtual-guest with tuned-adm profile virtual-guest.

Try these and see if the problem persists.


Run zfs set xattr=sa storage. Here's why. You may have to wipe the volumes and start again (I'd definitely recommend doing so).

  • Thanks a lot for your input, I have performed the changes you suggested and I'm running blogbench again, will post results. – MannerMan Mar 21 '14 at 10:00
  • Unfortunately, the issue remains. Strangely enough only one node got locked up this time.. still caused the cluster to be deadlocked though. Also increased ram to 4gb per node. – MannerMan Mar 21 '14 at 11:53
  • 1
    @MannerMan I'd need stack trace or dmesg output when this happens, then. – ewwhite Mar 21 '14 at 12:22
  • 1
    This may not be a ZFS problem. But please post the output of zpool get all, zpool status -v, zfs get all and zfs list – ewwhite Mar 21 '14 at 13:22
  • 1
    @MannerMan Excellent. – ewwhite Mar 21 '14 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.