First some background: I work at a company that runs a PHP-webapplication. We have a storage backend mounted over NFS on several webservers. Today we have the issue if one webserver writes a file over NFS, sometimes the file does not appear at other mounted clients until a few minutes later. It is also not redundant so we cannot perform any "invisible" maintenance.
I've been looking at migrating to a GlusterFS solution (two or three replicated bricks/machines for redundancy). Now, using XFS as the storage filesystem "behind" Gluster works very well, performance wise. Gluster also does not seem to have the sync problem mentioned above.
However, I would like to use ZFS as the backend filesystem, the reasons being;
- Cheap compression (currently storing 1.5TB uncompressed)
- Very easy to expand the storage-volume "live" (one command, compared the LVM mess)
- Snapshotting, bit-rot protection and all the other ZFS glory.
In my demo-setup of the solution I have three servers with Replicated Gluster with a ZFS backend pool at a separate disk on each server. I'm using CentOS 6.5 with ZFS on Linux (0.6.2) + GlusterFS 3.4. I have also tried with Ubuntu 13.10. Everything is in VMware ESX.
To test this setup I have mounted the volume over Gluster, and then running BlogBench (http://www.pureftpd.org/project/blogbench) to simulate load. The issue I'm having is that at the end of the test, the ZFS storage seems to get stuck in a deadlock. All three machines have "zfs_iput_taskq" running at 90-100% CPU, and the test freezes. If I abort the test, the deadlock does not go away, only option seems to be hard reboot.
I have tried:
- Disabled atime
- Disabled scheduler (noop)
- Different compression/no compression
- Blogbench directly on ZFS works fine
- Blogbench on Gluster + XFS as backend works fine
Ideas? Should I just drop ZFS and go with something else? alternatives?