I'm aware there are many similar questions around, several with great answers. I still haven't quite found what I'm looking for though: a distributed, scale-out FS that supports snapshots. Gluster with finished snapshot support would be great, but they seem to still be in the concept phase for that...

The system should be able to scale from about 20-40TB to beyond 100TB in multiple steps and ideally be liberally licensed and open source. I am somewhat interested in commercial solutions if they are very affordable (complete solutions with hardware are probably out of the picture). GPFS sounds interesting.


  • snapshots!
  • high availability
  • commodity or standard server hardware (no Infiniband or FibreChannel)
  • run on POSIX system

Not that important:

  • data locality awareness
  • performance (both in terms of throughput and latency)
  • POSIX compliance (ACLs, permissions,...)

I have considered using Gluster with ZFS for the bricks and simply snapshotting on the brick level, but I doubt that this would result in usable snapshots on the Gluster-level.

My other consideration would have been to use ZFS on a Ceph block device, with the danger of running into issues such as https://github.com/zfsonlinux/spl/issues/241, and probably bad performance. I am also worried about ZFS being so far removed from the actual hardware. Does anyone have experience with such a setup?

  • What if this does not exist? Then what?
    – ewwhite
    Feb 28 '14 at 13:18
  • I would first try a snapshot-enabled FS on a Ceph block device. The hard part would be to build a test system that would realistically reflect the performance and reliability of the full system. If that turned out not to work, I would probably take a look if snapshotting ZFS-bricks on Gluster can somehow be made to work (maybe live with taking the volume offline daily for a few seconds, and running automated snapshots). Other than that, probably moving the HA/distribution part away from the FS by using some existing solution, and again snapshotting on the level of separate FSs.
    – user206444
    Feb 28 '14 at 13:30
  • 1
    Speaking of hardware, will you be running this on a SAN? Why don't you just snapshot the SAN?
    – MDMoore313
    Feb 28 '14 at 13:32
  • Well, ideally I would have run Gluster on 1GE/10GE-IP-connected (both between nodes, and to the rest of the system) servers. By snapshotting a SAN you mean for example running ZFS on a SAN, and using ZFS-level snapshots? How would this cope with failure of one the attached servers? Would one adress this by e.g. creating ZFS volumes where redundant "disks" are located on different servers in the SAN? I don't know much about SANs, you might be able to tell ;)
    – user206444
    Feb 28 '14 at 14:04

Have you looked into CephFS (rather than running a filesystem on a Ceph block device)? I haven't used it in production, but it claims to have the features you want (including snapshots of the filesystem or arbitrary subtrees).

  • Sorry for the late comment - I have looked into it as it does, as you say, offer what I need. According to the Ceph people however, it's not production-ready yet, and the mailing list occasionally seems to confirm that.
    – user206444
    Apr 24 '14 at 10:15

Fast forward to 2019. Ceph is now around for quite some time and with the jewel Release Version 10.2.0 CephFS is declared production ready.

The Ceph setup has also been greatly reduced in complexity with the new BlueStore storage backend that was introduced in Ceph 12 Luminous release.

Give it a try.


Consider QuantaStor. They are experienced with Gluster atop ZFS (and the 30% performance hit). Ask them about snapshotting strategies.

  • That does look interesting. Thank you! I can probably live with that performance impact - this is mostly intended as a backup system.
    – user206444
    Feb 28 '14 at 14:11
  • I talked to them - sadly, they can't do snapshots in that scenario either.
    – user206444
    Mar 4 '14 at 8:39
  • @admr Bummer. I'm sorry. I'm not sure if what you're asking for really exists...
    – ewwhite
    Mar 4 '14 at 8:40

Spectrum Scale (GPFS)

Been around since 1998 (before if you consider Tiger Shark) and has most, if not all, the features you list

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy