I'm aware there are many similar questions around, several with great answers. I still haven't quite found what I'm looking for though: a distributed, scale-out FS that supports snapshots. Gluster with finished snapshot support would be great, but they seem to still be in the concept phase for that...
The system should be able to scale from about 20-40TB to beyond 100TB in multiple steps and ideally be liberally licensed and open source. I am somewhat interested in commercial solutions if they are very affordable (complete solutions with hardware are probably out of the picture). GPFS sounds interesting.
Important:
- snapshots!
- high availability
- commodity or standard server hardware (no Infiniband or FibreChannel)
- run on POSIX system
Not that important:
- data locality awareness
- performance (both in terms of throughput and latency)
- POSIX compliance (ACLs, permissions,...)
I have considered using Gluster with ZFS for the bricks and simply snapshotting on the brick level, but I doubt that this would result in usable snapshots on the Gluster-level.
My other consideration would have been to use ZFS on a Ceph block device, with the danger of running into issues such as https://github.com/zfsonlinux/spl/issues/241, and probably bad performance. I am also worried about ZFS being so far removed from the actual hardware. Does anyone have experience with such a setup?