Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm aware there are many similar questions around, several with great answers. I still haven't quite found what I'm looking for though: a distributed, scale-out FS that supports snapshots. Gluster with finished snapshot support would be great, but they seem to still be in the concept phase for that...

The system should be able to scale from about 20-40TB to beyond 100TB in multiple steps and ideally be liberally licensed and open source. I am somewhat interested in commercial solutions if they are very affordable (complete solutions with hardware are probably out of the picture). GPFS sounds interesting.

Important:

  • snapshots!
  • high availability
  • commodity or standard server hardware (no Infiniband or FibreChannel)
  • run on POSIX system

Not that important:

  • data locality awareness
  • performance (both in terms of throughput and latency)
  • POSIX compliance (ACLs, permissions,...)

I have considered using Gluster with ZFS for the bricks and simply snapshotting on the brick level, but I doubt that this would result in usable snapshots on the Gluster-level.

My other consideration would have been to use ZFS on a Ceph block device, with the danger of running into issues such as https://github.com/zfsonlinux/spl/issues/241, and probably bad performance. I am also worried about ZFS being so far removed from the actual hardware. Does anyone have experience with such a setup?

share|improve this question
    
What if this does not exist? Then what? –  ewwhite Feb 28 at 13:18
    
I would first try a snapshot-enabled FS on a Ceph block device. The hard part would be to build a test system that would realistically reflect the performance and reliability of the full system. If that turned out not to work, I would probably take a look if snapshotting ZFS-bricks on Gluster can somehow be made to work (maybe live with taking the volume offline daily for a few seconds, and running automated snapshots). Other than that, probably moving the HA/distribution part away from the FS by using some existing solution, and again snapshotting on the level of separate FSs. –  admr Feb 28 at 13:30
1  
Speaking of hardware, will you be running this on a SAN? Why don't you just snapshot the SAN? –  BigHomie Feb 28 at 13:32
    
Well, ideally I would have run Gluster on 1GE/10GE-IP-connected (both between nodes, and to the rest of the system) servers. By snapshotting a SAN you mean for example running ZFS on a SAN, and using ZFS-level snapshots? How would this cope with failure of one the attached servers? Would one adress this by e.g. creating ZFS volumes where redundant "disks" are located on different servers in the SAN? I don't know much about SANs, you might be able to tell ;) –  admr Feb 28 at 14:04

2 Answers 2

Have you looked into CephFS (rather than running a filesystem on a Ceph block device)? I haven't used it in production, but it claims to have the features you want (including snapshots of the filesystem or arbitrary subtrees).

share|improve this answer
    
Sorry for the late comment - I have looked into it as it does, as you say, offer what I need. According to the Ceph people however, it's not production-ready yet, and the mailing list occasionally seems to confirm that. –  admr Apr 24 at 10:15

Consider QuantaStor. They are experienced with Gluster atop ZFS (and the 30% performance hit). Ask them about snapshotting strategies.

share|improve this answer
    
That does look interesting. Thank you! I can probably live with that performance impact - this is mostly intended as a backup system. –  admr Feb 28 at 14:11
    
I talked to them - sadly, they can't do snapshots in that scenario either. –  admr Mar 4 at 8:39
    
@admr Bummer. I'm sorry. I'm not sure if what you're asking for really exists... –  ewwhite Mar 4 at 8:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.