What are acceptable/reasonable/best ways to provide large, growable, shared storage at AWS, exposed as a single filesystem?

We're currently making 1TB EBS volumes ~biweekly and NFS exporting with no_subtree_check and nohide. In this setup, distinct exports appear under a single mount on the client. This arrangement does not scale well.

The options we've considered:

  • LVM2 with ext4. resize2fs is too slow.
  • Btrfs on Linux. not obviously ready for prime time yet.
  • ZFS on Linux. not obviously ready for prime time yet (although LLNL uses it)
  • ZFS on Solaris. future of this combo is uncertain (to me), and new OS in the mix
  • glusterfs. heard mostly good but two scary (and maybe old?) stories.

The ideal solution would provide sharing, a single fs view, easy expandability, snapshots, and replication.

Thanks for sharing ideas and experience.

  • 2
    Have you run the numbers on this? It strikes me as being prohibitively expensive... – Michael Hampton Oct 25 '12 at 3:04
  • Good question. The answer is sorta, but we're early. Our data need is high and compute need is bursty. So, it's not clear which costs more: AWS with expensive storage and cheaper peak computing solution, or local with expensive computing and cheaper storage. (I'm not even convinced that the fully loaded storage cost really is cheaper.) We'd probably stash archive data in Glacier to reduce cost (the constraints happen to work for us). – Reece Oct 25 '12 at 5:16
  • 1
    Don't forget the bandwidth costs. – Michael Hampton Oct 25 '12 at 5:19
  • About ZFS, it's either Solaris or FreeBSD, but as you said it, the future is somewhat uncertain for Solaris and open-source ZFS is stuck at version 28 (whatever the OS). – Ouki Oct 25 '12 at 8:28

How about XFS (on top of LVM)? It is more mature than ext4, and there is the possibility to do incremental dumps of the filesystem. You might want to check its wikipedia article and the benchmarks against ext4 referenced therein.

| improve this answer | |
  • Thanks, Petr. I had considered (and forgotten) the LVM+XFS combo. I have some trepidation about LVM2 and XFS, mostly around uncertainty and new operational complexity for us. I don't expect to get something for nothing (i.e., expandable storage for no effort), but I would like to minimize the effort and risk for us. For example, what's the implication of losing one EBS volume that's part of a RAID1 array? It might be a non-issue, but at this point such questions create uncertainty for me. In an era of ZFS and btrfs, I was hoping to sidestep such complexity. – Reece Nov 2 '12 at 19:18
  • @Reece Question is whether you trust the device mapper - iirc, most of the LVM is actually user-space tools backed by dm. You would need not-too-ancinet kernel though (2.6.33+ - see wiki). – peterph Nov 2 '12 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.