Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I need to build a fault tolerant, highly available key/value storage (no posix, only same functionaluty as S3) using cheap existing hardware. The storage should be able to handle several billions of items. The maximum size of items is around 1GB, most are only several KB. What's the best software/ filesystem for this task?

I already had a brief look at mogilefs, mongodb (grid-fs) & glusterfs but I'm not really sure which is stable & fault tolerant enough. The simpler the setup and later expansion the better :).


share|improve this question
why is S3 not able to satisfy this need? –  anthonysomerset Mar 10 '11 at 16:56
It would be, but we already have lots of unused disk space and so using our own cluster storage instead of paying huge amounts for data & traffic seems a much better option for us. –  gucki Mar 10 '11 at 21:21

3 Answers 3

There are others but OCFS2 will do all of that.

share|improve this answer
OCFS2 seems much to complicated to set up and maintain imo. If I remember correctly it's also not designed for environments where failures can happen quite often (we use cheap hardware). –  gucki Mar 10 '11 at 21:23

I would look at GPFS and GLusterFS. Those are the two that we are exploring right now for our enterprise needs.

share|improve this answer
As far as I have seen GPFS is not free and requires highend (reliable) hardware? –  gucki Mar 10 '11 at 21:22
GPFS is most decidedly not free, and the support required to get it running reliably will be costly as well. My experience with it has been at arms' length in an HPC environment, but the executive summary has been: GPFS' performance is excellent, reliability is...not. –  Jeff Albert Mar 10 '11 at 23:15
My apologies on this - I forgot that GPFS is a paid solution. –  Mike Mar 11 '11 at 21:32

It really depends on the storage targets you're addressing, the clients you intend to serve to, and what your performance and reliability requirements are. Ceph comes to mind, if the answers to those questions involve 'Linux with stock 2.6.43 or greater kernel', and 'relatively unencumbered by scary corporate ownership', as has been the recent concern with Lustre (now wholly-owned by Oracle).

share|improve this answer
Ceph looks really great, thought I haven't tried it yet. Too bad it's still experimental. As I only need an S3 replacement (no real posix compatible filesystem), what would you suggest. The usage is the same as what's S3 used for (mostly static website assets). –  gucki Mar 11 '11 at 15:49

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.