Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to build a fault tolerant, highly available key/value storage (no posix, only same functionaluty as S3) using cheap existing hardware. The storage should be able to handle several billions of items. The maximum size of items is around 1GB, most are only several KB. What's the best software/ filesystem for this task?

I already had a brief look at mogilefs, mongodb (grid-fs) & glusterfs but I'm not really sure which is stable & fault tolerant enough. The simpler the setup and later expansion the better :).


share|improve this question
why is S3 not able to satisfy this need? – anthonysomerset Mar 10 '11 at 16:56
It would be, but we already have lots of unused disk space and so using our own cluster storage instead of paying huge amounts for data & traffic seems a much better option for us. – gucki Mar 10 '11 at 21:21

Minio is exactly built for such use case. It is open source, AWS S3 compatible API as single binary. You can run it on Windows, Linux & mac.

You can run it on your native filesystem say Ext4, NTFS, XFS.

Installation on Linux:

$ wget
$ chmod 755
$ ./minio server /datadir  

datadir: the directory which will store buckets and objects

You can access Minio client as well as Minio browser [bundled with the server]

mc implements the following commands.

  ls        List files and folders.
  mb        Make a bucket or folder.
  cat       Display contents of a file.
  pipe      Write contents of stdin to one or more targets. When no target is specified, it writes to stdout.
  share     Generate URL for sharing.
  cp        Copy one or more objects to a target.
  mirror    Mirror folders recursively from a single source to many destinations.
  diff      Compute differences between two folders.
  rm        Remove file or bucket [WARNING: Use with care].
  access    Manage bucket access permissions.
  session   Manage saved sessions of cp and mirror operations.
  config    Manage configuration file.
  update    Check for a new software update.
  version   Print version.

Hope it helps.

Disclaimer: I work for Minio

share|improve this answer

There are others but OCFS2 will do all of that.

share|improve this answer
OCFS2 seems much to complicated to set up and maintain imo. If I remember correctly it's also not designed for environments where failures can happen quite often (we use cheap hardware). – gucki Mar 10 '11 at 21:23

I would look at GPFS and GLusterFS. Those are the two that we are exploring right now for our enterprise needs.

share|improve this answer
As far as I have seen GPFS is not free and requires highend (reliable) hardware? – gucki Mar 10 '11 at 21:22
GPFS is most decidedly not free, and the support required to get it running reliably will be costly as well. My experience with it has been at arms' length in an HPC environment, but the executive summary has been: GPFS' performance is excellent, reliability is...not. – Jeff Albert Mar 10 '11 at 23:15
My apologies on this - I forgot that GPFS is a paid solution. – Mike Mar 11 '11 at 21:32

It really depends on the storage targets you're addressing, the clients you intend to serve to, and what your performance and reliability requirements are. Ceph comes to mind, if the answers to those questions involve 'Linux with stock 2.6.43 or greater kernel', and 'relatively unencumbered by scary corporate ownership', as has been the recent concern with Lustre (now wholly-owned by Oracle).

share|improve this answer
Ceph looks really great, thought I haven't tried it yet. Too bad it's still experimental. As I only need an S3 replacement (no real posix compatible filesystem), what would you suggest. The usage is the same as what's S3 used for (mostly static website assets). – gucki Mar 11 '11 at 15:49

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.