I'm working on an application that requires a large amount of storage space and I want to handle storage 'in-house' (Much cheaper than, say, S3) so we will have multiple servers (Initially 4) with large amounts of storage (6TB each).

The storage will need to be very flexible and configurable, each piece of data should be replicated on at least 2 servers and must be easily readable/writable from ether an API of a UNIX device/file/folder like a normal drive, I don't mind which. We must also be able to easily offload content to our HTTP CDN (Edgecast), it doesn't need to have built in HTTP support but if it doesn't I'm going to have to write something to get the files onto HTTP so they can be pulled by the CDN.

I've looked at a lot of solutions including

  • Eucalyptus Walrus
  • OpenStack Object Storage
  • MogileFS
  • and some others which I can't remember

All the servers will be running RHEL 6, they have 4x1.5TB drives which will be RAID1'd into a single partition. All the servers have 1GB/s connections between them and 100MB/s connections to the internet with unlimited bandwidth. They have 2x2.66ghz processors.

I understand there isn't a single, perfect answer but it would be nice to get some pointers.

  • 1
    . you are unlikely to get much useful discussion if you accept the first answer immediately. – tomjedrz May 8 '11 at 22:13

We very often get asked similar questions to this, I'm surprised none of these previous questions came up in the search as you asked this,

Either way I normally recommend GlusterFS or Lustre, but there are many other distributed file systems to choose.


Since you are already a RH shop the Red Hat Enterprise Clustering and Storage Management class may prove to be quite useful.


It would be good if you can share a little more about your storage workload. Depending on whether you require low latency or large content storage can lead to different storage solutions. For a content storage option then look at object storage solutions. (even those can be further subdivided - if you need to retain data for a long time and not read too often then consider solutions which support erasure coding). If you need low latency then look at clustered host based storage filesystems - ideally in conjunction with flash in the host. Note it is possible to also look at object storage in conjunction with flash cache acceleration in the host.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.