Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am looking for DFS (distributed file system) that is fault tolerant and easy to maintain. I will have tons (100M+) of small files (from 1K to 500K). Files will be located in some directories what will build a logical structure of the data.

I will have average read load of 100Mb/s and write load 100Mb/s.

I would like some input as to which file system works best for the given requirements.

Any thoughts?

share|improve this question
up vote 2 down vote accepted

Ceph is a pretty interesting one, with some neat features. One that's particularly cool is that the replication function (which decides what OSDs data goes to) is really flexible, and can be tuned for your reliability needs.

The general idea is that there are 3 types of daemons:

  • OSDs, which store objects (basically raw bytestreams)
  • MDSs, Metadata Servers, which provide filesystem semantics
    • MDSs also do adaptive load balancing - if some metadata is really heavily accessed, it'll spread to more MDSs to share the load.
  • Monitors, which maintain cluster state and configuration

The client has been upstream in the Linux kernel for some time now, and the server stuff runs entirely in userspace.

As far as performance goes, the original PhD thesis on Ceph noted that at 24 OSDs, the bottleneck was the throughput of the network switch and that performance scaled linearly with the number of nodes. (see the publications section on the ceph site). That was five years ago, and there has been a great deal of tuning since then.

On the subject of reliability, the project was started by the founder of Dreamhost and is being rolled out in their infrastructure.

share|improve this answer

GlusterFS, Lustre, etc... see for a list.

Also depends on what your trying to do. Workstations in a business accessing it? Internet-accessible?...?

share|improve this answer
With that many small files Lustre is not the answer. I would consider something with a distributed metadata. Although now filesystem likes that kind of load. I would look at a netapp as the performance required is small, and netapps are simple. – James Jun 30 '11 at 15:21

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.