Ceph is a pretty interesting one, with some neat features. One that's particularly cool is that the replication function (which decides what OSDs data goes to) is really flexible, and can be tuned for your reliability needs.
The general idea is that there are 3 types of daemons:
- OSDs, which store objects (basically raw bytestreams)
- MDSs, Metadata Servers, which provide filesystem semantics
- MDSs also do adaptive load balancing - if some metadata is really heavily accessed, it'll spread to more MDSs to share the load.
- Monitors, which maintain cluster state and configuration
The client has been upstream in the Linux kernel for some time now, and the server stuff runs entirely in userspace.
As far as performance goes, the original PhD thesis on Ceph noted that at 24 OSDs, the bottleneck was the throughput of the network switch and that performance scaled linearly with the number of nodes. (see the publications section on the ceph site). That was five years ago, and there has been a great deal of tuning since then.
On the subject of reliability, the project was started by the founder of Dreamhost and is being rolled out in their infrastructure.