1

Dear Serverfault community,

After researching a number of distributed file systems for deployment in a production environment with the main purpose of performing both batch and real-time distributed computing I've identified the following list as potential candidates, mainly on maturity, license and support:

  • Ceph
  • Lustre
  • GlusterFS
  • HDFS
  • FhGFS
  • MooseFS
  • XtreemFS

The key properties that our system should exhibit:

  • an open source, liberally licensed, yet production ready, e.g. a mature, reliable, community and commercially supported solution;
  • ability to run on commodity hardware, preferably be designed for it;
  • provide high availability of the data with the most focus on reads;
  • high scalability, so operation over multiple data centres, possibly on a global scale;
  • removal of single points of failure with the use of replication and distribution of (meta-)data, e.g. provide fault-tolerance.

The sensitivity points that were identified, and resulted in the following questions, are:

  1. transparency to the processing layer / application with respect to data locality, e.g. know where data is physically located on a server level, mainly for resource allocation and fast processing, high performance, how can this be accomplished? Do you from experience know what solutions provide this transparency and to what extent?
  2. posix compliance, or conformance, is mentioned on the wiki pages of most of the above listed solutions. The question here mainly is, how relevant is support for the posix standard? Hadoop for example isn't posix compliant by design, what are the pro's and con's?
  3. what about the difference between synchronous and asynchronous opeartion of a distributed file system. Though a synchronous distributed file system has the preference because of reliability it also imposes certain limitations with respect to scalability. What would be, from your expertise, the way to go on this?

I'm looking forward to your replies. Thanks in advance! :)

With kind regards,

Tim van Elteren

2
  • This is an excellent question on an interesting topic, but I fear it may tend too much towards a discussion rather than an objective question.
    – EEAA
    Oct 28, 2013 at 14:27
  • I have to agree with @EEAA Tim - you've obviously done quite a bit of research, but I'm not sure we can really provide the kind of information you're looking for in a Q&A format. I'll take a stab at some of it though.
    – voretaq7
    Oct 29, 2013 at 15:36

2 Answers 2

1

It sounds like you've already done a lot of the groundwork here. With respect to your key requirements, all of the filesystems you identified meet them to some extent. These are things distributed filesystems do by virtue of being distributed filesystems.

The only requirement that merits further discussion is licensing -- this is presumably a business consideration for your company, and serves to eliminate otherwise valid candidates which are not desirable for business reasons. That's a set of choices you need to make internally in consultation with the rest of your company, and it sounds like you already know what you want there.

We can't really tell you what filesystem to use (you'll need to do more research, and some lab testing/simulation would certainly be advisable -- take advantage of virtualization and the plethora of free hypervisors!), but I can give you a little insight into your sensitivity points"


Transparency to the processing layer / application with respect to data locality, e.g. know where data is physically located on a server level, mainly for resource allocation and fast processing, high performance, how can this be accomplished? Do you from experience know what solutions provide this transparency and to what extent?

Distributed filesystems are, in general, transparent to the rest of the OS (above the filesystem layer). The filesystem handles distributing the data to various nodes, replicating it around for fault tolerance, and moving it to/from client machines - your operating system just treats it like any other filesystem.

Any distributed filesystem that doesn't provide this level of abstraction is a non-starter in my view: client systems shouldn't need to know how the distributed FS works under the hood any more than they need to know the local disk organization of a NFS server.

Performance considerations are a separate issue - each filesystem you've mentioned above can be tuned to some extent. The best advice here is "Do your own benchmarks based on your workload", but you can also look at published benchmarks for a ballpark idea of what you can expect.


POSIX compliance, or conformance, is mentioned on the wiki pages of most of the above listed solutions. The question here mainly is, how relevant is support for the POSIX standard?

Only you can answer the question of relevance (based on your requirements), but as an old-school Unix guy I can tell you that I expect filesystems on a Unix/Linux host to conform to the POSIX specification - particularly in the most highly visible areas (filename restrictions, permissions, ACLs).

The Principle of Least Astonishment also dictates that on a Unix or Linux system your filesystems should conform to the POSIX standard, and expose permissions and ACLs through the operating system's standard tools, so I would consider that a strong vote in favor of using a distributed filesystem with a POSIX-conformant interface.


What about the difference between synchronous and asynchronous opeartion of a distributed file system. Though a synchronous distributed file system has the preference because of reliability it also imposes certain limitations with respect to scalability. What would be, from your expertise, the way to go on this?

Again, Only YOU can prevent forest fires make this call.

As you've rightly ascertained, synchronous distributed filesystems have some scalability concerns (mainly regarding write performance). They also have the advantage of strong consistency.
Asynchronous distributed filesystems almost always have better performance, but come with an inherent level of risk -- usually a greater chance of data inconsistency within the cluster, or data loss through transient single-points-of-failur while the data is being sync'd/replicated.

From my perspective (being an old-school Unix guy) consistency and stability trumps performance. There's a wonderful quote buried in McKusick's lecture on the history of FFS which is something to the effect of "Filesystems have to get it right because users get very upset when you lose their data" -- The corollary in the business/production environments world is that sysadmins get fired when they lose data that's worth millions of dollars, so taking an extra few milliseconds to ensure that data is properly replicated through the distributed filesystem makes a lot of sense to me, and I would elect to use a synchronous DFS unless there was a very good reason not to.

0

How fast do you need to read? And how important is global scalability?

Just off the top of my head, Lustre can be fast, but that's about the only feature it has from your list.

There are two things I'm aware that meet your requirements.

Xrootd - Very much read only, global HA, widely used in the science community.

http://xrootd.slac.stanford.edu/

OpenAFS - Been around a very long time and widely used in higher education.

http://www.openafs.org/

Neither will have the read speed of a properly tuned Lustre system, but both can scale to world wide deployments. What I know of the rest of the software on your list is that it is more suited to access in a single data center.

Not the answer you're looking for? Browse other questions tagged .