I have a series of servers (HP ProLiant, 34 servers) each of which with 500 G of hard drive space. These servers are part of a computational cluster which runs processes that roughly fall into two "phases":

  • phase 1: reading a small number of large (up to 6Gb) files and writing relatively large (up to 1-2 Gb) files.
  • phase 2: reading and writing lots (hundreds) of small files which then are combined into larger files; also these files are generated along with temporary files which serve as "transactional points".

The servers don't share the same enclosure and are connected via Gbit Ethernet.

I initially put a NFS share on a single server, as per my earlier question, but the level of concurrency caused issues of availability and locking, thus causing the processes to fail in the second phase, very often.

Now that I can use the disks in the servers, I thought about using a distributed filesystem. My intial approach (used thanks to successful tests elsewhere) was to use GlusterFS (distributed+replicated setup).

However, while it works perfectly for the first phase, it causes issues to no end in the second, as the latency in the network is not low enough to handle all these concurrent reads and writes by all servers in the pool, causing the various servers to go out of sync, and thus weird errors around the way (missing files, strange permission denied errors...).

Also the "issue" is that the servers themselves (or a part of them, I don't need to use all) will need to run the computation along with providing storage (it's a donated resource, so I can't do more than that).

All this to explain the use case, which then prompts the question: what would be the best distributed file system to handle "phase 2"? Notice that I need something at the file level, e.g. either a mount point or a virtual device.

  • 1
    Could any of this processing be limited to local-only filesystem for each machine until the final result is determined? You mention temporary files, those should easily reside only on the local machine which should save quite a bit of network-writing. – Regan Oct 25 '13 at 22:38
  • That is a limitation of the project I'm using but I'm also working with their maintainers to reduce the issues. – Einar Oct 28 '13 at 6:38
  • Does it use any scheme by which you could use symlinks to manipulate where it gets stored? If the servers have tons of RAM, you could even try using a ram disk too once the maintainers make a possible change. – Regan Oct 28 '13 at 9:39

I believe LizardFS and GfarmFS are well suited for the task.

Both storage systems use metadata (i.e. directory) servers allowing low-latency operations on millions of files.
LizardFS "master" (i.e. metadata server) uses RAM (memory footprint is less than 3 GiB for 7_000_000 files) while GfarmFS metadata server uses PostgreSQL.


Filesystems make crappy databases, and network files systems make even worse ones.

Phase 2: smells a lot like a database to me.

There are lot's of choices out there these days. A basic key/value store database can be relatively simple to setup and maintain. This is a great book for finding out what choices are possible.


  • No, it's not a database. It's a distributed series of programs runnning in parallel to optimize the computation time. – Einar Oct 28 '13 at 6:37
  • If you need syncronized access to data across multiple processes, it's a database. Maybe not traditional SQL, but there is a reason you're having trouble finding a network file system that handles small files and read/write locks effectively. – Fred the Magic Wonder Dog Oct 28 '13 at 14:35
  • You might find some useful tools here [link] hadoop.apache.org – Fred the Magic Wonder Dog Oct 28 '13 at 14:42

To throw in my $.02:

Take a look at ceph. Throw at it half a gigabyte of memory on each server (make them all OSDs) and designate three servers as MDS/MON (depending on the load they may or may not run other intensive stuff as well). Use it as a object storage or as a block device or as a filesystem ... that is up to you ... It can be redundat and it can be fast. It scales (with some tweaking) to petabytes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.