Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a series of servers (HP ProLiant, 34 servers) each of which with 500 G of hard drive space. These servers are part of a computational cluster which runs processes that roughly fall into two "phases":

  • phase 1: reading a small number of large (up to 6Gb) files and writing relatively large (up to 1-2 Gb) files.
  • phase 2: reading and writing lots (hundreds) of small files which then are combined into larger files; also these files are generated along with temporary files which serve as "transactional points".

The servers don't share the same enclosure and are connected via Gbit Ethernet.

I initially put a NFS share on a single server, as per my earlier question, but the level of concurrency caused issues of availability and locking, thus causing the processes to fail in the second phase, very often.

Now that I can use the disks in the servers, I thought about using a distributed filesystem. My intial approach (used thanks to successful tests elsewhere) was to use GlusterFS (distributed+replicated setup).

However, while it works perfectly for the first phase, it causes issues to no end in the second, as the latency in the network is not low enough to handle all these concurrent reads and writes by all servers in the pool, causing the various servers to go out of sync, and thus weird errors around the way (missing files, strange permission denied errors...).

Also the "issue" is that the servers themselves (or a part of them, I don't need to use all) will need to run the computation along with providing storage (it's a donated resource, so I can't do more than that).

All this to explain the use case, which then prompts the question: what would be the best distributed file system to handle "phase 2"? Notice that I need something at the file level, e.g. either a mount point or a virtual device.

share|improve this question
1  
Could any of this processing be limited to local-only filesystem for each machine until the final result is determined? You mention temporary files, those should easily reside only on the local machine which should save quite a bit of network-writing. –  Regan Oct 25 '13 at 22:38
    
That is a limitation of the project I'm using but I'm also working with their maintainers to reduce the issues. –  Einar Oct 28 '13 at 6:38
    
Does it use any scheme by which you could use symlinks to manipulate where it gets stored? If the servers have tons of RAM, you could even try using a ram disk too once the maintainers make a possible change. –  Regan Oct 28 '13 at 9:39

1 Answer 1

Filesystems make crappy databases, and network files systems make even worse ones.

Phase 2: smells a lot like a database to me.

There are lot's of choices out there these days. A basic key/value store database can be relatively simple to setup and maintain. This is a great book for finding out what choices are possible.

http://pragprog.com/book/rwdata/seven-databases-in-seven-weeks

share|improve this answer
    
No, it's not a database. It's a distributed series of programs runnning in parallel to optimize the computation time. –  Einar Oct 28 '13 at 6:37
    
If you need syncronized access to data across multiple processes, it's a database. Maybe not traditional SQL, but there is a reason you're having trouble finding a network file system that handles small files and read/write locks effectively. –  Fred the Magic Wonder Dog Oct 28 '13 at 14:35
    
You might find some useful tools here [link] hadoop.apache.org –  Fred the Magic Wonder Dog Oct 28 '13 at 14:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.