I have a series of servers (HP ProLiant, 34 servers) each of which with 500 G of hard drive space. These servers are part of a computational cluster which runs processes that roughly fall into two "phases":
- phase 1: reading a small number of large (up to 6Gb) files and writing relatively large (up to 1-2 Gb) files.
- phase 2: reading and writing lots (hundreds) of small files which then are combined into larger files; also these files are generated along with temporary files which serve as "transactional points".
The servers don't share the same enclosure and are connected via Gbit Ethernet.
I initially put a NFS share on a single server, as per my earlier question, but the level of concurrency caused issues of availability and locking, thus causing the processes to fail in the second phase, very often.
Now that I can use the disks in the servers, I thought about using a distributed filesystem. My intial approach (used thanks to successful tests elsewhere) was to use GlusterFS (distributed+replicated setup).
However, while it works perfectly for the first phase, it causes issues to no end in the second, as the latency in the network is not low enough to handle all these concurrent reads and writes by all servers in the pool, causing the various servers to go out of sync, and thus weird errors around the way (missing files, strange permission denied errors...).
Also the "issue" is that the servers themselves (or a part of them, I don't need to use all) will need to run the computation along with providing storage (it's a donated resource, so I can't do more than that).
All this to explain the use case, which then prompts the question: what would be the best distributed file system to handle "phase 2"? Notice that I need something at the file level, e.g. either a mount point or a virtual device.