Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Suppose that one has 100 machines (each with a 2TB hard drive), all connected in a network.

Is there a way (in Linux) to unite this combined 200TB of disk space into one folder, that can be shared using NFS among all machines that are in the network?

Going this route, all machines will be able to read/write to the same folder, and this way, I/O will be spread evenly between them. Is this a good idea if one needs a large file system to store 100s of TB of data? (note: data will be split to many smaller files, of size ~ 500GB each).

Is there a ready-made solution (prefereable, an open source one) that achieves just that?

share|improve this question
    
'I/O will be spread evenly between them' -- Keep in mind that any distributed solution will require bandwidth between the distributed servers ( e.g. host1 will need to have some knowledge of activity stemming from host2 though host 100). This bandwidth is non-trivial. –  Stefan Lasiewski Jul 12 '11 at 20:17
add comment

migrated from stackoverflow.com Jul 12 '11 at 20:06

This question came from our site for professional and enthusiast programmers.

6 Answers

up vote 7 down vote accepted

Two options that come to mind are GlusterFS and Hadoop HDFS.

share|improve this answer
    
Thanks; any words of info on the difference of installation / maintainance of GlusterFS vs. Hadoop? –  user76976 Jul 12 '11 at 19:59
1  
I haven't used HDFS; I imagine it fits will if you're already using Hadoop for parallel number-crunching. Gluster sounds more like what you're looking for -- a distributed NAS, in effect. –  bosmacs Jul 12 '11 at 20:04
    
Great; is it hard to install? when one wants to add another node to the picture, what is the overhead? –  user76976 Jul 12 '11 at 20:10
add comment

IBM's GPFS can do this (note: not open-source).

With GPFS you can create Network-Shared-Disks (NSDs) that are comprised of any type of block-storage (local or presented via iSCSI or FC, for example). It would be entirely possible to create a GPFS filesystem (device) that is comprised of NSDs that span each 2TB hard drive across your 100 servers.

I won't pretend to recall all the crazy marketing numbers, but GPFS is among the most popular clustered file systems for supercomputers in the "Top 500" list because it supports extremely large volumes of data and incredibly high parallel I/O. Wikipedia has some numbers.

GPFS can replicate data and metadata blocks across the filesystem. When you create your NSDs you define a "failure group", you do this so that GPFS does writes your block replicas elsewhere (that is, you don't want both copies of your block in the same failure group). You can also tier storage using their concept of "storage pools", through which you can define behavior like...files accessed in the last week live on my Fusion IO or SSD drives, but after that move the blocks to cheaper storage.

All the nodes in your cluster would have access to a device (like /dev/gpfs0) which they could mount and access as it the entire file system were local to each node. You mentioned NFS; in this model however, it is not necessary to introduce the additional protocol unless you have systems outside the 100 node cluster that are acting as consumers/clients of the data and you do not wish to make them GPFS clients/NSD servers (by loading the GPFS kernel module). However, you can trivially export GPFS file systems via NFS and even leverage Clustered-NFS (CNFS) if necessary.

  • I do not work for IBM, but I have played with GPFS a little bit and I liked it.
share|improve this answer
1  
thanks; I really prefer an open source solution though –  user76976 Jul 12 '11 at 23:06
add comment

I have not had a chance to play with it yet so I can't give a full review, but I would say have a look at the Openstack cloud stuffs -> http://www.openstack.org/projects/storage/

share|improve this answer
    
thank you Benjamin. –  user76976 Jul 12 '11 at 20:11
add comment

You probably want something like PVFS.

share|improve this answer
    
thanks ninjalj. –  user76976 Jul 12 '11 at 20:11
add comment

There is a full list on Wikipedia.

share|improve this answer
1  
It would be better to directly link the "Distributed parallel fault-tolerant file systems" –  yi_H Jul 12 '11 at 21:05
add comment

Ceph is another, though not prod ready yet. http://ceph.newdream.net/wiki/Main_Page

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.