Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to store 100+ tb data on several servers, so I understand this is possible with distributed file systems. I also need the system be fault tollerant.

So if one server is down, I do not want the data to die because of this. Also I want to do this in the most [space] efficent way.

I like how RAID5 works for example. Can I build a RAID5 like system between a few servers, or are there file systems that do the same thing as RAID5 between several servers?

share|improve this question

migrated from Aug 8 '12 at 14:23

This question came from our site for computer enthusiasts and power users.

Might I suggest you do a lot more research on this before doing anything? Your suggestion of a distributed RAID5 system for 100+ TB of data strikes terror into my very soul. – HopelessN00b Aug 8 '12 at 14:39
Whatever you do, do NOT use RAID 5 ok, promise me that - I could go into the details but if you try to do this with RAID 5 either it'll just not work on day one, which would be a blessing of a kind, or it would rape you in your bed one night, you'd switch off your light, snuggle up with your loved one without knowing that RAID 5 was inside you, mangling your insides (and less metaphorically your data but that's less fun/sexually-violent) and you'd wake up with your arse (i.e. data) dripping in clotted blood (i.e. corrupt) - so please, PLEASE use RAID 6 or 10, ok with that? – Chopper3 Aug 8 '12 at 15:22
RAID6 or 10 are better? – lebron2323 Aug 9 '12 at 9:48
@lebron2323 for large volumes they're the ONLY options. – Chopper3 Aug 9 '12 at 11:05
raid 5 is fine for small raids with small disks, and that's the ideal setup for storage speed. If you end up with anything 7200RPM or lower, you'll absolutely need raid 6 or 10. – Basil Aug 9 '12 at 13:40

It sounds like your requirements are wildly unrealistic.

Firstly, 100 TB of data is a lot. Do you really need all of it to be available at the same time? If so, you need to be looking at a lot more than just spreading it out over several servers. There are networking and access considerations and performance constraints to be taken into mind. If you really have a legitimate need for this much online data, you're going to have a lot of simultaneous access to it. Are your servers up to the task of all those IOPS? Ad then there's the issue of backups... to backup 100+ TB of data, you're going to need one hell of a monstrous backup system requiring multiple tape libraries and/or D2D backup systems. The way you've asked this question makes me absolutely confident you haven't considered any of this, and you'll need to.

Setting up this much data is several projects in one (or one very big project, depending on how your company does things.) And please don't take this the wrong way, but you're clearly not up to the task. In truth, very few individuals in the world are up to this kind of task on their own, and none of them would make a comment about liking RAID5 and wanting to know how to spread that out over multiple servers or a filesystem that does the same thing.

More importantly, there's no space efficient, fault-tolerant way to distribute large data sets. At a fundamental level, you sacrifice space efficiency for fault tolerance, because if one of your distributed nodes goes down, the only way to continue to provide access to the data it held is to have another copy or copies of that data in some form. So you can either have fault-tolerant access to your data, or you can have efficient use of your available disk space, not both. To have basic fault tolerant storage of 100+ TB of data, you'll need at least twice as much (200+ TB) disk space, period.

On a somewhat related note/tangent to the above, RAID5 is not magic. It achieves redundancy through calculation of parity bits. This allows space savings at the cost of computation time. As a method of fault tolerance, it is not really a good idea on large data sets either, because you will almost definitely run into a bit-level error at some point, eliminating your fault tolerance, and is very computationally intensive. It can take DAYS to rebuild a single failed disk on a large-ish RADI5 array. How long do you think it will take to do more complex parity calculations on say, 20TB of data from a failed node?

share|improve this answer
It's about this time that I'd be talking to vendors like Hitachi for their HNAS or Universal Storage platform, and Brocade for a very large 10Gbit Ethernet fabric. – Tom O'Connor Aug 8 '12 at 16:37
-1. 100TB is NOT "hugh". and it is not a hugh pain to back up. It is doable in ONE rack server (see: backblaze storagepod) and all you need is a decent LOWER END tape storage library to back it up. I know quit a lot of smaller shopts that run sizes like that or larger. Not something you do with "home level equipment", but not something terribly hard to do either. – TomTom Aug 9 '12 at 12:36
@TomTom Where did I say "hugh" (or huge)? The only place I can see that is in your comment. And regarding the rest, the current LTO-5 standard can handle 1.5TB (raw) per tape. For 100 TB, that's 67 tapes. For a single full backup. I don't know of any "LOWER END" tape storage libraries with that capacity, let alone the capacity you'd need to actually use it as desired, without rotating a completely fresh set of tapes in two or three times a week. I don't know how to say it tactfully, and I'm good at tha tshit anyway, so... no offense, but your comment is just plain wrong. – HopelessN00b Aug 9 '12 at 13:18
@TomTom: Are you sure you didn't misread the question or answer? 100TB. Not 100GB. – Basil Aug 9 '12 at 13:38
Yes, I did.… - 135TB for less than 8000 USD. This is NOT hugh by any means these days, just not "home level stuff". I am considering buying a little disc case with place for 60 discs - that is 180TB using 3TB discs. Need more space for my data ;) My last project DATABASE size was (oracle database) 21TB (yes, TB). It is NOT hugh with todays technology. – TomTom Aug 9 '12 at 21:14

For example GlusterFS can do this.

But you can find many other filesystems here:

share|improve this answer

Take a look at MooseFS, we are using it here with 105TB since 3 years and it works perfectly.

share|improve this answer
How in MooseFS replication work, how much additional space you need for replication to store 105TB? – lebron2323 Aug 9 '12 at 15:09
Total capacity of our hard disks is 120TB, down to 105TB with replication lost but we use only 62% of total capacity : lost can be more important. MosseFS split data in chunck and duplicate it on differents chunk servers. You just need to add a chunk server to increment total storage capacity. – profy Aug 10 '12 at 8:27

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.