Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am running a storage system. Fault tolerancy is handled at application level (files are stored on multiple servers, even datacenters).

Therefore I don't need nor even want redundancy within one server (Raid 5 would be OK but I run commodity budget servers, so anything that is resource intensive or requires expensive hardware is bad).

Currently I am using LVM to create a volume group out of all the drives in the server and then I put one big volume on it.

This works fine so far.

The problem is that I want to use bigger servers in future that contain up to 40+ drives.

The possibility of one drive failing and thus destroying the whole volume group is unacceptably high. The data IS stored elsewhere and can be recovered by the application, but restoring 100+ TB over internet peering is damn expensive and takes ages.

If one drive fails I would love to only loose the files stored on that disk.

It would be also perfectly if the files could be stored as a whole on a disk and be spread across multiple disks somewhat evenly (for performance reasons and reducing the amount of data that needs to be restored if one drive fails).

Is there something that suits my needs?

I think its a very common concern in big lvm volume groups.

A little program that presents a "merged view" and mounts via fuse would be perfect.

Unfortunately my application is not flexible enough to handle multiple volumes inside one server.

I know there are recovery means for lvm, but honestly, I really don't want to run them in a productive system...

Also: How does lvm "spread" data across multiple devices in a volume group? Does it add to the first one until its full and then starts the second one? Or is it more like randomly somewhere?


How is this not a real question??? This question is not about "what would be a good way too..." this question is also not about redundancy of fault tolerance.

My question is very specific. It is just a bit out of the ordinary. This is why I need to ask a question and not type it into google.

What program can I use to join multiple drives into one big volume AND only loose the files stored on ONE drive if ONE drive fails.

Raid -> does not do that. Raid can store redundancy information to prevent loss int he first plase. Raid can ALSO join multiple drives in to one.

Imagine a little secretary that gets a file. puts it on a drive that has free space. And if you request it he searches for it or remembers where it was and gives it to you from that drive.

If one drive is gone, the little secretary apologizes and says that he doesnt have the files on that device any more, but he can give you all the others.

share|improve this question
Most DFS filesystems/solutions do most of this... But at one part you mention "I don't need nor even want redundancy" then you mention "but restoring 100+ TB over internet peering is damn expensive and takes ages" which is one feature redunancy gives you... – HaydnWVN Jul 13 '12 at 10:45
How is your data layed out ? Typically, this could be handled easily by having one directory where you mount all your volumes as different folders - or better yet, as @HaydnWVN mentioned, use some form of DFS – Stephane Jul 13 '12 at 11:50
If you handle redundancy in your app then you don't need to combine drives into a volume, if you are handling it in hardware then you need RAID and/or a distributed file system. You havn't got answers that you like because the way you have narrowed down the scenario already is a bad way to do it. – JamesRyan Jul 13 '12 at 12:05
up vote 2 down vote accepted

I haven't tested this, but I'd mount every disk invididually and merge all of them together using a filesystem like UnionFS or mhddfs. That way, if a disk fails, only the files on that disk are lost, which is precisely what you are looking for.

share|improve this answer
exactly what i was looking for. thanks! – The Shurrican Jul 14 '12 at 17:14

This may be a stretch, but since you only care about the integrity of the files on a particular storage unit, look into the ZFS filesystem copies option mentioned here and detailed (as "ditto-blocks") on the Oracle site here and here. This is very much focused on file-level protection, but may be aligned with what you're looking for.

These days, ZFS is available for some Linux distributions, so a Solaris-derived OS isn't a strict requirement anymore.

share|improve this answer

No if's, no but's - you need some form of RAID in there on the actual machines, I know you're doing the whole replicating around bit but you can't expect that to deal with a single drive failure - even if you just have massive R6 arrays it's still better than nothing.

Other than that it sounds like you just need a clustered files system, something like gluster, hadoop dfs or similar.

share|improve this answer
No, I am perfectly fine with one drive failing! I just don't want to recover the whole server then. Raid will be expensive, resource intensive and waste storage. – The Shurrican Jul 13 '12 at 10:32
Sorry, you're not clear, if you want to handle one disk failing you have to use some form of RAID and if you're planning on using R5 for that many disks you're going to incur unrecoverable read errors, that's why I mentioned R6. – Chopper3 Jul 13 '12 at 10:34
I do NOT want to "handle". Its perfectly fine if they get lost. I just don't want to replicate the WHOLE server again if only one drive in it failed... Questions Title: How can I create one big volume from multiple drives, but loose only the files stored on one disk if it fails? – The Shurrican Jul 13 '12 at 10:36
Raid will be expensive, resource intensive and waste storage. False on the first two counts, true on the last. Software RAID comes with the most common server OS's today. I would argue software raid 5 would be better than one massive raid 0. I would seriously consider some form of RAID as this sounds, to be honest, a bit of a huge bodge job not really expected of users of this site. – tombull89 Jul 13 '12 at 10:45
If the app is already handling redundancy, RAID isn't needed and therefore should not be used: it just add complexity, costs and lower performances for not return – Stephane Jul 13 '12 at 11:47

If you are handling fault tolerance at the application level, I would use more cheaper/smaller servers instead. If the application is correctly distributed, this means that it will run even faster.

The other alternative (not sure if it is possible) might be to run more than one instance of the app directed to each one of the disks.

I think that Raid arrays are not a good fit for your use case.


share|improve this answer
thank you for the first useful answer! Treating every drive as a single server, which is basically what "multiple" instances would be is possible. But then the bandwdith awareness of the application when distributing files would not work any more. I also thought of that... – The Shurrican Jul 13 '12 at 11:09
The application was designed for multiple small servesr and works fine on that! If a server only holds 6 drives which they do at the moment, its totally fine if all of them fail. Its replicated fast. But its nowadays cheaper to make biiiig servers, and if they have to be restored as a whole its more effort and also the likelyhood of one drive failing in 40+ is higher than one in 6... so thats what i am worried about. – The Shurrican Jul 13 '12 at 11:10
But will your application without modification be able to restore the files missing from this single disk ?, because as you describe it seems that the system is programmed to replace entire servers and not particular files. (Now talking from ignorance so forgive me if this doesn't make sense) If the application can restore particular files, then why don't you just use the drives as they are, why do you need to build a big volume ? If you loose one of the drives then you loose only the part mounted on this drive. – Agustin Cautin Jul 13 '12 at 11:46

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.