I am running a storage system. Fault tolerancy is handled at application level (files are stored on multiple servers, even datacenters).
Therefore I don't need nor even want redundancy within one server (Raid 5 would be OK but I run commodity budget servers, so anything that is resource intensive or requires expensive hardware is bad).
Currently I am using LVM to create a volume group out of all the drives in the server and then I put one big volume on it.
This works fine so far.
The problem is that I want to use bigger servers in future that contain up to 40+ drives.
The possibility of one drive failing and thus destroying the whole volume group is unacceptably high. The data IS stored elsewhere and can be recovered by the application, but restoring 100+ TB over internet peering is damn expensive and takes ages.
If one drive fails I would love to only loose the files stored on that disk.
It would be also perfectly if the files could be stored as a whole on a disk and be spread across multiple disks somewhat evenly (for performance reasons and reducing the amount of data that needs to be restored if one drive fails).
Is there something that suits my needs?
I think its a very common concern in big lvm volume groups.
A little program that presents a "merged view" and mounts via fuse would be perfect.
Unfortunately my application is not flexible enough to handle multiple volumes inside one server.
I know there are recovery means for lvm, but honestly, I really don't want to run them in a productive system...
Also: How does lvm "spread" data across multiple devices in a volume group? Does it add to the first one until its full and then starts the second one? Or is it more like randomly somewhere?
How is this not a real question??? This question is not about "what would be a good way too..." this question is also not about redundancy of fault tolerance.
My question is very specific. It is just a bit out of the ordinary. This is why I need to ask a question and not type it into google.
What program can I use to join multiple drives into one big volume AND only loose the files stored on ONE drive if ONE drive fails.
Raid -> does not do that. Raid can store redundancy information to prevent loss int he first plase. Raid can ALSO join multiple drives in to one.
Imagine a little secretary that gets a file. puts it on a drive that has free space. And if you request it he searches for it or remembers where it was and gives it to you from that drive.
If one drive is gone, the little secretary apologizes and says that he doesnt have the files on that device any more, but he can give you all the others.