Alright, I'm hoping that this question is unique enough to not be considered another "What's the difference between LVM and RAID?" questions. Here goes...
I'm setting up a linux server which has a drive cage which will allow me to hot-plug in about 6 drives. The characteristics which I'm after are:
- Tolerance of drive failure (so... RAID, basically)
- On-line replacement of failed drives
- High utilization of drive space (ie, if I have 6 1TB drives, 5TB with one parity drive is preferable to 1TB on 6 mirrors)
- Flexible allocation of volumes across the total drive space (so... LVM)
- Here's the curveball: I also want to be able to, as much as possible, replace the drives with larger ones in the future with a minimum of down-time on the server.
If I only wanted to be able to add space by replacing drives with smaller ones, I could just use LVM with lvreduce (provided, of course, I'm using a shrinkable FS like Reiser or ext[2-4] and also that the current FS can be shrunk by the amount of the drive I want to pull). But that doesn't give me any redundancy.
The first solution I can think of to get both redundancy and the ability to replace drives is to use the drives as mirrored-pairs. One pair would be md0, next pair would be md1, etc. Then, I'd add those to a single volume-group. When I wanted to add space, I'd reduce one of the md* devices out of the volume-group, pull the two drives, add two bigger ones, make a new md mirror, and add them to the volume-group, expand the FS, and I'm done.
But that takes my space efficiency down to 50%, because each physical drive gets it's own mirror. I only get 3TB... and it's going to make me need to replace with larger drives sooner.
Of course, instead of mirrored-pairs, I could plug them as 3-drive RAID5 sets. Every 3 drives would give me 2TB. So, I'd get a total of 4TB, but now I'm running into a problem where, in order to pull a md device from the volume group, it takes a lot more space with it (pulling one would take 2TB instead of 1TB). This means that I'll need a lot more slack space in the FS or else I can't swap the drives out like this.
So, this is where I'm stuck. The only way I know of to get redundancy with heterogeneous drive sizes and on-line drive replacement requires me to make a trade-off decision between the higher space efficiency and higher slack-space requirement in order to pull a drive (or set of drives).
Is there some clever trick that I'm missing?