Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Alright, I'm hoping that this question is unique enough to not be considered another "What's the difference between LVM and RAID?" questions. Here goes...

I'm setting up a linux server which has a drive cage which will allow me to hot-plug in about 6 drives. The characteristics which I'm after are:

  • Tolerance of drive failure (so... RAID, basically)
  • On-line replacement of failed drives
  • High utilization of drive space (ie, if I have 6 1TB drives, 5TB with one parity drive is preferable to 1TB on 6 mirrors)
  • Flexible allocation of volumes across the total drive space (so... LVM)
  • Here's the curveball: I also want to be able to, as much as possible, replace the drives with larger ones in the future with a minimum of down-time on the server.

If I only wanted to be able to add space by replacing drives with smaller ones, I could just use LVM with lvreduce (provided, of course, I'm using a shrinkable FS like Reiser or ext[2-4] and also that the current FS can be shrunk by the amount of the drive I want to pull). But that doesn't give me any redundancy.

The first solution I can think of to get both redundancy and the ability to replace drives is to use the drives as mirrored-pairs. One pair would be md0, next pair would be md1, etc. Then, I'd add those to a single volume-group. When I wanted to add space, I'd reduce one of the md* devices out of the volume-group, pull the two drives, add two bigger ones, make a new md mirror, and add them to the volume-group, expand the FS, and I'm done.

But that takes my space efficiency down to 50%, because each physical drive gets it's own mirror. I only get 3TB... and it's going to make me need to replace with larger drives sooner.

Of course, instead of mirrored-pairs, I could plug them as 3-drive RAID5 sets. Every 3 drives would give me 2TB. So, I'd get a total of 4TB, but now I'm running into a problem where, in order to pull a md device from the volume group, it takes a lot more space with it (pulling one would take 2TB instead of 1TB). This means that I'll need a lot more slack space in the FS or else I can't swap the drives out like this.

So, this is where I'm stuck. The only way I know of to get redundancy with heterogeneous drive sizes and on-line drive replacement requires me to make a trade-off decision between the higher space efficiency and higher slack-space requirement in order to pull a drive (or set of drives).

Is there some clever trick that I'm missing?

share|improve this question

2 Answers 2

I think what you are proposing (LVM with multiple RAID-1 PVs) is correct, except for one detail: you don't necessarily need to pull a RAID set from your VG to replace it:

  1. Swap one disk in one of the RAID sets (mdadm fail, remove, physical swap, re-add)
  2. Rebuild
  3. Swap the second
  4. Rebuild
  5. mdadm --grow
  6. pvresize

Of course, this assumes that you have good backups, and that your single disk doesn't die while you're doing the rebuild.

share|improve this answer

One solution would be to rely on ZFS as the filesystem. However it is common to Solaris, several BSD variants, so not necessary on Linux. One possible solution would be to use Debian in the Debian/kFreeBSD variant, meaning a BSD kernel with a GNU userland.

ZFS is ideal for a very dynamic allocation of storage from a ZFS pool. With 6 drives you could either use one RAID-2Zs, which means from the 6 drives two can fail. However the price to pay is that you can only use disk space corresponding to 4 drives. There are other possibilities as in ZFS it is unimportant from which drive configuration a pool is built up.

Take a look at

share|improve this answer
    
Always nice getting the answer downvoted and not knowing why... –  GorillaPatch Dec 15 '12 at 10:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.