5

Alright, I'm hoping that this question is unique enough to not be considered another "What's the difference between LVM and RAID?" questions. Here goes...

I'm setting up a linux server which has a drive cage which will allow me to hot-plug in about 6 drives. The characteristics which I'm after are:

  • Tolerance of drive failure (so... RAID, basically)
  • On-line replacement of failed drives
  • High utilization of drive space (ie, if I have 6 1TB drives, 5TB with one parity drive is preferable to 1TB on 6 mirrors)
  • Flexible allocation of volumes across the total drive space (so... LVM)
  • Here's the curveball: I also want to be able to, as much as possible, replace the drives with larger ones in the future with a minimum of down-time on the server.

If I only wanted to be able to add space by replacing drives with smaller ones, I could just use LVM with lvreduce (provided, of course, I'm using a shrinkable FS like Reiser or ext[2-4] and also that the current FS can be shrunk by the amount of the drive I want to pull). But that doesn't give me any redundancy.

The first solution I can think of to get both redundancy and the ability to replace drives is to use the drives as mirrored-pairs. One pair would be md0, next pair would be md1, etc. Then, I'd add those to a single volume-group. When I wanted to add space, I'd reduce one of the md* devices out of the volume-group, pull the two drives, add two bigger ones, make a new md mirror, and add them to the volume-group, expand the FS, and I'm done.

But that takes my space efficiency down to 50%, because each physical drive gets it's own mirror. I only get 3TB... and it's going to make me need to replace with larger drives sooner.

Of course, instead of mirrored-pairs, I could plug them as 3-drive RAID5 sets. Every 3 drives would give me 2TB. So, I'd get a total of 4TB, but now I'm running into a problem where, in order to pull a md device from the volume group, it takes a lot more space with it (pulling one would take 2TB instead of 1TB). This means that I'll need a lot more slack space in the FS or else I can't swap the drives out like this.

So, this is where I'm stuck. The only way I know of to get redundancy with heterogeneous drive sizes and on-line drive replacement requires me to make a trade-off decision between the higher space efficiency and higher slack-space requirement in order to pull a drive (or set of drives).

Is there some clever trick that I'm missing?

2 Answers 2

2

I think what you are proposing (LVM with multiple RAID-1 PVs) is correct, except for one detail: you don't necessarily need to pull a RAID set from your VG to replace it:

  1. Swap one disk in one of the RAID sets (mdadm fail, remove, physical swap, re-add)
  2. Rebuild
  3. Swap the second
  4. Rebuild
  5. mdadm --grow
  6. pvresize

Of course, this assumes that you have good backups, and that your single disk doesn't die while you're doing the rebuild.

0

One solution would be to rely on ZFS as the filesystem. However it is common to Solaris, several BSD variants, so not necessary on Linux. One possible solution would be to use Debian in the Debian/kFreeBSD variant, meaning a BSD kernel with a GNU userland.

ZFS is ideal for a very dynamic allocation of storage from a ZFS pool. With 6 drives you could either use one RAID-2Zs, which means from the 6 drives two can fail. However the price to pay is that you can only use disk space corresponding to 4 drives. There are other possibilities as in ZFS it is unimportant from which drive configuration a pool is built up.

Take a look at

1
  • Always nice getting the answer downvoted and not knowing why... Dec 15, 2012 at 10:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .