I run a multi-disk server with LVM2. Several large disks serve as LVM2 physical volumes for one volume group, containing one logical volume formatted with ext4. Nothing fancy, just your standard linear setup.

Recently an additional, very small disk was added as physical volume to that volume group and I expanded both the logical volume, and the ext4 file system therein onto that disk.

This lv is used to store incremental backups using rsync and is only about 30% full, there have rarely been any files deleted from it, only incremental writes.

Now this new HDD I added to the pre-existing volume group has unexpectedly died on me, and the volume group won't come up because it is missing one physical volume.

As fate will have it, this WAS the "in an event of catastrophic failure on the primary server"-backup, the event happened, the boss is not happy, so this kinda has to work...

According to this (Part 3): http://www.novell.com/coolsolutions/appnote/19386.html it is possible to trick LVM into starting anyway by creating a new pv with identical metadata to the failed disk, which will make the volume accessible, but of course leave giant holes in the file system. I have'n tried it yet, because it involves repairing (writing to) the file system which eliminates the possibility of trying other things if it fails.

Now my question is: How does this setup actually allocate disk space for new data? Is it allocated linearly from beginning to end of PVs, in the order they were added to the vg? Is it striped somehow in order to increase performance/balance load?

since this defective disk was added only later to an existing lvm2 vg and lv, containing a half-empty ext4, what are the chances that there was never any data written to the defective disk?

In other words: what are the chances of recovering all my data, even without the defective disk, by just starting the volume group as-is? Am I about to go spend $1500 on having 250GB of empty space recovered when I send the defective disk in for repair?

Is there a way to check without mounting the file system and opening the files, hoping they contain something other than zeros? (comparing addresses of used data blocks inside ext4 to address ranges that were on the missing pv, something like that, preferably easy to automate)

I know bitwise-copying the entire lv into an image file before trying to repair the ext4 would probably be a good idea, but since this lv is very large and I just suffered major file system failure on several systems it is probably a luxury I don't have...

Any suggestions?

  • 2
    On the contrary, I think not creating a dd image of all disks involved before continuing is a luxury you can't afford. – Sven Nov 16 '12 at 2:34

To wrap this up: I decided to give it a try and reassembled the lvm with the broken disk missing, after I made backups of all disks with ddrescue.

Outcome: out of 68,000 files there were roughly 12,000 missing completely and another 38,000 had bad checksums after recovery (pure luck I had the checksums on record).

So I guess it's safe to say this is not the best strategy to go with if the files are important.

Also, repairing the file system took close to 3 days of processing time and well in excess of 32GB of RAM on an Ivy-bridge Xeon.

I sent in the drive for repair after all.

| improve this answer | |
  • Thank you for sharing this experience! This can be very important knowledge for others in similar situations. – Sander Steffann Dec 10 '12 at 12:22

Way too many questions, I will answer the LVM part:

As you were able to add single disk, I guess it's not striped volume nor mirror (or you would not have the problem with missing data), just plain (linear) target.

There is a metadata backup file in /etc/lvm/backup/$VG on the machine where the LV was used - look there - you should find all the details there.

Also before you do anything with the FS: create a snapshot and use that for repairing - so you can return to previous state.

Re: FS usage: I can not help, but as you said, no deletes, just appending files, I expect the FS does not fragment the free space. You can try consulting wikipedia: https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout or search for #ext4 IRC channel.

And before doing any non reversible changes go to #lvm IRC channel on freenode where you may find online guidance.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.