Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am currently dealing with a multiple disk failure on a Linux LVM Volume Group that is backed up by a RAID-5 md device. One disk has been taken out completely and another one is showing a limited number of corrupt sectors, due to what seems to have been a misbehaving power supply.

The problem is that once an I/O error hits, md takes the array down, since it does not have enough devices for it to be operational. Where md the only one involved, I could mdadm --stop the array and then recreate it to get all devices active again.

Unfortunately, the array is a PV in an LVM volume group and I cannot seem to get the kernel to release it. vgchange -an does not seem to do anything, bar spew out a couple of I/O errors.

I am obviously missing something, but how in the name of -insert-favorite-deity- do I get LVM to release the underlying PV without rebooting the server?

share|improve this question

2 Answers 2

I would say that you are in a pretty bad shape. If you are simply desperate to somehow recover some of your data and don't care much about the LVM (whose metadata is probably already corrupt, judging by the IO errors from vgchange -an), I would recommend going low level. Remember, LVM is just a wrapper around the kernel device mapper, so you can use dmsetup to manipulate your logical volumes. dmsetup table will give you a list of currently active logical disks so it is a good idea to back up its output in case you cannot even access your LVM later on. Then you can try stopping the devices you want to stop with dmsetup remove or even dmsetup remove_all. But make sure they are unmounted first.

And of course, copy as much data as possible to a safe location.

share|improve this answer
    
Actually the LVM is fine - as long as md does not decide to throw out one of the disks. When that happens every single access to the system results in an I/O error. BTW dmsetup remove_all --force does not seem to be able overcome the obstacle of LVM being set in its ways... –  thkala Sep 17 '12 at 18:41

When vgchange -an does not seem to do anything (on an Ubuntu system) then consider blaming udev rules after reading this bug report (I found it Googling for "dmsetup remove lvm vgchange"): https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1088081

The workaround is as follows (thanks to Martin):

sudo umount /mnt # asume the drive in question is mounted under /mnt
sudo service udev stop
sudo lvchange -a n <LV-name>
sudo cryptsetup luksClose <LUKS-devicename>
sudo service udev start

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.