36

We had a disk fail in a server and replaced it before removing the drive from LVM.

The server has 4 physical drives (PV's), each with it's own volume group (VG). Each VG has 2 or more logical volumes (LV's.) Now LVM is complaining about the missing drive. So we have a VG (vg04) with two LV's that have become orphans than we need to clear out of the system.

The problem is every time we run any LVM command we get these 'read failed' errors:

# lvscan
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error

# vgreduce vg04 --removemissing --force
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# vgchange -a n /dev/vg04
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# lvchange -a n /dev/vg04/vz
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found
  Skipping volume group vg04

The missing VG and LV's are not important, we just want to remove them.

As you can see we've tried all the suggestions made, so far without luck.

Output from 'lvm dumpconfig' can be checked at http://pastebin.com/MHiBzrLJ

Improve this question

3 Answers 3

Reset to default
55

The solution was to run dmsetup, in this case the two commands

dmsetup remove vg04-vz
dmsetup remove vg04-swap

Before doing this, I checked with the command 'dmsetup info' that the 'open count' for both LV's were zero.

WARNING: dmsetup can wreck serious havoc with your disks so anyone using this information in the future please make sure you read the man page.

Improve this answer
3
  • 3
    That did the trick. It's too bad LVM is such a leaky abstraction, but I guess this is as good as it gets.
    – Navin
    Mar 5, 2016 at 14:45
  • 1
    Good call with that warning. If I understand correctly, the similarly named remove_all --force --deferred --retry would blow away every LVM managed drive within reach. It would keep trying and not give up, like the Terminator. I am super curious abut why a footgun like this was warranted.
    – Dan Ross
    Mar 8, 2017 at 0:12
  • There is also a single device "terminate with extreme prejudice" (and/ or vengeance, if the device kept you busy whole night): # dmsetup --force --retry --deferred remove <YOURVG>-<YOURLV> This is especially true, if for instance a iscsi device was hanging and you removed it with # iscsiadm -m node -U all -T <FQDN>
    – AdamKalisz
    Jul 20, 2017 at 14:32
10

I experienced similar after removing a VMware disk /dev/sdb.

Use lsscsi to determine which device is being used (you may need to install the program) then run lsscsi:

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb <== the removed physical disk
[2:0:2:0]    disk    VMware   Virtual disk     1.0   /dev/sdc

Now write the delete file with by redirecting 1 into it.

# echo 1 > '/sys/class/scsi_device/2:0:1:0/device/delete'

The device is immediately gone.

Improve this answer
2
  • 1
    Thank you!! This worked perfectly for me for getting rid of errors like "/dev/sdb: read failed after 0 of 4096 at 0: Input/output error" after removing a stale vg1-xyz via dmsetup.
    – cat pants
    Jun 26, 2013 at 22:16
  • 1
    Only this one works for me. You made my day.
    – odiszapc
    Jul 13, 2014 at 9:23
2

vgchange -a n /dev/vg04

If you have logical volumes still active you may have to

lvchange -a n /dev/vg04/swap
lvchange -a n dev/vg04/vz

etc.

Improve this answer
2
  • 2
    Unfortunatlely that doesn't fix it either, it just keeps throwing the same error:
    – Dokbua
    Aug 28, 2012 at 17:00
  • On my system lvchange -a n <VGNAME> does work perfekt!
    – zzeroo
    Jan 26, 2013 at 16:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .