Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We had a disk fail in a server and replaced it before removing the drive from LVM.

The server has 4 physical drives (PV's), each with it's own volume group (VG). Each VG has 2 or more logical volumes (LV's.) Now LVM is complaining about the missing drive. So we have a VG (vg04) with two LV's that have become orphans than we need to clear out of the system.

The problem is every time we run any LVM command we get these 'read failed' errors:

# lvscan
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error

# vgreduce vg04 --removemissing --force
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# vgchange -a n /dev/vg04
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found

# lvchange -a n /dev/vg04/vz
  /dev/vg04/swap: read failed after 0 of 4096 at 4294901760: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4294959104: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/swap: read failed after 0 of 4096 at 4096: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903864832: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 995903922176: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg04/vz: read failed after 0 of 4096 at 4096: Input/output error
  Volume group "vg04" not found
  Skipping volume group vg04

The missing VG and LV's are not important, we just want to remove them.

As you can see we've tried all the suggestions made, so far without luck.

Output from 'lvm dumpconfig' can be checked at http://pastebin.com/MHiBzrLJ

share|improve this question
add comment

5 Answers

up vote 9 down vote accepted

The solution was to run dmsetup, in this case the two commands

dmsetup remove vg04-vz
dmsetup remove vg04-swap

Before doing this, I checked with the command 'dmsetup info' that the 'open count' for both LV's were zero.

WARNING: dmsetup can wreck serious havoc with your disks so anyone using this information in the future please make sure you read the man page.

share|improve this answer
add comment

I experienced similar after removing a VMware disk /dev/sdb.

Use lsscsi to determine which device is being used (you may need to install the program) then run lsscsi:

# lsscsi
[1:0:0:0]    cd/dvd  NECVMWar VMware IDE CDR10 1.00  /dev/sr0
[2:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[2:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb <== the removed physical disk
[2:0:2:0]    disk    VMware   Virtual disk     1.0   /dev/sdc

Now write the delete file with

# echo 1 > /sys/class/scsi_device/2\:0\:1\:0/device/delete

The device is immediately gone.

share|improve this answer
    
Thank you!! This worked perfectly for me for getting rid of errors like "/dev/sdb: read failed after 0 of 4096 at 0: Input/output error" after removing a stale vg1-xyz via dmsetup. –  cat pants Jun 26 '13 at 22:16
add comment

vgchange -a n /dev/vg04

If you have logical volumes still active you may have to

lvchange -a n /dev/vg04/swap
lvchange -a n dev/vg04/vz

etc.

share|improve this answer
    
Unfortunatlely that doesn't fix it either, it just keeps throwing the same error: –  Rene Aug 28 '12 at 17:00
    
On my system lvchange -a n <VGNAME> does work perfekt! –  zzeroo Jan 26 '13 at 16:30
add comment

pvremove, vgremove, lvremove - its hard to tell you exact commands since we don't see full configuration

share|improve this answer
    
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. –  Jeff Ferland Aug 27 '12 at 23:11
    
Well I have 4 phys drives/PV's, each has it's own VG, and each VG some LV's. One phy drive failed and was replace without telling LVM so now it is complaining about missing the drive. Essentially. So I have a vg04 with two LV's that have become orphans than I need to clear out of the system. –  Rene Aug 28 '12 at 5:22
    
@Rene please do "lvm dumpconfig" and posted it somewhere so I can take a look. –  alexus Aug 28 '12 at 19:05
    
@alexus - pastebin.com/MHiBzrLJ –  Rene Aug 29 '12 at 7:08
add comment

Try vgreduce --removemissing.

If one or more physical volumes in the volume group are lost, consider vgreduce --removemissing to make the volume group metadata consistent again.

man vgreduce

share|improve this answer
1  
You probably meant vgreduce. Thanks but it doesn't fix it. It just throws the same error as before. –  Rene Aug 28 '12 at 5:20
    
Corrected the wrong command. Sorry it didn't work for you. I'll leave it in case it works for someone searching in the future. –  Aaron Copley Aug 28 '12 at 13:29
    
Thanks for the edit, @Rene. I had the right man page though as that was where I pulled the quote from. Either way, I guess man vgreduce was more relevant. –  Aaron Copley Aug 28 '12 at 19:38
add comment

protected by MadHatter Jun 24 '13 at 10:44

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.