LVM file locking error after moving extents to new physical volume

Question

On a CentOS 6.7 system, I started getting SMART errors on a disk that was part of a volume group under LVM. So I used the recommended procedure:

• add a new disk to the system, create a new physical volume on it and added it to the relevant volume group
• used pvmove to shift all extents off the damaged disk (/dev/sdc) and onto the newly added disk - check that no extents are left on the damaged device
• used vgreduce to remove the damaged disk from the volume group, and confirmed that it had been removed from that volume group using 'pvs'
• used pvremove on /dev/sdc to remove the physical volume label
• physically removed the damaged disk from the machine and rebooted

After reboot the system seems to restart OK, but stops with a 'File-based locking initialization failed', with options to a) drop to a root prompt or b) continue .The latter just reboots the machine.

Can anyone advise me on how to troubleshoot this?

EDIT: Here's the last few lines I now see on the startup sequence:

Setting up hostname ice:  [  OK  ]
Setting up Logical Volume Management:   3 logical volume(s) in volume group "vg_ice1" now active  [  OK  ]
Checking filesystems
/dev/mapper/vg_ice1_lv_root: clean, 1005974/3276800 files, 11295675/13107200 blocks
/dev/sda1: clean, 54/128016 files, 132466/512000 blocks
/dev/mapper/vg_ice1-lv_home contains a file system with errors, check forced
/dev/mapper/vg_ice1-lv_home:
Inode 38404397 has an invalid extent node (blk 153677788, lblk 0)

/dev/mapper/vg_ice1-lv_home: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
         (i.e., without -a or -p options)
                                                    [  FAILED  ]

*** An error occurred during the file system check
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type Control-D to continue):

I tried using 'vgchange -ay', and it simply reported that 3 logical volumes were now active, but the fundamental problem persists.

I tried dropping to the root prompt and running fsck, but it seemed to be reporting many faults, so I cancelled all changes and exited. There may be a possibility that what I am seeing is some corrupted files that were copied from the failing disk, but on the other hand I don't want to risk trashing the files that were moved over to the new disk successfully.

Can anyone help? Thanks!

Answer 1

As the error says, you have a problem with the vg_ice1/lv_home filesystem. Type the root password to get into a prompt and run fsck /dev/mapper/vg_ice1-lv_home. There is good chance that the error you are seeing is not related to you moving the disks around. Or it could be that the dying disk did manage to corrupt the home filesystem before you manage to replace it.