On a CentOS 6.7 system, I started getting SMART errors on a disk that was part of a volume group under LVM. So I used the recommended procedure:
- add a new disk to the system, create a new physical volume on it and added it to the relevant volume group
- used pvmove to shift all extents off the damaged disk (/dev/sdc) and onto the newly added disk - check that no extents are left on the damaged device
- used vgreduce to remove the damaged disk from the volume group, and confirmed that it had been removed from that volume group using 'pvs'
- used pvremove on /dev/sdc to remove the physical volume label
- physically removed the damaged disk from the machine and rebooted
After reboot the system seems to restart OK, but stops with a 'File-based locking initialization failed', with options to a) drop to a root prompt or b) continue .The latter just reboots the machine.
Can anyone advise me on how to troubleshoot this?
EDIT: Here's the last few lines I now see on the startup sequence:
Setting up hostname ice: [ OK ] Setting up Logical Volume Management: 3 logical volume(s) in volume group "vg_ice1" now active [ OK ] Checking filesystems /dev/mapper/vg_ice1_lv_root: clean, 1005974/3276800 files, 11295675/13107200 blocks /dev/sda1: clean, 54/128016 files, 132466/512000 blocks /dev/mapper/vg_ice1-lv_home contains a file system with errors, check forced /dev/mapper/vg_ice1-lv_home: Inode 38404397 has an invalid extent node (blk 153677788, lblk 0) /dev/mapper/vg_ice1-lv_home: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) [ FAILED ] *** An error occurred during the file system check *** Dropping you to a shell; the system will reboot *** when you leave the shell. Give root password for maintenance (or type Control-D to continue):
I tried using 'vgchange -ay', and it simply reported that 3 logical volumes were now active, but the fundamental problem persists.
I tried dropping to the root prompt and running fsck, but it seemed to be reporting many faults, so I cancelled all changes and exited. There may be a possibility that what I am seeing is some corrupted files that were copied from the failing disk, but on the other hand I don't want to risk trashing the files that were moved over to the new disk successfully.
Can anyone help? Thanks!