Due to clvmd stalling the kernel (https://serverfault.com/questions/226466/lvm-running-vgscan-causes-clvmd-to-block) I decided to hard reboot the host because the soft reboot stalled. But when it booted up it fails to run e2fsck on one of the logical volumes and then fails to boot. The following error message I maually entered because I can't copy and paste this since from the serial console.
Checking filessytems /dev/mapper/vg_gothamdb1-lv_root: clean, 59708/5120000 files, 712853/20480000 blocks /dev/sda1: clean, 39/128016 files, 51688/512000 blocks fsck.ext4: No such files or directory while trying to open /dev/mapper/vg_gothamdb1-lv_home /dev/mapper/vg_gothamdb1-lvhome: The superblock could not be read or does not describe a correct ext2 filesystem. if the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with and alternate superblock:
Note that I have 2 volume groups in this setup. One of them is part of the cluster. The other contains logical volumes for home, root and swap. In other words the layout looks like this:
vg1: lv_root lv_home lv_swap vg2 (part of a cluster): lv_stuff
The weird thing is that lv_home stuffed up even though it's not in the volume groups thats in the cluster.
How can I fix this? lv_home should exist as a logical volume. I just want to recover lv_home.