We made a stupid upgrade on a running server using wrong repositories and the system became totally unbootable.
The system, a SLES 11 we used a openSuse repository to upgrade, and everything went horribly wrong. It boots now only in (repair filesystem).
On boot it fails to mount the RAID1 for the main LVMs. At this point we are only interested in accessing the data to move it to a functional server.
There are 2 HDs of 2TB, a boot (root) partition of 4G, a swap and the main partition. Boot partition (/dev/md1) is there.
On booting it fails to assemble the big raid (/dev/md3) and fails to create the Volume Group and the LVMs.
After booting in repair filesystem, we've tried this steps to rebuild the raid:
mdadm --assemble /dev/md3 /dev/sda3 /dev/sdb3 (repair filesystem) # cat /prot c/mdstat Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4] md3 : active raid1 sda3 sdb3 1947090880 blocks super 1.2 [2/2] [UU] md1 : active raid1 sda1 sdb1 4194240 blocks [2/2] [UU] unused devices: <none>
Then I went into /etc/lvm/backup/vg00 where backup of the volume group was stored and found out the UUID of the physical volume as it was unaccessible.
pvcreate --uuid 12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2 /dev/md3 vgcfgrestore vg00 vgchange -a y vg00
After these steps the volume group is created and the LVMs are there... the output of the commands ...
(repair filesystem) # pvdisplay --- Physical volume --- PV Name /dev/md3 VG Name vg00 PV Size 1.81 TiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 475395 Free PE 192259 Allocated PE 283136 PV UUID 12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2 (repair filesystem) # vgdisplay --- Volume group --- VG Name vg00 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 9 VG Access read/write VG Status resizable MAX LV 0 Cur LV 4 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 1.81 TiB PE Size 4.00 MiB Total PE 475395 Alloc PE / Size 283136 / 1.08 TiB Free PE / Size 192259 / 751.01 GiB VG UUID e51mlr-zA1U-n0Of-k3zE-Q5PP-aULU-7rTXhC (repair filesystem) # lvdisplay --- Logical volume --- LV Name /dev/vg00/usr VG Name vg00 LV UUID SxlDZT-KYf9-q4jS-i5kz-FzRl-Xttk-ilJLuP LV Write Access read/write LV Status available # open 0 LV Size 302.00 GiB Current LE 77312 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 253:0 --- Logical volume --- LV Name /dev/vg00/var VG Name vg00 LV UUID lTHXSr-wUea-gqLI-n2KX-OBEE-fGRt-JLYWbk LV Write Access read/write LV Status available # open 0 LV Size 200.00 GiB Current LE 51200 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 253:1 --- Logical volume --- LV Name /dev/vg00/home VG Name vg00 LV UUID 853Lhz-J6DX-DTgc-zleK-RHIb-XDOA-tHguo9 LV Write Access read/write LV Status available # open 0 LV Size 600.00 GiB Current LE 153600 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 253:2 --- Logical volume --- LV Name /dev/vg00/srv VG Name vg00 LV UUID 7KKWlv-ADsx-WeUB-i8Vm-VJhL-w0nX-5MhmP2 LV Write Access read/write LV Status available # open 0 LV Size 4.00 GiB Current LE 1024 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 1024 Block device 253:3 (repair filesystem) # cat /etc/fstab /dev/md1 / ext3 acl,user_xattr 1 1 /dev/sda2 none swap sw /dev/sdb2 none swap sw /dev/vg00/usr /usr xfs defaults 1 2 /dev/vg00/var /var xfs defaults 1 2 /dev/vg00/home /home xfs defaults 1 2 proc /proc proc defaults 0 0 sysfs /sys sysfs noauto 0 0 debugfs /sys/kernel/debug debugfs noauto 0 0 usbfs /proc/bus/usb usbfs noauto 0 0 devpts /dev/pts devpts mode=0620,gid=5 0 0
So after that I had my hopes high on the possibility of recovering the data and trying to move it to a new box. But when I try to mount the volumes ... (I'm using the srv LVM because it's empty or nonimportant)
mkdir /mnt/srvb mount vg00-srv /mnt/srvb mount: you must specify the filesystem type (repair filesystem) # fsck /dev/vg00/srv fsck from util-linux-ng 2.16 e2fsck 1.41.14 (22-Dec-2010) fsck.ext2: Superblock invalid, trying backup blocks... fsck.ext2: Bad magic number in super-block while trying to open /dev/dm-3 The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device> e2fsck -b 32768 /dev/mapper/vg00-srv
I keep trying ....
(repair filesystem) # mke2fs -n -S /dev/mapper/vg00-srv mke2fs 1.41.14 (22-Dec-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=1 blocks, Stripe width=1 blocks 262144 inodes, 1048576 blocks 52428 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=1073741824 32 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736
(repair filesystem) # e2fsck -b 32768 /dev/mapper/vg00-srv e2fsck 1.41.14 (22-Dec-2010) e2fsck: Bad magic number in super-block while trying to open /dev/mapper/vg00-srv The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device>
No luck with any of them. Also I'm not sure why but I'm afraid there's an incosistency with the filesystem types, I very much doubt they are xfs they should be ext-4 or ext-3 (Like the primary partition that works.)
I can't be sure because these were created automatically on the server data center. The new servers use ext4 but this one was older and with an old OS so my guess should be ext3.
Trying the above commands as fsck.ext3 still give the same results.
Can any one guide me a little on where to go from here? I insist I just really need to access the data in the /home LVM to try to copy it to a new machine.
Thanks a lot. I hope I was clear and wish anyone can help me.
---- EDIT -----
rescue:~# lsblk --fs NAME FSTYPE LABEL MOUNTPOINT sda |-sda1 linux_raid_member | `-md1 ext3 root |-sda2 swap `-sda3 linux_raid_member rescue:3 `-md3 LVM2_member |-vg00-usr (dm-0) |-vg00-var (dm-1) |-vg00-home (dm-2) `-vg00-srv (dm-3) sdb |-sdb1 linux_raid_member | `-md1 ext3 root |-sdb2 swap `-sdb3 linux_raid_member rescue:3 `-md3 LVM2_member |-vg00-usr (dm-0) |-vg00-var (dm-1) |-vg00-home (dm-2) `-vg00-srv (dm-3)
The command /blkid /dev/vg00/srv gives no result
rescue:~# blkid /dev/sda1: UUID="a15ef723-f84f-7aaa-1f51-fb8978ee93fe" TYPE="linux_raid_member" /dev/sda2: UUID="804e745e-8bc4-47bc-bf2e-5e95c620d9ca" TYPE="swap" /dev/sda3: UUID="3b31972d-e311-8292-4fc6-2add1afd58fe" UUID_SUB="f6d18087-8acd-3229-523d-a0a9960c1717" LABEL="rescue:3" TYPE="linux_raid_member" /dev/sdb1: UUID="a15ef723-f84f-7aaa-1f51-fb8978ee93fe" TYPE="linux_raid_member" /dev/sdb2: UUID="143565ee-04ac-4b20-93c2-4c81e4eb738e" TYPE="swap" /dev/sdb3: UUID="3b31972d-e311-8292-4fc6-2add1afd58fe" UUID_SUB="1c8aa8bc-4a43-17c5-4b94-f56190083bdb" LABEL="rescue:3" TYPE="linux_raid_member" /dev/md1: LABEL="root" UUID="635b7b96-6f32-420d-8431-074303eeee11" SEC_TYPE="ext2" TYPE="ext3" /dev/md3: UUID="12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2" TYPE="LVM2_member" rescue:~# mdadm --detail /dev/md3 /dev/md3: Version : 1.2 Creation Time : Wed Mar 4 01:03:28 2015 Raid Level : raid1 Array Size : 1947090880 (1856.89 GiB 1993.82 GB) Used Dev Size : 1947090880 (1856.89 GiB 1993.82 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Sat Mar 14 19:58:45 2015 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : rescue:3 (local to host rescue) UUID : 3b31972d:e3118292:4fc62add:1afd58fe Events : 450 Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 8 19 1 active sync /dev/sdb3
I can't boot from a Live CD but I can boot from a Debian 64 rescue which seems to be more stable, as the normal boot is now also messed up with bad libraries from the wrong kernel update.
Still I can't find a way to mount the LVs in either boot.
Also This comes out when booting the rescue:
[ 5.693921] md: Waiting for all devices to be available before autodetect [ 5.707605] md: If you don't use raid, use raid=noautodetect [ 5.719376] md: Autodetecting RAID arrays. [ 5.775853] md: invalid raid superblock magic on sdb3 [ 5.786069] md: sdb3 does not have a valid v0.90 superblock, not importing! [ 5.821768] md: invalid raid superblock magic on sda3 [ 5.831986] md: sda3 does not have a valid v0.90 superblock, not importing! [ 5.846010] md: Scanned 4 and added 2 devices. [ 5.855010] md: autorun ... [ 5.860707] md: considering sda1 ... [ 5.867974] md: adding sda1 ... [ 5.874524] md: adding sdb1 ... [ 5.881491] md: created md1 [ 5.887204] md: bind<sdb1> [ 5.892738] md: bind<sda1> [ 5.898262] md: running: <sda1><sdb1>
I can mount /dev/md1 without problems, but this was the original boot /root partition without VGs.