We made a stupid upgrade on a running server using wrong repositories and the system became totally unbootable.
The system, a SLES 11 we used a openSuse repository to upgrade, and everything went horribly wrong. It boots now only in (repair filesystem).
On boot it fails to mount the RAID1 for the main LVMs. At this point we are only interested in accessing the data to move it to a functional server.
There are 2 HDs of 2TB, a boot (root) partition of 4G, a swap and the main partition. Boot partition (/dev/md1) is there.
On booting it fails to assemble the big raid (/dev/md3) and fails to create the Volume Group and the LVMs.
After booting in repair filesystem, we've tried this steps to rebuild the raid:
mdadm --assemble /dev/md3 /dev/sda3 /dev/sdb3
(repair filesystem) # cat /prot c/mdstat
Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sda3[0] sdb3[1]
1947090880 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sda1[0] sdb1[1]
4194240 blocks [2/2] [UU]
unused devices: <none>
Then I went into /etc/lvm/backup/vg00 where backup of the volume group was stored and found out the UUID of the physical volume as it was unaccessible.
pvcreate --uuid 12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2 /dev/md3
vgcfgrestore vg00
vgchange -a y vg00
After these steps the volume group is created and the LVMs are there... the output of the commands ...
(repair filesystem) # pvdisplay
--- Physical volume ---
PV Name /dev/md3
VG Name vg00
PV Size 1.81 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 475395
Free PE 192259
Allocated PE 283136
PV UUID 12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2
(repair filesystem) # vgdisplay
--- Volume group ---
VG Name vg00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 9
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.81 TiB
PE Size 4.00 MiB
Total PE 475395
Alloc PE / Size 283136 / 1.08 TiB
Free PE / Size 192259 / 751.01 GiB
VG UUID e51mlr-zA1U-n0Of-k3zE-Q5PP-aULU-7rTXhC
(repair filesystem) # lvdisplay
--- Logical volume ---
LV Name /dev/vg00/usr
VG Name vg00
LV UUID SxlDZT-KYf9-q4jS-i5kz-FzRl-Xttk-ilJLuP
LV Write Access read/write
LV Status available
# open 0
LV Size 302.00 GiB
Current LE 77312
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 253:0
--- Logical volume ---
LV Name /dev/vg00/var
VG Name vg00
LV UUID lTHXSr-wUea-gqLI-n2KX-OBEE-fGRt-JLYWbk
LV Write Access read/write
LV Status available
# open 0
LV Size 200.00 GiB
Current LE 51200
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 253:1
--- Logical volume ---
LV Name /dev/vg00/home
VG Name vg00
LV UUID 853Lhz-J6DX-DTgc-zleK-RHIb-XDOA-tHguo9
LV Write Access read/write
LV Status available
# open 0
LV Size 600.00 GiB
Current LE 153600
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 253:2
--- Logical volume ---
LV Name /dev/vg00/srv
VG Name vg00
LV UUID 7KKWlv-ADsx-WeUB-i8Vm-VJhL-w0nX-5MhmP2
LV Write Access read/write
LV Status available
# open 0
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 253:3
(repair filesystem) # cat /etc/fstab
/dev/md1 / ext3 acl,user_xattr 1 1
/dev/sda2 none swap sw
/dev/sdb2 none swap sw
/dev/vg00/usr /usr xfs defaults 1 2
/dev/vg00/var /var xfs defaults 1 2
/dev/vg00/home /home xfs defaults 1 2
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
So after that I had my hopes high on the possibility of recovering the data and trying to move it to a new box. But when I try to mount the volumes ... (I'm using the srv LVM because it's empty or nonimportant)
mkdir /mnt/srvb
mount vg00-srv /mnt/srvb
mount: you must specify the filesystem type
(repair filesystem) # fsck /dev/vg00/srv
fsck from util-linux-ng 2.16
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/dm-3
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
e2fsck -b 32768 /dev/mapper/vg00-srv
I keep trying ....
(repair filesystem) # mke2fs -n -S /dev/mapper/vg00-srv
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=1 blocks
262144 inodes, 1048576 blocks
52428 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
And ...
(repair filesystem) # e2fsck -b 32768 /dev/mapper/vg00-srv
e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Bad magic number in super-block while trying to open /dev/mapper/vg00-srv
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
No luck with any of them. Also I'm not sure why but I'm afraid there's an incosistency with the filesystem types, I very much doubt they are xfs they should be ext-4 or ext-3 (Like the primary partition that works.)
I can't be sure because these were created automatically on the server data center. The new servers use ext4 but this one was older and with an old OS so my guess should be ext3.
Trying the above commands as fsck.ext3 still give the same results.
Can any one guide me a little on where to go from here? I insist I just really need to access the data in the /home LVM to try to copy it to a new machine.
Thanks a lot. I hope I was clear and wish anyone can help me.
---- EDIT -----
rescue:~# lsblk --fs
NAME FSTYPE LABEL MOUNTPOINT
sda
|-sda1 linux_raid_member
| `-md1 ext3 root
|-sda2 swap
`-sda3 linux_raid_member rescue:3
`-md3 LVM2_member
|-vg00-usr (dm-0)
|-vg00-var (dm-1)
|-vg00-home (dm-2)
`-vg00-srv (dm-3)
sdb
|-sdb1 linux_raid_member
| `-md1 ext3 root
|-sdb2 swap
`-sdb3 linux_raid_member rescue:3
`-md3 LVM2_member
|-vg00-usr (dm-0)
|-vg00-var (dm-1)
|-vg00-home (dm-2)
`-vg00-srv (dm-3)
The command /blkid /dev/vg00/srv gives no result
rescue:~# blkid
/dev/sda1: UUID="a15ef723-f84f-7aaa-1f51-fb8978ee93fe" TYPE="linux_raid_member"
/dev/sda2: UUID="804e745e-8bc4-47bc-bf2e-5e95c620d9ca" TYPE="swap"
/dev/sda3: UUID="3b31972d-e311-8292-4fc6-2add1afd58fe" UUID_SUB="f6d18087-8acd-3229-523d-a0a9960c1717" LABEL="rescue:3" TYPE="linux_raid_member"
/dev/sdb1: UUID="a15ef723-f84f-7aaa-1f51-fb8978ee93fe" TYPE="linux_raid_member"
/dev/sdb2: UUID="143565ee-04ac-4b20-93c2-4c81e4eb738e" TYPE="swap"
/dev/sdb3: UUID="3b31972d-e311-8292-4fc6-2add1afd58fe" UUID_SUB="1c8aa8bc-4a43-17c5-4b94-f56190083bdb" LABEL="rescue:3" TYPE="linux_raid_member"
/dev/md1: LABEL="root" UUID="635b7b96-6f32-420d-8431-074303eeee11" SEC_TYPE="ext2" TYPE="ext3"
/dev/md3: UUID="12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2" TYPE="LVM2_member"
rescue:~# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Wed Mar 4 01:03:28 2015
Raid Level : raid1
Array Size : 1947090880 (1856.89 GiB 1993.82 GB)
Used Dev Size : 1947090880 (1856.89 GiB 1993.82 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sat Mar 14 19:58:45 2015
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : rescue:3 (local to host rescue)
UUID : 3b31972d:e3118292:4fc62add:1afd58fe
Events : 450
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
I can't boot from a Live CD but I can boot from a Debian 64 rescue which seems to be more stable, as the normal boot is now also messed up with bad libraries from the wrong kernel update.
Still I can't find a way to mount the LVs in either boot.
Also This comes out when booting the rescue:
[ 5.693921] md: Waiting for all devices to be available before autodetect
[ 5.707605] md: If you don't use raid, use raid=noautodetect
[ 5.719376] md: Autodetecting RAID arrays.
[ 5.775853] md: invalid raid superblock magic on sdb3
[ 5.786069] md: sdb3 does not have a valid v0.90 superblock, not importing!
[ 5.821768] md: invalid raid superblock magic on sda3
[ 5.831986] md: sda3 does not have a valid v0.90 superblock, not importing!
[ 5.846010] md: Scanned 4 and added 2 devices.
[ 5.855010] md: autorun ...
[ 5.860707] md: considering sda1 ...
[ 5.867974] md: adding sda1 ...
[ 5.874524] md: adding sdb1 ...
[ 5.881491] md: created md1
[ 5.887204] md: bind<sdb1>
[ 5.892738] md: bind<sda1>
[ 5.898262] md: running: <sda1><sdb1>
I can mount /dev/md1 without problems, but this was the original boot /root partition without VGs.