-1

We made a stupid upgrade on a running server using wrong repositories and the system became totally unbootable.

The system, a SLES 11 we used a openSuse repository to upgrade, and everything went horribly wrong. It boots now only in (repair filesystem).

On boot it fails to mount the RAID1 for the main LVMs. At this point we are only interested in accessing the data to move it to a functional server.

There are 2 HDs of 2TB, a boot (root) partition of 4G, a swap and the main partition. Boot partition (/dev/md1) is there.

On booting it fails to assemble the big raid (/dev/md3) and fails to create the Volume Group and the LVMs.

After booting in repair filesystem, we've tried this steps to rebuild the raid:

mdadm --assemble /dev/md3 /dev/sda3 /dev/sdb3 
(repair filesystem) # cat /prot c/mdstat
Personalities : [raid1] [raid0] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sda3[0] sdb3[1]
  1947090880 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
  4194240 blocks [2/2] [UU]

unused devices: <none> 

Then I went into /etc/lvm/backup/vg00 where backup of the volume group was stored and found out the UUID of the physical volume as it was unaccessible.

pvcreate --uuid 12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2 /dev/md3
vgcfgrestore vg00
vgchange -a y vg00

After these steps the volume group is created and the LVMs are there... the output of the commands ...

(repair filesystem) # pvdisplay
 --- Physical volume ---
 PV Name               /dev/md3
 VG Name               vg00
 PV Size               1.81 TiB / not usable 4.00 MiB
 Allocatable           yes
 PE Size               4.00 MiB
 Total PE              475395
 Free PE               192259
 Allocated PE          283136
 PV UUID               12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2

(repair filesystem) # vgdisplay
  --- Volume group ---
  VG Name               vg00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  9
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.81 TiB
  PE Size               4.00 MiB
  Total PE              475395
  Alloc PE / Size       283136 / 1.08 TiB
  Free  PE / Size       192259 / 751.01 GiB
  VG UUID               e51mlr-zA1U-n0Of-k3zE-Q5PP-aULU-7rTXhC


(repair filesystem) # lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg00/usr
  VG Name                vg00
  LV UUID                SxlDZT-KYf9-q4jS-i5kz-FzRl-Xttk-ilJLuP
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                302.00 GiB
  Current LE             77312
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg00/var
  VG Name                vg00
  LV UUID                lTHXSr-wUea-gqLI-n2KX-OBEE-fGRt-JLYWbk
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                200.00 GiB
  Current LE             51200
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/vg00/home
  VG Name                vg00
  LV UUID                853Lhz-J6DX-DTgc-zleK-RHIb-XDOA-tHguo9
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                600.00 GiB
  Current LE             153600
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:2

  --- Logical volume ---
  LV Name                /dev/vg00/srv
  VG Name                vg00
  LV UUID                7KKWlv-ADsx-WeUB-i8Vm-VJhL-w0nX-5MhmP2
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:3


(repair filesystem) # cat /etc/fstab
/dev/md1        /               ext3    acl,user_xattr       1 1
/dev/sda2       none            swap    sw
/dev/sdb2       none            swap    sw
/dev/vg00/usr   /usr            xfs     defaults             1 2
/dev/vg00/var   /var            xfs     defaults             1 2
/dev/vg00/home  /home           xfs     defaults             1 2
proc            /proc                proc       defaults              0 0
sysfs           /sys                 sysfs      noauto                0 0
debugfs         /sys/kernel/debug    debugfs    noauto                0 0
usbfs           /proc/bus/usb        usbfs      noauto                0 0
devpts          /dev/pts             devpts     mode=0620,gid=5       0 0

So after that I had my hopes high on the possibility of recovering the data and trying to move it to a new box. But when I try to mount the volumes ... (I'm using the srv LVM because it's empty or nonimportant)

mkdir /mnt/srvb
mount vg00-srv  /mnt/srvb

mount: you must specify the filesystem type

(repair filesystem) # fsck /dev/vg00/srv
fsck from util-linux-ng 2.16
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext2: Superblock invalid, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/dm-3

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

e2fsck -b 32768 /dev/mapper/vg00-srv

I keep trying ....

(repair filesystem) # mke2fs -n -S /dev/mapper/vg00-srv
mke2fs 1.41.14 (22-Dec-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1 blocks, Stripe width=1 blocks
262144 inodes, 1048576 blocks
52428 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736

And ...

(repair filesystem) # e2fsck -b 32768 /dev/mapper/vg00-srv
e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Bad magic number in super-block while trying to open /dev/mapper/vg00-srv

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

No luck with any of them. Also I'm not sure why but I'm afraid there's an incosistency with the filesystem types, I very much doubt they are xfs they should be ext-4 or ext-3 (Like the primary partition that works.)

I can't be sure because these were created automatically on the server data center. The new servers use ext4 but this one was older and with an old OS so my guess should be ext3.

Trying the above commands as fsck.ext3 still give the same results.

Can any one guide me a little on where to go from here? I insist I just really need to access the data in the /home LVM to try to copy it to a new machine.

Thanks a lot. I hope I was clear and wish anyone can help me.

---- EDIT -----

rescue:~# lsblk --fs
NAME                   FSTYPE            LABEL    MOUNTPOINT
sda
|-sda1                 linux_raid_member
| `-md1                ext3              root
|-sda2                 swap
`-sda3                 linux_raid_member rescue:3
  `-md3                LVM2_member
    |-vg00-usr (dm-0)
    |-vg00-var (dm-1)
    |-vg00-home (dm-2)
    `-vg00-srv (dm-3)
sdb
|-sdb1                 linux_raid_member
| `-md1                ext3              root
|-sdb2                 swap
`-sdb3                 linux_raid_member rescue:3
  `-md3                LVM2_member
    |-vg00-usr (dm-0)
    |-vg00-var (dm-1)
    |-vg00-home (dm-2)
    `-vg00-srv (dm-3)

The command /blkid /dev/vg00/srv gives no result

rescue:~# blkid
/dev/sda1: UUID="a15ef723-f84f-7aaa-1f51-fb8978ee93fe" TYPE="linux_raid_member"
/dev/sda2: UUID="804e745e-8bc4-47bc-bf2e-5e95c620d9ca" TYPE="swap"
/dev/sda3: UUID="3b31972d-e311-8292-4fc6-2add1afd58fe" UUID_SUB="f6d18087-8acd-3229-523d-a0a9960c1717" LABEL="rescue:3" TYPE="linux_raid_member"
/dev/sdb1: UUID="a15ef723-f84f-7aaa-1f51-fb8978ee93fe" TYPE="linux_raid_member"
/dev/sdb2: UUID="143565ee-04ac-4b20-93c2-4c81e4eb738e" TYPE="swap"
/dev/sdb3: UUID="3b31972d-e311-8292-4fc6-2add1afd58fe" UUID_SUB="1c8aa8bc-4a43-17c5-4b94-f56190083bdb" LABEL="rescue:3" TYPE="linux_raid_member"
/dev/md1: LABEL="root" UUID="635b7b96-6f32-420d-8431-074303eeee11" SEC_TYPE="ext2" TYPE="ext3"
/dev/md3: UUID="12pHfn-ibCI-pS8a-YOcc-LVNy-UMyp-lg9tG2" TYPE="LVM2_member"


rescue:~# mdadm --detail /dev/md3
/dev/md3:
        Version : 1.2
  Creation Time : Wed Mar  4 01:03:28 2015
     Raid Level : raid1
     Array Size : 1947090880 (1856.89 GiB 1993.82 GB)
  Used Dev Size : 1947090880 (1856.89 GiB 1993.82 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sat Mar 14 19:58:45 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : rescue:3  (local to host rescue)
           UUID : 3b31972d:e3118292:4fc62add:1afd58fe
         Events : 450

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3

I can't boot from a Live CD but I can boot from a Debian 64 rescue which seems to be more stable, as the normal boot is now also messed up with bad libraries from the wrong kernel update.

Still I can't find a way to mount the LVs in either boot.

Also This comes out when booting the rescue:

[    5.693921] md: Waiting for all devices to be available before autodetect
[    5.707605] md: If you don't use raid, use raid=noautodetect
[    5.719376] md: Autodetecting RAID arrays.
[    5.775853] md: invalid raid superblock magic on sdb3
[    5.786069] md: sdb3 does not have a valid v0.90 superblock, not importing!
[    5.821768] md: invalid raid superblock magic on sda3
[    5.831986] md: sda3 does not have a valid v0.90 superblock, not importing!
[    5.846010] md: Scanned 4 and added 2 devices.
[    5.855010] md: autorun ...
[    5.860707] md: considering sda1 ...
[    5.867974] md:  adding sda1 ...
[    5.874524] md:  adding sdb1 ...
[    5.881491] md: created md1
[    5.887204] md: bind<sdb1>
[    5.892738] md: bind<sda1>
[    5.898262] md: running: <sda1><sdb1>

I can mount /dev/md1 without problems, but this was the original boot /root partition without VGs.

  • Did vg00/srv contain some partitions? Have you tried to boot with a Fedora/Suse LiveCD? Can you post the output of "lsblk --fs" and "blkid /dev/vg00/srv" ? – shodanshok Mar 14 '15 at 19:56
  • Hi shodanshok. I have updated the post with the info you suggested. Hope you can give me more clues. – Marius Jensen Mar 15 '15 at 0:25
  • Try to check the filesystem using another superblock, for example issuing the following command: e2fsck -b 229376 /dev/mapper/vg00-srv. If even that fail, maybe the failed upgrade modified something at filesystem level and the Debian rescue can understand the new on-disk format. Try with a Fedora/Ubuntu livecd, as they have much newer kernel and userspace tools. – shodanshok Mar 15 '15 at 12:34
  • Hi. I can't boot with a livecd the box is in a Data Center where I have no access. We hired a data-recovery company that has dded the disks and is recovering files in a bulk mode. My hopes were that I could try to rebuild the LVMs once they had the disk images. I've tried everything and can't. Now my problem seems to be that filesystem type LVM2_member is unusable. I could try to change it to ext3 with testdisk... I don't dare really. – Marius Jensen Mar 15 '15 at 13:27
0

For md3 it reports "Creation Time : Wed Mar 4 01:03:28 2015", that seems rather recent to be the true creation date. Perhaps a recreate was done?

In any case where this is RAID1 you should be able to run testdisk against one of the partitions and let it search for a filesystem. Try... testdisk /dev/sda3

Note in data recovery situations it is a mistake to execute write operations on the original drives unless you are very sure of the results.

  • Hi. Yes, apparently the first action was a mdadm --create and the raid started syncing on the first panic stage a couple of admins went in and tried to solve the problem, apparently that was done the morning after. That was probably a mistake, but reading about mdadm it says that --create overwrites the metadata but never the data in the drives. Doing a testdisk /dev/sda3 does not find a partition table, it autoselects NONE and a list of partitions does not come up. – Marius Jensen Mar 17 '15 at 15:57
  • My guess is that the upgrade you did installed a newer version of mdadm. So the --create was done with this newer version. Not good. – S.Haran Mar 17 '15 at 16:32
  • The --create done with the newer version installed an mdadm v1.2 superblock where initially it used v0.90. This is not good, but recovery may still possible I think as yes just the meta data is affected. It's true testdisk will not find a partition table on sda3 but you can still have testdisk do a search for a filesystem. Let it try to find a backup ext superblock. If that is not successful there are other things to try. – S.Haran Mar 17 '15 at 16:54
  • Thanks so much for the help S Haran. I will try to do a search for a filesystem later. We've had good news from a data-recovery company just now, and apparently we'll be getting our data after all. I'll keep you informed on my advances with testdisk. – Marius Jensen Mar 17 '15 at 18:18
  • Hi again. I still had not a chance to keep trying with testdisk. We'll be getting the images of both hard drives and I'll build a box with them and try a frew things. What you say about different versions makes a lot of sense, but, the raid is working, the first partition /dev/md1 is completely functional, and everything in it works just right. The discrepancy must come from different versions on LVM2. After the first boot, /dev/md3 was never mounted, that's why we tried the --create after unsuccessful --assemble. The Physical volume was missing. – Marius Jensen Mar 18 '15 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.