Recover BTRFS? bad magic on superblock on /dev/md2

Question

so after a recent power outage my DS718+ RAID crashed.

After investigation with Syno Support we also identified that my RAM was broken (already removed) and as it was not a officially supported RAM module they now will no longer help me with recovering my RAID (thx for that) because they suspect the broken file system came from that and not from the power outage.

I do have a backup but it's quiet old so I would like to go recover if possible. If not I'm fine with restoring.

What i can tell you so far:

btrfs-show-super /dev/md2

superblock: bytenr=65536, device=/dev/md2
---------------------------------------------------------
ERROR: bad magic on superblock on /dev/md2 at 65536

fdisk -l

Disk /dev/ram0: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram1: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram2: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram3: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram4: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram5: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram6: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram7: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram8: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram9: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram10: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram11: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram12: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram13: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram14: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/ram15: 640 MiB, 671088640 bytes, 1310720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WD40EFRX-68N32N0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: B0741029-0239-4B70-BDC9-115FFE436B29

Device       Start        End    Sectors  Size Type
/dev/sda1     2048    4982527    4980480  2.4G Linux RAID
/dev/sda2  4982528    9176831    4194304    2G Linux RAID
/dev/sda5  9453280 7813830239 7804376960  3.6T Linux RAID


Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WD4003FFBX-68MU3N0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 7D01494F-CF7B-4DFF-8735-C9F326A16775

Device       Start        End    Sectors  Size Type
/dev/sdb1     2048    4982527    4980480  2.4G Linux RAID
/dev/sdb2  4982528    9176831    4194304    2G Linux RAID
/dev/sdb5  9453280 7813830239 7804376960  3.6T Linux RAID


Disk /dev/md0: 2.4 GiB, 2549940224 bytes, 4980352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/zram0: 275 MiB, 288358400 bytes, 70400 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/zram1: 275 MiB, 288358400 bytes, 70400 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/zram2: 275 MiB, 288358400 bytes, 70400 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/zram3: 275 MiB, 288358400 bytes, 70400 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md1: 2 GiB, 2147418112 bytes, 4194176 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/synoboot: 120 MiB, 125829120 bytes, 245760 sectors
Disk model: DiskStation
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0B6F377D-3CFE-49FF-825F-7662245E3112

Device         Start    End Sectors Size Type
/dev/synoboot1  2048  67583   65536  32M EFI System
/dev/synoboot2 67584 239615  172032  84M Linux filesystem


Disk /dev/md2: 3.6 TiB, 3995839758336 bytes, 7804374528 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/vg1000-lv: 3.6 TiB, 3995837923328 bytes, 7804370944 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/cachedev_0: 3.6 TiB, 3995837923328 bytes, 7804370944 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

mdadm --examine /dev/sda5

/dev/sda5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 4df9f767:00c49417:945e810a:3c7ae5cf
           Name : DiskStation:2
  Creation Time : Thu Aug 27 16:55:24 2015
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 7804374912 (3721.42 GiB 3995.84 GB)
     Array Size : 3902187264 (3721.42 GiB 3995.84 GB)
  Used Dev Size : 7804374528 (3721.42 GiB 3995.84 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=384 sectors
          State : clean
    Device UUID : eeb9349e:2df293df:e3e4d149:341a8cd9

    Update Time : Tue Oct 19 21:43:44 2021
       Checksum : 1776634c - correct
         Events : 236201


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

mdadm --examine /dev/sdb5

/dev/sdb5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 4df9f767:00c49417:945e810a:3c7ae5cf
           Name : DiskStation:2
  Creation Time : Thu Aug 27 16:55:24 2015
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 7804374912 (3721.42 GiB 3995.84 GB)
     Array Size : 3902187264 (3721.42 GiB 3995.84 GB)
  Used Dev Size : 7804374528 (3721.42 GiB 3995.84 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
   Unused Space : before=1968 sectors, after=384 sectors
          State : clean
    Device UUID : b0de0510:3424cbdd:380068fb:af4fa72d

    Update Time : Tue Oct 19 21:43:44 2021
       Checksum : 8d300ae5 - correct
         Events : 236201


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Commands dumpe2fs and fsck cannot be found btw - not sure why.

When trying mke2fs -n /dev/md2 it says:

mke2fs 1.44.1 (24-Mar-2018)
/dev/md2 contains a LVM2_member file system
Proceed anyway? (y,N) y
/dev/md2 is apparently in use by the system; will not make a filesystem here!

I'm without my data/NAS for almost two weeks now so any help is greatly appreciated. As I'm not a Linux guy I'd appreciate straight forward advice. I do understand that sda5 and sdb5 are my two disks, md2 is the RAID. Not sure what LVM2_member is though or how I'm able to repair it now - if that's possible at all.

Edit:

mdadm --assemble --scan -v

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/dm-1
mdadm: no recogniseable superblock on /dev/dm-0
mdadm: no recogniseable superblock on /dev/md2
mdadm: no recogniseable superblock on /dev/synoboot2
mdadm: Cannot assemble mbr metadata on /dev/synoboot1
mdadm: Cannot assemble mbr metadata on /dev/synoboot
mdadm: no recogniseable superblock on /dev/md1
mdadm: no recogniseable superblock on /dev/zram3
mdadm: no recogniseable superblock on /dev/zram2
mdadm: no recogniseable superblock on /dev/zram1
mdadm: no recogniseable superblock on /dev/zram0
mdadm: no recogniseable superblock on /dev/md0
mdadm: /dev/sdb5 is busy - skipping
mdadm: /dev/sdb2 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sda5 is busy - skipping
mdadm: /dev/sda2 is busy - skipping
mdadm: /dev/sda1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: no recogniseable superblock on /dev/ram15
mdadm: no recogniseable superblock on /dev/ram14
mdadm: no recogniseable superblock on /dev/ram13
mdadm: no recogniseable superblock on /dev/ram12
mdadm: no recogniseable superblock on /dev/ram11
mdadm: no recogniseable superblock on /dev/ram10
mdadm: no recogniseable superblock on /dev/ram9
mdadm: no recogniseable superblock on /dev/ram8
mdadm: no recogniseable superblock on /dev/ram7
mdadm: no recogniseable superblock on /dev/ram6
mdadm: no recogniseable superblock on /dev/ram5
mdadm: no recogniseable superblock on /dev/ram4
mdadm: no recogniseable superblock on /dev/ram3
mdadm: no recogniseable superblock on /dev/ram2
mdadm: no recogniseable superblock on /dev/ram1
mdadm: no recogniseable superblock on /dev/ram0
mdadm: No arrays found in config file or automatically

cat /etc/fstab

none /proc proc defaults 0 0
/dev/root / ext4 defaults 1 1
/dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,ssd,synoacl,relatime,ro,nodev 0 0

cat /proc/mdstat

Personalities : [raid1]
md2 : active raid1 sdb5[3] sda5[2]
      3902187264 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      2097088 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      2490176 blocks [2/2] [UU]

unused devices: <none>

Edit2:

pvdisplay

  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               vg1000
  PV Size               3.63 TiB / not usable 1.75 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              952682
  Free PE               0
  Allocated PE          952682
  PV UUID               Yj7BOC-Ni0B-Q1NG-CHKy-Fmq0-620T-aHeUYR

vgdisplay

  --- Volume group ---
  VG Name               vg1000
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.63 TiB
  PE Size               4.00 MiB
  Total PE              952682
  Alloc PE / Size       952682 / 3.63 TiB
  Free  PE / Size       0 / 0
  VG UUID               grXROS-RIVn-C0Nx-cKqF-lOyB-YFHJ-HDWngh

lvdisplay

  --- Logical volume ---
  LV Path                /dev/vg1000/lv
  LV Name                lv
  VG Name                vg1000
  LV UUID                xgX5UJ-vk3r-eGX0-3bxj-339u-B3sV-y1jldv
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 1
  LV Size                3.63 TiB
  Current LE             952682
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     384
  Block device           252:0

Support told me in the beginning:

The system is installed in RAID md0, a RAID level 1, which is expanded (in fail-safety, not increased) by disks added later. On md1 the swap is housed, also in RAID 1. From md2 upwards the volumes are placed. The RAID configuration of the NAS is not faulty, but the file system on volume1 is. The file system can no longer be mounted. Not even read-only

Screenshot

Edit3: btrfs-show-super /dev/mapper/cachedev_0

superblock: bytenr=65536, device=/dev/mapper/cachedev_0
---------------------------------------------------------
csum                    0x63aa7385 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    1e556138-5480-44ce-8c65-bfa2d7fd54cb
label                   2018.09.26-18:00:59 v23739
generation              3652464
root                    547766255616
sys_array_size          258
chunk_root_generation   3579752
root_level              1
chunk_root              3611627094016
chunk_root_level        1
log_root                547767943168
log_root_transid        0
log_root_level          0
log tree reserve bg     0
total_bytes             3995837923328
bytes_used              2851638943744
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             1
compat_flags            0x8000000000000000
compat_ro_flags         0x3
                        ( FREE_SPACE_TREE |
                          FREE_SPACE_TREE_VALID )
incompat_flags          0x16b
                        ( MIXED_BACKREF |
                          DEFAULT_SUBVOL |
                          COMPRESS_LZO |
                          BIG_METADATA |
                          EXTENDED_IREF |
                          SKINNY_METADATA )
csum_type               0
csum_size               4
cache_generation        18446744073709551615
uuid_tree_generation    3652464
dev_item.uuid           521f7cdc-497c-4c69-886e-c3974042337a
dev_item.fsid           1e556138-5480-44ce-8c65-bfa2d7fd54cb [match]
dev_item.type           0
dev_item.total_bytes    3995837923328
dev_item.bytes_used     3139713368064
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

Edit4:

e2fsck -nvf -C 0 /dev/md2

e2fsck 1.44.1 (24-Mar-2018)
Warning!  /dev/md2 is in use.
ext2fs_open2: Bad magic number in super-block
e2fsck: Superblock invalid, trying backup blocks...

openfs: invalid superblock checksum.
e2fsck: Bad magic number in super-block while trying to open /dev/md2

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

/dev/md2 contains a LVM2_member file system

e2fsck -b 8193 /dev/md2

e2fsck 1.44.1 (24-Mar-2018)
/dev/md2 is in use.
e2fsck: Cannot continue, aborting.

How do I safely unmount, try the command and remount again?

Thanks a lot!

Answer 1

Have you tried just assembling the array? I have a system that always drops its array at reboot. I've tried to make an mdadm.conf with no luck, so I just reassemble it at the next reboot using the create command. As long as the devices are in the same order, it'll assemble just fine. In my case, I use raid5:

    mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd[a1,b1,c1,d1]

Answer 2

Your RAID is fine. The metadata in MD RAID superblocks is consistent. So I agree with Synology support in that the RAID is intact. You may still want to check its data, but I suggest you to wait until data is recovered.

So don't do anything else with/dev/md2 and don't search anything on it, because system already found and recognized what's there.

The system sees it is in use and prevented an access to it because it knows it is the physical volume (LVM PV) of the active volume group (VG) called vg1000. In this volume group, you have a logical volume (LV), which is called simply lv. It is the block device, which is accessible as /dev/vg1000/lv (which is a symlink to some /dev/dm-X, where X is dynamic and can change after reboot or addition or removal of another device mapper entities).

The next comes cache layer. I don't know which caching technology it uses, but the cached volume has the name inside /dev/mapper, this uses a Linux Device Mapper framework. One such technology is LVM Cache. The LV /dev/vg1000/lv is a backing device. There could be also a caching device. Even if you don't have any SSDs installed presently, this layering might be created to facilitate further introduction of actual cache. You may put some SSD inside, tick some check box in the GUI and voila, you have a cache instantly without reformatting and time consuming data move.

Your file system appears to be sitting in this cached volume, which is seen in the system as /dev/mapper/cachedev_0 (which is, again, a symlink to some block device /dev/dm-Y, where Y is dynamic). This is supported by the following line in the fstab file:

/dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,ssd,synoacl,relatime,ro,nodev 0 0

This device is where you should search for BTRFS file system. Try btrfs-show-super /dev/mapper/cachedev_0, it should find a superblock.

The screenshot you showed mentions problems on the /dev/dm-1. If that is the target of /dev/mapper/cachedev_0 (check with readlink /dev/mapper/cachedev_0), you're out of luck. You may try to give this system to BTRFS specialist in the hope he can extract some information, but be prepared to pay a bill for this.