2

I have a raid5 array on 4 disks, sda1, sdb1, sdd1 and sde1. sdd1 was removed but is a healthy drive. While attempting to re-add it, I accidentally zeroed out the superblock for sdb1.

sdb1's data is consistent with sda1 and sde1, just missing it's superblock.

Is it possible to recover the superblock on that device and reassemble the array (degraded) without losing any data?

Below is the output of 'mdadm -E /dev/sd{a,d,e}1'.

Am I reading the output correctly in that sde1 is Device 0, sdd1 is Device 2 and sda1 is Device 3? Then that would mean that sdb1 was Device 1. Could I recreate the array with the following command without data loss:

$ sudo mdadm --create /dev/md0 --assume-clean --level=5 --raid-devices=4 /dev/sde1 /dev/sdb1 missing /dev/sda1

/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e
           Name : cowbell:0
  Creation Time : Sun Sep 25 20:24:46 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
    Data Offset : 1024 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a0abbc72:f8bb1e2b:e8f81f5c:ed62a979

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 23 18:05:09 2013
       Checksum : 9f46a56 - correct
         Events : 717820

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.A ('A' == active, '.' == missing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e
           Name : cowbell:0
  Creation Time : Sun Sep 25 20:24:46 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027053 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 79f5632d:46d5c083:a1c0130c:b83b0654

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 23 17:42:02 2013
       Checksum : a78d6f5b - correct
         Events : 717392

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 81a36846:cf4f0489:219e2546:b1f5b90e
           Name : cowbell:0
  Creation Time : Sun Sep 25 20:24:46 2011
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907027053 (1863.02 GiB 2000.40 GB)
     Array Size : 5860538880 (5589.05 GiB 6001.19 GB)
  Used Dev Size : 3907025920 (1863.02 GiB 2000.40 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 6abd9160:c8ae382c:e6c73d01:37ac057b

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 23 18:05:09 2013
       Checksum : 9f5d8fa6 - correct
         Events : 717820

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.A ('A' == active, '.' == missing)
Improve this question
2
  • Before you do anything, have you made a backup?
    – Shane Madden
    Jul 24, 2013 at 17:36
  • I have a backup of the important things, yes. There are a bunch of things I'd rather not lose, but wouldn't be the end of the world if I did.
    – Justin Miller
    Jul 24, 2013 at 18:23

2 Answers 2

Reset to default
2

In addition to backups, you might also want to consider just making full dd copies of the disks before attempting any recovery.

That being said, looks like you're on the right track for recovery without data loss. You're interpreting the device numbers correctly. That command looks like what you need.

See my answer here - these things are surprisingly resilient against data loss, and the destruction of the superblock is not damaging to the ability to rebuild the array in the same geometry (see test 4 in that answer).

Improve this answer
4
  • Ok, so I executed the command above and the array started, but when I tried to mount it, I got the "wrong fs type, bad option, bad superblock on /dev/md0...". I created the array according to my output above, so I'm not sure what I did wrong. I don't recall changing any of the defaults like chunk size or anything, so what could be going wrong?
    – Justin Miller
    Jul 24, 2013 at 21:44
  • @JustinMiller Yeah, the left-symmetric layout and 512k chunk sizes are defaults, so that should match your initial build. Maybe take a look at some of the forensics work that Anton did in this answer, and compare what you're seeing on your system?
    – Shane Madden
    Jul 24, 2013 at 22:25
  • WOOOOOO! I got it mounted and all my data is there! Turns out the above command was fine, the superblocks just had the wrong data offsets. Thankfully I had the 'mdadm -E' from all but one of the drives and fortunately it only took 2 tries before I got the right one for sdb1 (it was same as one of the other drives added around the same time). I will write songs about you and Anton!!! Seriously though, the info that you both posted should, hands down, be on a blog about advanced RAID recovery. Invaluable.
    – Justin Miller
    Jul 25, 2013 at 12:44
  • @JustinMiller Awesome, great to hear it!
    – Shane Madden
    Jul 25, 2013 at 15:58
0

I would try to verify that the underlying physical disks are synced first before I continue re-creating MD device with assumtion it is clean. You can do that with 

mdadm -E /dev/sd[abc]1 | grep Event
Events : 0.53120
Events : 0.53108
Events : 0.53120

The output shows that sda and sdc are synced while sdc is late.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .