Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

OS: Debian Squeeze Kernel: 2.6.32-5-amd64 mdadm version: 3.1.4 (31 August 2010)

mdarrays

md1 = raid1 (this is working)                          (devices: /dev/sd[fg]1)
md0 = raid6 (previously raid5, now failing to start)   (devices: /dev/sd[abcde]1)

Description: I had a 4 disk raid 5 that I decided to convert to a 6 disk raid 6. Added two more disks to the array and started reshape Early in the process I lost one disk (smart errors). Disk was removed and reshape continued I had three powerfailures while reshaping, but after each poweron the reshape continued. Reshape finished but one disk was "removed" e.g. did not report back it's version to the OS (appeared as unknown model).

At this stage the raid6 had two missing disk and the array was operational (I am waiting for the replacement disk still so I am missing one disk to complete the array).

I decided to power down the machine and to power up again to see if the lost disk appeared again. It did but the array failed to assemble with the following error:

madm --assemble /dev/md0 
mdadm: superblock on /dev/sdd1 doesn't match others - assembly aborted

I also tried with mdadm --assemble --force /dev/md0 but with the exact same error.

Now I tried this:

mdadm --verbose --assemble /dev/md0
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/md/1: Device or resource busy
mdadm: /dev/md/1 has wrong uuid.
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: /dev/sdg1 has wrong uuid.
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: /dev/sdg has wrong uuid.
mdadm: cannot open device /dev/sdf2: Device or resource busy
mdadm: /dev/sdf2 has wrong uuid.
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: /dev/sdf1 has wrong uuid.
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: /dev/sdf has wrong uuid.
mdadm: no RAID superblock on /dev/sde2
mdadm: /dev/sde2 has wrong uuid.
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sde has wrong uuid.
mdadm: cannot open device /dev/sdd2: Device or resource busy
mdadm: /dev/sdd2 has wrong uuid.
mdadm: superblock on /dev/sdd1 doesn't match others - assembly aborted

Also the output of mdadm -Esvvv is:

mdadm -Esvvv
mdadm: No md superblock detected on /dev/md/1.
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b7433980:f3a086eb:7f8f8a15:093b5909
           Name : cell01:1  (local to host cell01)
  Creation Time : Sun Apr  8 15:30:47 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1681342464 (801.73 GiB 860.85 GB)
     Array Size : 1681342192 (801.73 GiB 860.85 GB)
  Used Dev Size : 1681342192 (801.73 GiB 860.85 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 74d30246:840a1947:b4eeecc1:0dd95b2a

    Update Time : Fri Aug 24 00:06:16 2012
       Checksum : b302004f - correct
         Events : 244


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sdg.
/dev/sdf2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b7433980:f3a086eb:7f8f8a15:093b5909
           Name : cell01:1  (local to host cell01)
  Creation Time : Sun Apr  8 15:30:47 2012
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 1681342464 (801.73 GiB 860.85 GB)
     Array Size : 1681342192 (801.73 GiB 860.85 GB)
  Used Dev Size : 1681342192 (801.73 GiB 860.85 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5847c4e4:319a9c6e:353f67c8:765e4b47

    Update Time : Fri Aug 24 00:06:16 2012
       Checksum : 9cb0d9ca - correct
         Events : 244


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sdf1.
mdadm: No md superblock detected on /dev/sdf.
mdadm: No md superblock detected on /dev/sde2.
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5343f9b9:ccb86e50:33ed2a4b:d494c7dc
           Name : cell01:0  (local to host cell01)
  Creation Time : Sat Jul  2 11:22:04 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3904927744 (1862.01 GiB 1999.32 GB)
     Array Size : 15619706880 (7448.06 GiB 7997.29 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d13f32aa:7da122d9:77ba06f0:bc57d063

    Update Time : Thu Aug 23 22:15:58 2012
       Checksum : 6212fa6e - correct
         Events : 134404

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AA.A.A ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sde.
mdadm: No md superblock detected on /dev/sdd2.
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 5343f9b9:ccb86e50:33ed2a4b:d494c7dc
           Name : cell01:0  (local to host cell01)
  Creation Time : Sat Jul  2 11:22:04 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3904927744 (1862.01 GiB 1999.32 GB)
     Array Size : 15619706880 (7448.06 GiB 7997.29 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 3d05a6b3:3a05d5bc:fc1d01a6:10c055eb

  Reshape pos'n : 7732852736 (7374.62 GiB 7918.44 GB)
  Delta Devices : 1 (5->6)
     New Layout : left-symmetric

    Update Time : Thu Aug 23 09:48:17 2012
       Checksum : 83a30996 - correct
         Events : 134174

         Layout : left-symmetric-6
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA.A ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sdd.
mdadm: No md superblock detected on /dev/sda2.
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5343f9b9:ccb86e50:33ed2a4b:d494c7dc
           Name : cell01:0  (local to host cell01)
  Creation Time : Sat Jul  2 11:22:04 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3904927744 (1862.01 GiB 1999.32 GB)
     Array Size : 15619706880 (7448.06 GiB 7997.29 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c66da6b9:aa010cbc:8e4ab376:7a623f21

    Update Time : Thu Aug 23 22:15:58 2012
       Checksum : 988b9d9b - correct
         Events : 134404

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.A.A ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sda.
mdadm: No md superblock detected on /dev/sdc2.
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5343f9b9:ccb86e50:33ed2a4b:d494c7dc
           Name : cell01:0  (local to host cell01)
  Creation Time : Sat Jul  2 11:22:04 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3904927744 (1862.01 GiB 1999.32 GB)
     Array Size : 15619706880 (7448.06 GiB 7997.29 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e7afc574:abca367c:ebb58fe5:c4961cb7

    Update Time : Thu Aug 23 22:15:58 2012
       Checksum : 188fd371 - correct
         Events : 134404

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.A.A ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sdc.
mdadm: No md superblock detected on /dev/sdb2.
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5343f9b9:ccb86e50:33ed2a4b:d494c7dc
           Name : cell01:0  (local to host cell01)
  Creation Time : Sat Jul  2 11:22:04 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 3904927744 (1862.01 GiB 1999.32 GB)
     Array Size : 15619706880 (7448.06 GiB 7997.29 GB)
  Used Dev Size : 3904926720 (1862.01 GiB 1999.32 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : fb3a88de:834f2bc0:452db777:48ef3b21

    Update Time : Thu Aug 23 22:15:58 2012
       Checksum : c28d282b - correct
         Events : 134404

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.A.A ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sdb.

The only difference I can see is that the feature map is set to 0x4 for the mismatching disk and the others are set to 0x0. The UUID for the array is the same on all disks.

I am pretty sure the data is still there and I am tempted to try to zero the superblock for /dev/sdd1 and re-add it to the array as I assume the array should work with two disks missing. However since I does not have a backup (yes, I know it is stupid) and the data is kind of important to me (the reason I went for raid6) I would like to try the absolute safest way of recovering the array.

I was also considering upgrading to Debian wheezy since it has a newer mdadm but I would prefer to stay on Debian stable until wheezy is released.

As you can imagine this is quite scary and I would like to not loose my data so I would appreciate if someone with knowledge and experience can guide me to what to do next.

share|improve this question

1 Answer 1

The solution was surprisingly easy

mdadm --assemble /dev/md0 /dev/sd1 /dev/sdb1 /dev/sdc1 /dev/sde1

then

mdadm --add /dev/md0 /dev/sdd1

The raid is now recovering!

share|improve this answer
1  
Remember to back up. RAID is not a substitute for backup, as you've learned the hard way. –  Michael Hampton Aug 24 '12 at 19:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.