mdadm assembles with drives instead of partitions

Question

I have a RAID 5 array built from partitions on 5 drives - /dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1, and /dev/sde1. When I assemble the array manually, it works fine.

Unfortunately, there seem to also be superblocks on /dev/sdb and /dev/sdc (probably something I messed up early on), and when the array assembles automatically at startup, it builds with /dev/sdb instead of /dev/sdb1 (the same with /dev/sdc), and fails to mount (with an invalid fs error).

$ sudo mdadm --assemble --scan --force (presumably what happens on startup)
mdadm: WARNING /dev/sdb1 and /dev/sdb appear to have very similar superblocks.
      If they are really different, please --zero the superblock on one
      If they are the same or overlap, please remove one from the
      DEVICE list in mdadm.conf.

$ sudo mdadm --detail /dev/md7
/dev/md7:
        Version : 00.90
  Creation Time : Wed Apr  6 18:17:07 2011
     Raid Level : raid5
     Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 7
    Persistence : Superblock is persistent

    Update Time : Sat Apr 30 13:55:50 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 224009a1:e4173bf3:2490f10a:1455ce9c (local to host ravenscar)
         Events : 0.2

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd1
       4       8       65        4      active sync   /dev/sde1

$ sudo mdadm --examine /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 224009a1:e4173bf3:2490f10a:1455ce9c (local to host ravenscar)
  Creation Time : Wed Apr  6 18:17:07 2011
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
     Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 7

    Update Time : Wed Apr 27 21:15:03 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : f83035d3 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb
   2     2       8       33        2      active sync   /dev/sdc
   3     3       8       49        3      active sync   /dev/sdd1
   4     4       8       65        4      active sync   /dev/sde1

Then I do:

$ sudo mdadm --stop /dev/md7
$ sudo mdadm --assemble /dev/md7 /dev/sd[abcde]1

And then it works.

My question: Is it safe to use the --zero-superblock option to remove those erroneous superblocks? what other measures should I take? Did I make the right choice building the array with partitions instead of drives (/dev/sda1 instead of /dev/sda, and so on).

Answer 1

Do not zero the super block. Edit as mdadm suggested /etc/mdadm.conf or /etc/mdadm/mdadm.conf and change DEVICE .... to:

DEVICE /dev/sda1
DEVICE /dev/sdb1
DEVICE /dev/sdc1
DEVICE /dev/sdd1
DEVICE /dev/sde1

Answer 2

What seems to happen is MD sees the same superblock from two devices (sdb1 is a subset of sdb). Killing the superblock would kill the real superblock so don't do that.

MD v0.90 superblocks are stored at the end of the device on a 64k-aligned block. In your case sdb1 is the only partition and the superblock can be found at the end of both sdb and sdb1. Somehow (this appears to be an an old issue, see link at the end) MD got confused and started to think you have two copies of the superblock on two separate devices.

The fix is easy if you have some extra space on that partition - you can simply wipe the superbblock then repartition the disk ending the partition 64k before the end of the disk; MD won't see a suberblock on sdb and assemble the array properly using sdb1. If you cannot take out 64k from sdb, you'll have to rebuild the array. If this isn't your boot disk, you can also use a newer superblock version. v.1.1 and v.1.2 both store the superblock somewhere on the beginning of the array, which will solve this specific problem.

I've just experienced the exact same problem on a raid1. Using "fdisk -u" (use sectors as units) the partition was ending on sector 143374743. I subtracted 128 from this number (128 512-byte sectors = 64k) and used the resulting number as the end of the partition. I could then verify that MD wouldn't see the superblock on sda with:

# mdadm --misc -E /dev/sda

It seems the recent behavioral change of using sectors and dropping DOS compatibility in partitioning tools led to having the last partition always end at the very end of the disk (rather than on a cylinder boundary). Some other parameters may come into play as - for instance - I have a running system which does show the duplicate superblocks when disks are queried directly, and still mounts its array properly.

For more info on MD superblock formats, see:

https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format

The Debian FAQ (point 11.) mention this issue:

http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=blob_plain;f=debian/FAQ;hb=HEAD

Answer 3

If you don't have any data on the array I'd just wipe it and start over. If you do have data on it I'd back it up before playing around w/ the superblocks.

Personally, I'd get the data off of it and dd a few sectors of /dev/zero over the beginning of each disk, then repartition and rebuild the array. I'd feel more comfortable knowing that everything was setup soundly to begin with.

As an aside: That's a pretty big array to be doing RAID-5 on, to my mind. Keep an eye on the rebuild time. In the event of a single disk failure you'll have catastrophic data loss if you have another error while reading during rebuild. You may want to consider RAID-6.