Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a RAID 5 array built from partitions on 5 drives - /dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1, and /dev/sde1. When I assemble the array manually, it works fine.

Unfortunately, there seem to also be superblocks on /dev/sdb and /dev/sdc (probably something I messed up early on), and when the array assembles automatically at startup, it builds with /dev/sdb instead of /dev/sdb1 (the same with /dev/sdc), and fails to mount (with an invalid fs error).

$ sudo mdadm --assemble --scan --force (presumably what happens on startup)
mdadm: WARNING /dev/sdb1 and /dev/sdb appear to have very similar superblocks.
      If they are really different, please --zero the superblock on one
      If they are the same or overlap, please remove one from the
      DEVICE list in mdadm.conf.

$ sudo mdadm --detail /dev/md7
/dev/md7:
        Version : 00.90
  Creation Time : Wed Apr  6 18:17:07 2011
     Raid Level : raid5
     Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 7
    Persistence : Superblock is persistent

    Update Time : Sat Apr 30 13:55:50 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 224009a1:e4173bf3:2490f10a:1455ce9c (local to host ravenscar)
         Events : 0.2

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       48        3      active sync   /dev/sdd1
       4       8       65        4      active sync   /dev/sde1

$ sudo mdadm --examine /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 224009a1:e4173bf3:2490f10a:1455ce9c (local to host ravenscar)
  Creation Time : Wed Apr  6 18:17:07 2011
     Raid Level : raid5
  Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
     Array Size : 7814047744 (7452.06 GiB 8001.58 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 7

    Update Time : Wed Apr 27 21:15:03 2011
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : f83035d3 - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb
   2     2       8       33        2      active sync   /dev/sdc
   3     3       8       49        3      active sync   /dev/sdd1
   4     4       8       65        4      active sync   /dev/sde1

Then I do:

$ sudo mdadm --stop /dev/md7
$ sudo mdadm --assemble /dev/md7 /dev/sd[abcde]1

And then it works.

My question: Is it safe to use the --zero-superblock option to remove those erroneous superblocks? what other measures should I take? Did I make the right choice building the array with partitions instead of drives (/dev/sda1 instead of /dev/sda, and so on).

share|improve this question
    
What does /mdadm --examine /dev/sdb and /mdadm --examine /dev/sdb1 show? –  sciurus Apr 30 '11 at 17:36
add comment

4 Answers 4

up vote 1 down vote accepted

Do not zero the super block. Edit as mdadm suggested /etc/mdadm.conf or /etc/mdadm/mdadm.conf and change DEVICE .... to:

DEVICE /dev/sda1
DEVICE /dev/sdb1
DEVICE /dev/sdc1
DEVICE /dev/sdd1
DEVICE /dev/sde1
share|improve this answer
add comment

What seems to happen is MD sees the same superblock from two devices (sdb1 is a subset of sdb). Killing the superblock would kill the real superblock so don't do that.

MD v0.90 superblocks are stored at the end of the device on a 64k-aligned block. In your case sdb1 is the only partition and the superblock can be found at the end of both sdb and sdb1. Somehow (this appears to be an an old issue, see link at the end) MD got confused and started to think you have two copies of the superblock on two separate devices.

The fix is easy if you have some extra space on that partition - you can simply wipe the superbblock then repartition the disk ending the partition 64k before the end of the disk; MD won't see a suberblock on sdb and assemble the array properly using sdb1. If you cannot take out 64k from sdb, you'll have to rebuild the array. If this isn't your boot disk, you can also use a newer superblock version. v.1.1 and v.1.2 both store the superblock somewhere on the beginning of the array, which will solve this specific problem.

I've just experienced the exact same problem on a raid1. Using "fdisk -u" (use sectors as units) the partition was ending on sector 143374743. I subtracted 128 from this number (128 512-byte sectors = 64k) and used the resulting number as the end of the partition. I could then verify that MD wouldn't see the superblock on sda with:

# mdadm --misc -E /dev/sda

It seems the recent behavioral change of using sectors and dropping DOS compatibility in partitioning tools led to having the last partition always end at the very end of the disk (rather than on a cylinder boundary). Some other parameters may come into play as - for instance - I have a running system which does show the duplicate superblocks when disks are queried directly, and still mounts its array properly.

For more info on MD superblock formats, see:

https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#The_version-0.90_Superblock_Format

The Debian FAQ (point 11.) mention this issue:

http://git.debian.org/?p=pkg-mdadm/mdadm.git;a=blob_plain;f=debian/FAQ;hb=HEAD

share|improve this answer
add comment

If you don't have any data on the array I'd just wipe it and start over. If you do have data on it I'd back it up before playing around w/ the superblocks.

Personally, I'd get the data off of it and dd a few sectors of /dev/zero over the beginning of each disk, then repartition and rebuild the array. I'd feel more comfortable knowing that everything was setup soundly to begin with.

As an aside: That's a pretty big array to be doing RAID-5 on, to my mind. Keep an eye on the rebuild time. In the event of a single disk failure you'll have catastrophic data loss if you have another error while reading during rebuild. You may want to consider RAID-6.

share|improve this answer
    
wiping isn't really an option, unfortunately. And yeah, I've heard that about raid 5. we should be fine though, right? –  colinmarc Apr 30 '11 at 18:29
    
Your circumstance scares me. It may work fine for you, but I'd be wary. Starting a new 7TB volume off with a checkered past sounds like a formula for a bad future, even if it seems like an innocuous problem. re: a 7TB RAID-5 array - You'll be fine if you end up being fine. If the "luck" of an unrecoverable read error happens during a rebuild, someday in the future, you won't be fine. It's like winning the lottery, except in reverse. Using RAID-5 means you're buying more tickets in the "lose all my data in a rebuild lottery" than with RAID-6 or RAID-10. –  Evan Anderson Apr 30 '11 at 18:34
    
The superblock can be at the beginning or at the end of the partition, depending on the mdadm version. –  Mircea Vutcovici Jun 7 '11 at 3:46
add comment

Unluckily I deleted the RAID-Superblock when reading following massage:

mdadm: WARNING /dev/sdb1 and /dev/sdb appear to have very similar superblocks.
      If they are really different, please --zero the superblock on one
      If they are the same or overlap, please remove one from the
      DEVICE list in mdadm.conf.

Afterwards my RAID1 couldn't start any more.

Luckily, creating new RAID1 with identical parameters as original done, did not delete old one: http://lists.debian.org/debian-user-french/2006/03/msg00607.html via http://kevin.deldycke.com/2007/03/how-to-recover-a-raid-array-after-having-zero-ized-superblocks/

As partly thumb/misleading/unhelpful above mdadm message is, as smart is mdadm with recreation.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.