I had a 3 disk mdadm raid5 volume (made of /dev/sd[abc]) that was in the middle of a reshape, going to 5 disks (adding /dev/sd[df]), when the power to the server failed. this is on ubuntu 8.10 intrepid 64bit server (kernel 2.6.27), mdadm version 2.6.4, built from the source. The ups monitor daemon was nice enough to execute the following commands at that time

mdadm --stop /dev/md0
shutdown -P now

when i got the box back up and running, i used the following command to bring the raid back up, since it wouldnt come up by itself.

mdadm --assemble /dev/md0 /dev/sd[abcdf]

it came back up and restarted along its reshape process, but with only 4 discs. /dev/sdf had no superblock, so it wouldnt come along for the ride. i tried adding it back in:

mdadm --add /dev/md0 /dev/sdf

but that just added it as a hot spare, not as the missing 5th member of the raid. so i removed it with --remove, since a hot spare at this point in the process is kinda useless. to make matters worse, this added a superblock to the disk that identifies it as a hotspare now. easy enough to get rid of with --zero-superblock though.

what i want to know is, how do i massage this disk to get it back into the array in its proper place? i have no reason to believe that the data is bad, just the superblock. or is it not even appropriate at this point to try to re-add it now that the reshape has progressed without it?

contents of /proc/mdstat:

root@FS1:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sdd[4] sdc[2] sdb[1]
      1953524736 blocks super 0.91 level 5, 512k chunk, algorithm 2 [5/4] [UUU_U]
      [======>..............]  reshape = 33.4% (326807040/976762368) finish=1175.3min speed=9216K/sec

unused devices: <none>

contents of /etc/mdadm/mdadm.conf

root@FS1:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
# Please refer to mdadm.conf(5) for information about this file.

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=4128dcfb:b5e905ae:60b633ef:af4ede72

# This file was auto-generated on Wed, 15 Jul 2009 09:09:57 -0400
# by mkconf $Id$

output of mdadm --examine --scan

root@FS1:~# mdadm --examine --scan
ARRAY /dev/md0 level=raid5 num-devices=5 UUID=4128dcfb:b5e905ae:60b633ef:af4ede72

output of mdadm -D

root@FS1:~# mdadm -D /dev/md0
        Version : 00.91.03
  Creation Time : Sun Jul 12 15:57:27 2009
     Raid Level : raid5
     Array Size : 1953524736 (1863.03 GiB 2000.41 GB)
  Used Dev Size : 976762368 (931.51 GiB 1000.20 GB)
   Raid Devices : 5
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Jul 16 23:34:21 2009
          State : clean, degraded, recovering
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

 Reshape Status : 33% complete
  Delta Devices : 2, (3->5)

           UUID : 4128dcfb:b5e905ae:60b633ef:af4ede72 (local to host FS1)
         Events : 0.216044

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       0        0        3      removed
       4       8       48        4      active sync   /dev/sdd

First off, let the md0 device finish digesting the existing setup. Don't monkey with it at this moment, as it's busy trying to follow its one and only imperative - get the array back into a clean state!

Once things have stabilized, proceed to remove the "old" entry for the fifth drive. Then proceed to add the drive into the array and reshape it as you initially did, although this time you'll start with 4 disks instead of 3.

| improve this answer | |
  • that makes sense, but i do have one follow-up question about your answer. you say to, once the initial reshape is done, to remove the entry for the missing drive, add the drive back in, and do another reshape. why is that a better course than to add the new drive, do a resync, and then remove the failed entry? please dont think im being argumentative, i'd just like to understand the difference. – user13328 Jul 17 '09 at 13:06
  • Run back through the sequence of events you just went through at the 10,000 foot level. Notice how they play out. Now, consider what will happen when you try to stretch a 4-drive array into a 5-drive array while it's still shuffling blocks around for 4 drives. The driver would have to, in effect, do this twice, but simultaneously, and you would have a potential race condition on top of that...will it complete the initial "stretch" in time before the other one, hot on its heels, completes as well? – Avery Payne Jul 17 '09 at 14:42
  • So the short of it is, the device driver has "decided" to take this course of action, and it seems to think that everything is "normal" in this regard (that the array will be "clean"). Might as well let it finish up, rather than give it "instructions" that might be, at this point in time, confusing to "it". After it's done doing the Curly Shuffle, you can then poke the driver's brain with a sharp stick (get the 5th drive added to the array properly) and it'll stand up and run around. – Avery Payne Jul 17 '09 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy