I had a 3 disk mdadm raid5 volume (made of /dev/sd[abc]) that was in the middle of a reshape, going to 5 disks (adding /dev/sd[df]), when the power to the server failed. this is on ubuntu 8.10 intrepid 64bit server (kernel 2.6.27), mdadm version 2.6.4, built from the source. The ups monitor daemon was nice enough to execute the following commands at that time
mdadm --stop /dev/md0 shutdown -P now
when i got the box back up and running, i used the following command to bring the raid back up, since it wouldnt come up by itself.
mdadm --assemble /dev/md0 /dev/sd[abcdf]
it came back up and restarted along its reshape process, but with only 4 discs. /dev/sdf had no superblock, so it wouldnt come along for the ride. i tried adding it back in:
mdadm --add /dev/md0 /dev/sdf
but that just added it as a hot spare, not as the missing 5th member of the raid. so i removed it with --remove, since a hot spare at this point in the process is kinda useless. to make matters worse, this added a superblock to the disk that identifies it as a hotspare now. easy enough to get rid of with --zero-superblock though.
what i want to know is, how do i massage this disk to get it back into the array in its proper place? i have no reason to believe that the data is bad, just the superblock. or is it not even appropriate at this point to try to re-add it now that the reshape has progressed without it?
contents of /proc/mdstat:
root@FS1:~# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sda sdd sdc sdb 1953524736 blocks super 0.91 level 5, 512k chunk, algorithm 2 [5/4] [UUU_U] [======>..............] reshape = 33.4% (326807040/976762368) finish=1175.3min speed=9216K/sec unused devices: <none>
contents of /etc/mdadm/mdadm.conf
root@FS1:~# cat /etc/mdadm/mdadm.conf # mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default, scan all partitions (/proc/partitions) for MD superblocks. # alternatively, specify devices to scan, using wildcards if desired. DEVICE partitions # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays ARRAY /dev/md0 level=raid5 num-devices=3 UUID=4128dcfb:b5e905ae:60b633ef:af4ede72 # This file was auto-generated on Wed, 15 Jul 2009 09:09:57 -0400 # by mkconf $Id$
output of mdadm --examine --scan
root@FS1:~# mdadm --examine --scan ARRAY /dev/md0 level=raid5 num-devices=5 UUID=4128dcfb:b5e905ae:60b633ef:af4ede72 spares=1
output of mdadm -D
root@FS1:~# mdadm -D /dev/md0 /dev/md0: Version : 00.91.03 Creation Time : Sun Jul 12 15:57:27 2009 Raid Level : raid5 Array Size : 1953524736 (1863.03 GiB 2000.41 GB) Used Dev Size : 976762368 (931.51 GiB 1000.20 GB) Raid Devices : 5 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Thu Jul 16 23:34:21 2009 State : clean, degraded, recovering Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Reshape Status : 33% complete Delta Devices : 2, (3->5) UUID : 4128dcfb:b5e905ae:60b633ef:af4ede72 (local to host FS1) Events : 0.216044 Number Major Minor RaidDevice State 0 8 0 0 active sync /dev/sda 1 8 16 1 active sync /dev/sdb 2 8 32 2 active sync /dev/sdc 3 0 0 3 removed 4 8 48 4 active sync /dev/sdd