My situation is different from this one.

I have a CentOS system with 3 hard drives, and the following software RAID arrays:

/boot on RAID 1 over 2 disks
/ on RAID 5 over 3 disks
swap on RAID 0 over 2 disks (I believe)

My 3rd drive failed. At the beginning, no big deal, the array was still working. But after 1 day, when I got ready to swap the bad disk, the system cannot boot anymore with the new disk in:

md: md2: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md2
raid5: failed to run raid set md2
Kernel panic

It stops there. I have no shell. I've tried to but on the Rescue disk, but I don't know how to go from there: my arrays are not seen, so I cannot rebuild them. Exact same issue if I boot with 2 disks, or with the bad disk as my 3rd drive.

How can I fix the array now that I have a new drive?


Somehow you've managed to stop the array in a dirty state (which means that the RAID system can't be sure that the parity on all the disks is OK). This can happen if the machine was abruptly powered off, or some other write hole-inducing event.

I suspect that reassembling the array by hand, from a rescue CD, using the --force option might work, like so:

mdadm --assemble --force /dev/md2 /dev/sda2 /dev/sdb2 missing

(replacing /dev/sd... with the existing devices that make up your RAID-5 array). Assuming that that works and /proc/mdstat shows the array assembled (in a degraded state), then you can add the new partition, like so:

mdadm /dev/md2 --add /dev/sdc2

If the initial forced assemble doesn't do the trick, then you're deeply up the creek. A couple of minutes with Google has found http://www.linuxforums.org/forum/servers/77867-eeek-cant-assemble-degraded-dirty-raid6-array.html which seems to deal with a similar problem, so it might be worth trying what is described as working in there (echo "clean" > /sys/block/md0/md/array_state) but that's a slightly uglier way of doing things.

Regardless of how you manage to get the RAID set back together, the fact that it is dirty and degraded means that the contents really can't be trusted any more. The filesystem could have metadata corruption (which a fsck should fix), or the contents of one or more files could be corrupted (which you won't know without verifying the contents of all files on the partition).

  • 1
    +1 was about to post a similar solution. I'll just add that if you have backups, which if you're asking this question probably isn't the case, I would consider restoring from the backup. B/c there is a chance that forcing the raid together can still result in data corruption. Especially when the array is already in a known degraded state it lacks the parity data to verify itself. – 3dinfluence Jun 28 '09 at 4:25
  • Ah, thanks, forgot to mention that. Answer updated. – womble Jun 28 '09 at 4:30
  • The first command gives me this: mdadm: no devices found for /dev/md2 – Julien Jun 28 '09 at 4:41
  • You need to specify the correct component devices for your array. – womble Jun 28 '09 at 7:31
  • Julien if you're unsure what partition belongs to what array you can run this command on all your partitions "mdadm -E /dev/sda1". Replacing the sda1 portion with the relevant devices in your system. This will print out the raid metadata from the partition you pass to it if there is any. Doing this will show you which partition belongs too what array. Which you can then use with the command that womble suggested. – 3dinfluence Jun 28 '09 at 13:49

The System Rescue CD has the mdadm tools, so if you know how to use them it should be useful to you.

  • 1
    If the OP knew how to use mdadm, this question wouldn't have been asked. – womble Jun 28 '09 at 4:07
  • 1
    There are plenty of people who are quite capable of setting up complex systems and environments from a gui, either the installer or post-install, but still get lost with the command line. Generally people coming across from windows. I've come across it often enough that it's no longer safe to assume they know the command line tools. – pgs Jun 28 '09 at 4:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.