Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My situation is different from this one.

I have a CentOS system with 3 hard drives, and the following software RAID arrays:

/boot on RAID 1 over 2 disks
/ on RAID 5 over 3 disks
swap on RAID 0 over 2 disks (I believe)

My 3rd drive failed. At the beginning, no big deal, the array was still working. But after 1 day, when I got ready to swap the bad disk, the system cannot boot anymore with the new disk in:

md: md2: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md2
raid5: failed to run raid set md2
[...]
Kernel panic

It stops there. I have no shell. I've tried to but on the Rescue disk, but I don't know how to go from there: my arrays are not seen, so I cannot rebuild them. Exact same issue if I boot with 2 disks, or with the bad disk as my 3rd drive.

How can I fix the array now that I have a new drive?

share|improve this question

2 Answers 2

up vote 7 down vote accepted

Somehow you've managed to stop the array in a dirty state (which means that the RAID system can't be sure that the parity on all the disks is OK). This can happen if the machine was abruptly powered off, or some other write hole-inducing event.

I suspect that reassembling the array by hand, from a rescue CD, using the --force option might work, like so:

mdadm --assemble --force /dev/md2 /dev/sda2 /dev/sdb2 missing

(replacing /dev/sd... with the existing devices that make up your RAID-5 array). Assuming that that works and /proc/mdstat shows the array assembled (in a degraded state), then you can add the new partition, like so:

mdadm /dev/md2 --add /dev/sdc2

If the initial forced assemble doesn't do the trick, then you're deeply up the creek. A couple of minutes with Google has found http://www.linuxforums.org/forum/servers/77867-eeek-cant-assemble-degraded-dirty-raid6-array.html which seems to deal with a similar problem, so it might be worth trying what is described as working in there (echo "clean" > /sys/block/md0/md/array_state) but that's a slightly uglier way of doing things.

Regardless of how you manage to get the RAID set back together, the fact that it is dirty and degraded means that the contents really can't be trusted any more. The filesystem could have metadata corruption (which a fsck should fix), or the contents of one or more files could be corrupted (which you won't know without verifying the contents of all files on the partition).

share|improve this answer
1  
+1 was about to post a similar solution. I'll just add that if you have backups, which if you're asking this question probably isn't the case, I would consider restoring from the backup. B/c there is a chance that forcing the raid together can still result in data corruption. Especially when the array is already in a known degraded state it lacks the parity data to verify itself. –  3dinfluence Jun 28 '09 at 4:25
    
Ah, thanks, forgot to mention that. Answer updated. –  womble Jun 28 '09 at 4:30
    
The first command gives me this: mdadm: no devices found for /dev/md2 –  Julien Jun 28 '09 at 4:41
    
You need to specify the correct component devices for your array. –  womble Jun 28 '09 at 7:31
    
Julien if you're unsure what partition belongs to what array you can run this command on all your partitions "mdadm -E /dev/sda1". Replacing the sda1 portion with the relevant devices in your system. This will print out the raid metadata from the partition you pass to it if there is any. Doing this will show you which partition belongs too what array. Which you can then use with the command that womble suggested. –  3dinfluence Jun 28 '09 at 13:49

The System Rescue CD has the mdadm tools, so if you know how to use them it should be useful to you.

share|improve this answer
1  
If the OP knew how to use mdadm, this question wouldn't have been asked. –  womble Jun 28 '09 at 4:07
1  
There are plenty of people who are quite capable of setting up complex systems and environments from a gui, either the installer or post-install, but still get lost with the command line. Generally people coming across from windows. I've come across it often enough that it's no longer safe to assume they know the command line tools. –  pgs Jun 28 '09 at 4:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.