So I have a RAID5 spanning across 5 disk. Last week, we had 1 drive completely went bad, plus another drive that mdadm took offline due to bad sectors.

I got a new drive and mdadm is rebuilding it. However, I notice that later, YET ANOTHER drive (3rd drive) went offline with a bad sector. So /proc/mdstat looks like [UU_U_]. The one that was rebuilding shows up as SPARE. And the one with bad sectors shows up as FAIL.

Now, when I boot, mdadm can't assemble the drive.

1.) how do I know that the SPARE has finish rebuilding or not?

2.) If so, how can I get mdadm to add it back to array?

Is there a howto somewhere on how to do this from a live-CD? I am suspicious that the copy of linux running off of the degraded RAID5 maybe corrupted...

Update: At this point, I check the HD that was supposedly bad. I think what happened was that TLER kicked in & the RAID mark it as faulty. From SMART, it shows that HD have 200~300 reallocated sectors. Not ideal, but at ~2 years old WD, I hope it's not the end of life. Most likely 99% of the drives are still intact, and I would like to recovery it.

My biggest hurdle is that mdadm marks those drives as "spare", even after I did sudo mdadm --add /dev/md0 /dev/sde2. How do I get them to be part of the array again?


Two thoughts:

  1. If you had a 5-disk RAID5 array and two disks failed before you replaced one, you would have lost all of your data. Therefore, you should plan on building a new array and restoring your data from backup. If you need to be able to withstand two disk failures, you need to use RAID 6.

  2. If three out of five original disks have gone bad within a short timeframe, your original array may have been built from a batch of defective drives -- or, if these drives are old, they may simply be reaching the end of their useful lives. Either way, if I were in your situation, I would not trust any of the remaining disks from the original array.

  • Good advise. In a perfect world, I would swapped with new drives. However, at this point, I have to make do with what I have available. – fseto Sep 7 '10 at 5:40
  • 3 failures within 1 week on a population of 5 drives would appear to give you an MTBF of less than 2 weeks. Keep us posted! – Skyhawk Sep 8 '10 at 5:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.