So I have a RAID5 spanning across 5 disk. Last week, we had 1 drive completely went bad, plus another drive that mdadm took offline due to bad sectors.
I got a new drive and mdadm is rebuilding it. However, I notice that later, YET ANOTHER drive (3rd drive) went offline with a bad sector. So /proc/mdstat looks like [UU_U_]. The one that was rebuilding shows up as SPARE. And the one with bad sectors shows up as FAIL.
Now, when I boot, mdadm can't assemble the drive.
1.) how do I know that the SPARE has finish rebuilding or not?
2.) If so, how can I get mdadm to add it back to array?
Is there a howto somewhere on how to do this from a live-CD? I am suspicious that the copy of linux running off of the degraded RAID5 maybe corrupted...
Update: At this point, I check the HD that was supposedly bad. I think what happened was that TLER kicked in & the RAID mark it as faulty. From SMART, it shows that HD have 200~300 reallocated sectors. Not ideal, but at ~2 years old WD, I hope it's not the end of life. Most likely 99% of the drives are still intact, and I would like to recovery it.
My biggest hurdle is that mdadm marks those drives as "spare", even after I did
sudo mdadm --add /dev/md0 /dev/sde2. How do I get them to be part of the array again?