I have a Linux system set up with 3 software RAID1 devices, each of which comprises two identical partitions on two identical disks. Recently, one of the non-root partitions on one disk began experiencing DMA errors; I therefore marked it as failed. When I rebooted the machine, it launched the kernel successfully but began printing DMA errors (presumably associated with the failed partition) almost immediately. Shouldn't marking the problematic partition as failed permit the machine to boot without any errors? If not, how can I get the system to boot? I tried modifying the mdadm.conf file in the machine's boot image to not list the problematic partition in the device list of the RAID device that comprised it, but that didn't seem to have any effect. I should also note that I can access the degraded RAID device if I boot from a rescue CD and manually assemble the device from the remaining good partition.
It sounds like you are using Linux Software RAID, and you've got the RAID devices set up using partitions instead of whole disks.
In this case simply failing the partition won't help you: The failing drive (hardware component) is what's throwing the errors. Any time the operating system tries to access that hardware component you'll have problems, and since drives don't typically go bad in just one spot the problems will spread across all partitions until the drive finally gives up and dies.
My suggestion to you is to back up your data NOW, using the rescue CD (which apparently works per your question), and then replace the failing hardware component, rebuilding your RAID array(s) as appropriate.
Long-Term you will want to structure your software RAID the same way you would a hardware RAID (using whole drives, not partitions) - If necessary create the RAID across the physical drives you have, then partition the virtual (RAID) device. This allows you to fail a dying drive (hardware component), or if necessary to remove it and allow the system to boot without it with a known and well defined set of side effects, rather than being surprised as a drive's ever-escalating level of failure causes more and more partition-RAIDs to go wonky...
If you are trusting software RAID in production you should also be running
|show 5 more comments|