I have a Linux system set up with 3 software RAID1 devices, each of which comprises two identical partitions on two identical disks. Recently, one of the non-root partitions on one disk began experiencing DMA errors; I therefore marked it as failed. When I rebooted the machine, it launched the kernel successfully but began printing DMA errors (presumably associated with the failed partition) almost immediately. Shouldn't marking the problematic partition as failed permit the machine to boot without any errors? If not, how can I get the system to boot? I tried modifying the mdadm.conf file in the machine's boot image to not list the problematic partition in the device list of the RAID device that comprised it, but that didn't seem to have any effect. I should also note that I can access the degraded RAID device if I boot from a rescue CD and manually assemble the device from the remaining good partition.
-
1@mailq This isn't a dupe of that.– MDMarraOct 4, 2011 at 20:12
1 Answer
It sounds like you are using Linux Software RAID, and you've got the RAID devices set up using partitions instead of whole disks.
In this case simply failing the partition won't help you: The failing drive (hardware component) is what's throwing the errors. Any time the operating system tries to access that hardware component you'll have problems, and since drives don't typically go bad in just one spot the problems will spread across all partitions until the drive finally gives up and dies.
My suggestion to you is to back up your data NOW, using the rescue CD (which apparently works per your question), and then replace the failing hardware component, rebuilding your RAID array(s) as appropriate.
Long-Term you will want to structure your software RAID the same way you would a hardware RAID (using whole drives, not partitions) - If necessary create the RAID across the physical drives you have, then partition the virtual (RAID) device. This allows you to fail a dying drive (hardware component), or if necessary to remove it and allow the system to boot without it with a known and well defined set of side effects, rather than being surprised as a drive's ever-escalating level of failure causes more and more partition-RAIDs to go wonky...
If you are trusting software RAID in production you should also be running smartd from the smartmontools suite and have it configured to alert you when drives start to look flaky...
-
Right - I am using software RAID. Out of curiosity, is there some way to reconfigure the OS such that it could boot off of the degraded array if I physically remove the faulty drive?– lebedovOct 4, 2011 at 20:28
-
@lebedov, isn't it setup that way already? If you had RAID1, the everything should be identical, aside from maybe the bootloader. Oct 4, 2011 at 20:36
-
A lot of that depends on your server's BIOS or EFI configuration, not the OS. The question @mailq pointed you at (serverfault.com/questions/196445/…) may have some pointers on the Linux-specific stuff you need to do.– voretaq7Oct 4, 2011 at 20:36
-
@Zoredache good point - If the bootloader is only on the failed drive you're SOL. Again why you should mirror the whole disk, including the boot sector. I know that's possible on FreeBSD (GEOM_MIRROR metadata goes at the end of the drive), and I think it's possible on Linux too...– voretaq7Oct 4, 2011 at 20:37
-
The problem is that if I remove the bad disk (sda), the undamaged mirror partitions on the good disk (sdb) are seen as sda* by the OS; the system refuses to boot when it can't find the sdb* partitions.– lebedovOct 4, 2011 at 20:39