rebuilding raid5 array

Question

I inherited a machine that was running Debian with a RAID 5 array. I installed a bunch of updates (1700 or so) that the OS recommended, then after rebooting, the raid array did not mount. The device /dev/md0 now does not exist, and I do not know why.

The /etc/mdadm/mdadm.conf contains:

DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=138b0c65:20644731:39e394c4:192c7227

I tried to do mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1 . This makes a device md0, but it is listed as "degraded," and the last drive in the list is for some reason considered "spare." I strongly suspect, though I can not be sure, that it was sdb, sdc, and sdd that were involved in the RAID-5 array

I tried all 6 possible orderings of the devices, but the last one would always come up spare. I also tried --spare-devices=0 --force, which successfully got all three drives into the array with a "clean" status, but I was unable to actually mount the device md0. When I run "file -s" on /dev/md0, I get GLS_BINARY_LSB_FIRST, which seems unhelpful.

I have no reason to believe any of the devices are faulty; all of this seems to stem from the recent upgrading. How can I resurrect the old RAID 5 array? Have my --create machinations somehow messed it up? Note that I have never successfully mounted md0.

Please advise. I know this is always the story, but I am in big trouble if I can't resurrect this thing, so anyone who helps has my eternal gratitude, for whatever it's worth.

Answer 1

I suspect you may be testing out your restore procedures in the near future.

Running --create on an existing array is... well, "misguided" is about as nice a spin as I can put on it. That's intended only to create a new array -- which you specifically don't want to do.

What you want is --assemble, or even better, work out why the system decided to stop automatically assembling the array on boot. The fact that you've been creating arrays all over the place has probably hosed any chance you had of the array ever working again, though.

By the by, the reason why your newly-created RAID5 array starts in degraded mode is given in the mdadm manpage (which would have been a good first thing to read):

When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because building the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be overridden with the --force option.

As far as diagnosing why your RAID array wasn't assembling after the upgrade, a glance at dmesg probably would have put you right; unfortunately, that data is probably either (a) gone, or (b) at least, thoroughly irrelevant now.

An mdadm ninja might be able to put things back to normal, if they had access to the machine in question and a lot of time and patience, but in your case, I'd recommend just taking the downtime hit and restoring from backup.

Answer 2

Your create machinations have likely messed it up.

--create initializes a new array.

You wanted --assemble.

Did --create give you any warnings which you overrode?

Please paste the output of mdadm -E on each partition. It may be recoverable.