I have a server with 6 disks in it, 5 of which are in a RAID5 array. I came home from work today to find that the landlord had been through the house with a sparky, and the power had been reset. Now, my server refuses to boot up correctly.
During the boot process, I am warned that the raid array has been degraded, and asked if I want to boot it anyway. No matter if I say yes or no, I am loaded into initramfs. From there, I've discovered that 2 disks in the array are being flagged as removed, howeverI can re-add these via mdadm and it reports them as being fine and the whole array is in active sync, and it is active but not started.
I've checked, and the drive that the OS is on is fine. I can mount it and browse it, but I cant figure out how to get it to load. Even then, I cant figure out why 2 of my 5 drives have been removed from the array at the same time. It seems unlikely that I would have 2 failures in the same day, and there are no failed drive messages in dmesg showing up.
Any suggestions? I have a lot of irreplaceable stuff on there that I assumed was safe due to the raid array.
UPDATE: As mentioned below I am able to assemble the raid array using
mdadm --assemble /dev/md0
I then re-add the removed drives
mdadm /dev/md0 --re-add /dev/sda1 mdadm /dev/md0 --re-add /dev/sdb1
I can then start the array and then mount it to a folder in initramfs
mdadm -R /dev/md0 mount /mnt/raid
However as soon as I restart the machine the raid array shows up as degraded and the server boots into initramfs. I've tried navigating through the raid array and everything appears to be there, and fully accessible. I've checked and the UUID is definitely the one that is shown in my OS disk.
UPDATE 2: Ok, not really sure what happened here. I attempted to boot into the OS by typing 'exit' which took me out of initramfs. The first time, it kernel panicked, saying it couldnt mount the drive (possibly because I mounted it in initramfs?). I then rebooted the server, typed exit as soon as initramfs loaded, and everything is back. All of my files are intact. I'm running a check on the drive now. Any suggestions as to things I should do before rebooting next time to make sure this doesnt happen again?