1

I have a server with 6 disks in it, 5 of which are in a RAID5 array. I came home from work today to find that the landlord had been through the house with a sparky, and the power had been reset. Now, my server refuses to boot up correctly.

During the boot process, I am warned that the raid array has been degraded, and asked if I want to boot it anyway. No matter if I say yes or no, I am loaded into initramfs. From there, I've discovered that 2 disks in the array are being flagged as removed, howeverI can re-add these via mdadm and it reports them as being fine and the whole array is in active sync, and it is active but not started.

I've checked, and the drive that the OS is on is fine. I can mount it and browse it, but I cant figure out how to get it to load. Even then, I cant figure out why 2 of my 5 drives have been removed from the array at the same time. It seems unlikely that I would have 2 failures in the same day, and there are no failed drive messages in dmesg showing up.

Any suggestions? I have a lot of irreplaceable stuff on there that I assumed was safe due to the raid array.

UPDATE: As mentioned below I am able to assemble the raid array using

mdadm --assemble /dev/md0

I then re-add the removed drives

mdadm /dev/md0 --re-add /dev/sda1
mdadm /dev/md0 --re-add /dev/sdb1

I can then start the array and then mount it to a folder in initramfs

mdadm -R /dev/md0
mount /mnt/raid

However as soon as I restart the machine the raid array shows up as degraded and the server boots into initramfs. I've tried navigating through the raid array and everything appears to be there, and fully accessible. I've checked and the UUID is definitely the one that is shown in my OS disk.

UPDATE 2: Ok, not really sure what happened here. I attempted to boot into the OS by typing 'exit' which took me out of initramfs. The first time, it kernel panicked, saying it couldnt mount the drive (possibly because I mounted it in initramfs?). I then rebooted the server, typed exit as soon as initramfs loaded, and everything is back. All of my files are intact. I'm running a check on the drive now. Any suggestions as to things I should do before rebooting next time to make sure this doesnt happen again?

1 Answer 1

1

So if you can mount it I assume at least 4 of the 5 disks are fine, try running:

mdadm --assemble --force /dev/md0 

If md0 is your raid array. I've never used that command before but, according to man pages:

       Assemble
          Assemble the parts of a previously created array into an  active
          array.  Components  can  be  explicitly given or can be searched
          for.  mdadm checks that the  components  do  form  a  bona  fide
          array,  and can, on request, fiddle superblock information so as
          to assemble a faulty array.
4
  • Sorry, should have explained it a bit better. The drive that the OS is on, which is not part of the array, mounts fine however I cannot boot into it. The array itself assemles fine when running the command you specified, however the state becomes active, Not Started. When I save the UUID to /etc/mdadm/mdadm.conf it still fails to load the array on reboot. Jul 27, 2012 at 9:59
  • Could you please boot the server from cold and state the exact error messages you get?
    – Alex Berry
    Jul 27, 2012 at 10:53
  • Also, once you have made the disk readable and mountable you could try and update /etc/mdadm/mdadm.conf in the boot partition then run update-initramfs -u to make sure it has the current config.
    – Alex Berry
    Jul 27, 2012 at 10:58
  • Thanks. As stated in my 2nd update, I rebooted the server from cold, was put into initramfs where I exited straight away and everything seems fine. I have run update-initramfs -u to hopefully prevent this from happening again. Jul 27, 2012 at 11:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .