Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

The simple question: how does initramfs know how to assemble mdadm RAID arrays at startup?

My problem: I boot my server and get:

Gave up waiting for root device.
ALERT! /dev/disk/by-uuid/[UUID] does not exist. Dropping to a shell!

This happens because /dev/md0 (which is /boot, RAID 1) and /dev/md1 (which is /, RAID 5) are not being assembled correctly. What I get is /dev/md0 isn't assembled at all. /dev/md1 is assembled, but instead of using /dev/sda2, /dev/sdb2, /dev/sdc2, and /dev/sdd2, it uses /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd.

To fix this and boot my server I do:

$(initramfs) mdadm --stop /dev/md1
$(initramfs) mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
$(initramfs) mdadm --assemble /dev/md1 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
$(initramfs) exit

And it boots properly and everything works. Now I just need the RAID arrays to assemble properly at boot so I don't have to manually assemble them. I've checked /etc/mdadm/mdadm.conf and the UUIDs of the two arrays listed in that file match the UUIDs from $ mdadm --detail /dev/md[0,1].

Other details: Ubuntu 10.10, GRUB2, mdadm 2.6.7.1

UPDATE: I have a feeling it has to do with superblocks. $ mdadm --examine /dev/sda outputs the same thing as $ mdadm --examine /dev/sda2. $ mdadm --examine /dev/sda1 seems to be fine because it outputs information about /dev/md0. I don't know if this is the problem or not, but it seems to fit with /dev/md1 getting assembled with /dev/sd[abcd] instead of /dev/sd[abcd]2.

I tried zeroing the superblock on /dev/sd[abcd]. This removed the superblock from /dev/sd[abcd]2 as well and prevented me from being able to assemble /dev/md1 at all. I had to $ mdadm --create to get it back. This also put the super blocks back to the way they were.

share|improve this question

7 Answers 7

Well looking at the scripts used to assemble the initramfs, I'm thinking the problem is probably just that your /etc/mdadm/mdadm.conf is out of date.

When your system's up with the arrays assemble, execute the following command to update your mdadm config. You may want to double check it just in case as well.

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Once done, update your initramfs with :

update-initramfs

If this consistently fails, then your superblocks ( the metadata used to assemble the arrays ) may be shot. You may want to examine each of your drives and their partitions to verify. Worse case, zero out out superblocks via mdadm and recreate.

share|improve this answer
    
Note that >> will append to the existing mdadm.conf file which may not be desired (use > to overwrite) –  Peter Gibson Oct 18 '11 at 3:44
    
This turned out to be the reason for failure in my case (I think) due to duplicate entries in mdadm.conf. –  Peter Gibson Oct 18 '11 at 4:10

It sounds like your initramfs was created when your RAID setup was wrong (or just different to now) and hasn't been updated since.

You could run update-initramfs (which is normally run after kernel updates) and hopefully this will rebuild your initramfs file, including building in the right raid configuration files.

share|improve this answer
    
I've tried that a couple times, it doesn't help. It still loads /dev/sd[abcd] into /dev/md1 instead of /dev/sd[abcd]2. –  Brad Dec 5 '10 at 22:16

Here's a workaround I came up with:

Add this script to /etc/initramfs-tools/scripts/local-top:

 #!/bin/sh
 sleep 6
 mdadm --stop /dev/md1
 mdadm --stop /dev/md0
 sleep 6
 mdadm --assemble --scan

This fixes the RAID arrays before the system tries to mount md1 to /root. I had to add the pauses in in order to get the commands to work consistently.

This doesn't actually fix the problem, but it's the best solution I've found that doesn't require changing the RAID arrays or upgrading software.

share|improve this answer
    
+1 Thanks for posting back a possible solution for others who might be in the same position at some point. –  David Spillett Jan 15 '11 at 13:16

I have the same problem, and found this link that explains why it happens: https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/599515 seems that your sda2 partition goes all the way to the end of the disk and overwrites the disk superblock, so that sda and sda2 are the same thing to mdadm and it ends up assembling md1 with sda instead of sda2

share|improve this answer

To answer the question: yes, it does have to do with superblocks. The technical documentation is here: https://raid.wiki.kernel.org/index.php/RAID_superblock_formats

share|improve this answer
    
I recreated the array: $ mdadm --create -n 4 -l 5 -e 1.1 /dev/md1 /dev/sd[abcd]2 and it seemed to work and mdadm --examine looked better. However, I can no longer mount /dev/md1 or /dev/md/1. –  Brad Dec 5 '10 at 23:46

Are /dev/sd[abcd]2 set as type "fd" (RAID auto-detect) in the partition table? Run fdisk -l | less to see the partition tables. It sounds like the initrd is not detecting the partitions, but then on the raw device it is seeing the superblock. Or it could be that there's an incorrect mdadm.conf on the initrd, but I would expect that update-initramfs would fix that.

You can extract the initrd by creating a directory, cd into it and then run:

gunzip </path/to/initrd | cpio -ivd

Then you can see all the files that make up the initrd and any scripts that it's running. Investigating these may help track down what exactly is causing it.

But first check the partition tables...

share|improve this answer
    
All of the partitions are marked as fd - Linux raid autodetect. –  Brad Dec 6 '10 at 0:05
    
Thanks for the tip on how to look at the initrd file. It looked correct, but it helped me figure out what initramfs is doing. –  Brad Dec 6 '10 at 6:55
    
@Brad: Any luck fixing it? I was really hoping the partition type would do it, otherwise I think it's going to take some digging. –  Sean Reifschneider Dec 6 '10 at 7:34
    
I was able to come up with a workaround to get it to boot, but I wasn't able to actually fix the issue. –  Brad Dec 13 '10 at 4:22

Similar problem with RAID + LVM on a Debian Lenny box. Before exiting the initramfs shell do :

$(initramfs) vgchange MyLvmVol -a y
$(initramfs) exit

then

update-initramfs -u
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.