First, I launch a fresh large instance using the Elestic AMI ami-c162a9a8. Initially, I run:

df -h

/dev/xvda1            9.9G  814M  8.6G   9% /
udev                  3.7G  4.0K  3.7G   1% /dev
tmpfs                 1.5G  156K  1.5G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  3.7G     0  3.7G   0% /run/shm
/dev/xvdb             414G  199M  393G   1% /mnt

cat /etc/fstab

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
/dev/xvdb   /mnt    auto    defaults,nobootwait,comment=cloudconfig 0   2

I then run this script:

apt-get install -y mdadm --no-install-recommends

# Configure Raid - take into account xvdb or sdb
DRIVE=`df -h | grep -v grep | awk 'NR==2{print $1}'`

if [ "$DRIVE" == "/dev/xvda1" ]; then

umount /mnt
dd if=/dev/zero of=$DRIVE_1 bs=4096 count=1024
dd if=/dev/zero of=$DRIVE_2 bs=4096 count=1024
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=2 $DRIVE_1 $DRIVE_2
echo DEVICE $DRIVE_1 $DRIVE_2 | tee /etc/mdadm.conf 
mdadm --detail --scan | tee -a /etc/mdadm.conf
blockdev --setra 65536 /dev/md0
mkfs -t ext3 /dev/md0
mkdir -p /mnt/myraid && mount -t ext3 -o noatime /dev/md0 /mnt/myraid

# Remove xvdb/sdb from fstab
chmod 777 /etc/fstab
sed -i '$ d' /etc/fstab

# Make raid appear on reboot
echo "/dev/md0 /mnt/myraid ext3 noatime 0 0" | tee -a /etc/fstab

When the script finishes, I again run:

df -h

/dev/xvda1            9.9G  815M  8.6G   9% /
udev                  3.7G  8.0K  3.7G   1% /dev
tmpfs                 1.5G  164K  1.5G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  3.7G     0  3.7G   0% /run/shm
/dev/md0              827G  201M  785G   1% /mnt/myraid

cat /etc/fstab

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
/dev/md0 /mnt/myraid ext3 noatime 0 0

When I reboot the box, I get the error: port 22: Connection refused. When I tried to debug the script, I found that stopping after this command, mkdir -p /mnt/myraid && mount -t ext3 -o noatime /dev/md0 /mnt/myraid, rebooting, then trying to manually mount gives me this error: special device /dev/md0 does not exist. I am in the dark here, what is going on?


After I raided and mounted (but didn't modify /etc/fstab), I ran:

mdadm -D /dev/md0

        Version : 1.2
  Creation Time : Mon Dec 19 06:13:44 2011
     Raid Level : raid0
     Array Size : 880730112 (839.93 GiB 901.87 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Mon Dec 19 06:13:44 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 256K

           Name : ip-10-91-18-80:0  (local to host ip-10-91-18-80)
           UUID : 36946c0e:db95eb34:bf22c078:45958378
         Events : 0

    Number   Major   Minor   RaidDevice State
       0     202       16        0      active sync   /dev/xvdb
       1     202       32        1      active sync   /dev/xvdc

Then I rebooted and ran:

mdadm --assemble /dev/md0 /dev/xvdb /dev/xvdc:

mdadm: cannot open device /dev/xvdb: Device or resource busy
mdadm: /dev/xvdb has no superblock - assembly aborted

I also tried:

mdadm --assemble --scan

mdadm: No arrays found in config file or automatically

When I make it verbose:

mdadm --assemble --scan --verbose

mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/xvdb: Device or resource busy
mdadm: /dev/xvdb has wrong uuid.
mdadm: cannot open device /dev/xvdc: Device or resource busy
mdadm: /dev/xvdc has wrong uuid.

Typically the mdadm.conf file will contain the information necessary to reassemble the array on boot. Of particular importance is the UUID contained in that file - which must match that of your array (find it with mdadm -D /dev/md0).

If nothing seems wrong, one option you may pursue is to rename the mdadm.conf file and use dpkg-reconfigure mdadm to regenerate it with the correct information from your array.

This thread on the topic is worth a read.

For your mount to be successful, you typically need to:

  1. Load the md module (either with modprobe md (or raid0 instead of md) or by adding it to /etc/modules)
  2. Assemble the array e.g.:
    mdadm --assemble /dev/md0 /dev/xvdb /dev/xvdc
    (Alternatively, use --scan to assemble all, instead of specifying an array)

If all else fails, you can add the above to an init script, to have the array assemble on boot.

Since the mounting is dependent on the existence of the device (md0), you may want add nobootwait to your fstab (or, move the mounting into the script that runs the assemble command).

  • So, I added some updates under "Edits" after trying your suggestions. I've seen various blogs point config to /etc/mdadm.conf, and others point to /etc/mdadm/mdadm.conf...do you think this could be the culprit? I tried both at one point, but not with updated suggestions. – user52077 Dec 19 '11 at 6:36
  • Your script is using /etc/mdadm.conf (but the one mdadm uses depends on how you set it up) - worst case, try with a copy in both places (your --scan command suggests that it is reading a config file without any arrays listed). Since you don't seem to be able to manually assemble the array, I don't think the problem has to do with your mdadm.conf. Try adding a --verbose to the assemble command to see if you can get more information, and perhaps run it with --examine on /dev/xvdb to see if you can determine why there is no superblock. Ephemeral storage should persist between reboots. – cyberx86 Dec 19 '11 at 7:03
  • So, everything seems to go OK when raiding. I added a verbose argument after rebooting and not modifying fstab, and got this: sudo mdadm --assemble --scan --verbose mdadm: looking for devices for /dev/md0 mdadm: cannot open device /dev/xvdb: Device or resource busy mdadm: /dev/xvdb has wrong uuid. mdadm: cannot open device /dev/xvdc: Device or resource busy mdadm: /dev/xvdc has wrong uuid. – user52077 Dec 19 '11 at 23:51
  • 2
    Well, this turned out to be a fun problem - I gave it a try with your AMI. The problem is that on reboot your instance is mapping the raid array to a different block device (i.e. not md0 - in my case it was md127) - find yours with ls /dev/md*. Once you stop mdadm (mdadm --stop /dev/md127 - replace with your device) you can successfully assemble the array - which will map to md0 (should have restarted here, but didn't); Run dpkg-reconfigure mdadm to update your config and make the changes persist. Finally - /etc/mdadm/mdadm.conf is the default, the other (/etc/mdadm.conf) is alternate. – cyberx86 Dec 20 '11 at 2:13

The key insight here is that your md device is getting renamed on reboot. It was created as /dev/md0 but after a reboot it showed up as /dev/md127 (this was previously mentioned in a comment I'm bubbling it up as an answer for easier reading.)

I just ran into exactly the same thing (created md0, rebooted, showed up as md127). Rebooted again, /dev/md127 still present. I'm just going to run with it.

echo "/dev/md127 /mnt/myraid xfs noatime 0 0" | tee -a /etc/fstab

Someone with more time on his or her hands should dig into why this is happening (and if it always happens precisely this way) and report back here ;- )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy