Ubuntu EC2 Raid0 Ephemeral — SSH connection after reboot

Question

First, I launch a fresh large instance using the Elestic AMI ami-c162a9a8. Initially, I run:

df -h

/dev/xvda1            9.9G  814M  8.6G   9% /
udev                  3.7G  4.0K  3.7G   1% /dev
tmpfs                 1.5G  156K  1.5G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  3.7G     0  3.7G   0% /run/shm
/dev/xvdb             414G  199M  393G   1% /mnt

cat /etc/fstab

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
/dev/xvdb   /mnt    auto    defaults,nobootwait,comment=cloudconfig 0   2

I then run this script:

#!/bin/bash
apt-get install -y mdadm --no-install-recommends

# Configure Raid - take into account xvdb or sdb
DRIVE=`df -h | grep -v grep | awk 'NR==2{print $1}'`

if [ "$DRIVE" == "/dev/xvda1" ]; then
        DRIVE_1=/dev/xvdb
        DRIVE_2=/dev/xvdc
else
        DRIVE_1=/dev/sdb
        DRIVE_2=/dev/sdc
fi

umount /mnt
dd if=/dev/zero of=$DRIVE_1 bs=4096 count=1024
dd if=/dev/zero of=$DRIVE_2 bs=4096 count=1024
partprobe
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=2 $DRIVE_1 $DRIVE_2
echo DEVICE $DRIVE_1 $DRIVE_2 | tee /etc/mdadm.conf 
mdadm --detail --scan | tee -a /etc/mdadm.conf
blockdev --setra 65536 /dev/md0
mkfs -t ext3 /dev/md0
mkdir -p /mnt/myraid && mount -t ext3 -o noatime /dev/md0 /mnt/myraid

# Remove xvdb/sdb from fstab
chmod 777 /etc/fstab
sed -i '$ d' /etc/fstab

# Make raid appear on reboot
echo "/dev/md0 /mnt/myraid ext3 noatime 0 0" | tee -a /etc/fstab

When the script finishes, I again run:

df -h

/dev/xvda1            9.9G  815M  8.6G   9% /
udev                  3.7G  8.0K  3.7G   1% /dev
tmpfs                 1.5G  164K  1.5G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  3.7G     0  3.7G   0% /run/shm
/dev/md0              827G  201M  785G   1% /mnt/myraid

cat /etc/fstab

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
/dev/md0 /mnt/myraid ext3 noatime 0 0

When I reboot the box, I get the error: port 22: Connection refused. When I tried to debug the script, I found that stopping after this command, mkdir -p /mnt/myraid && mount -t ext3 -o noatime /dev/md0 /mnt/myraid, rebooting, then trying to manually mount gives me this error: special device /dev/md0 does not exist. I am in the dark here, what is going on?

EDIT:

After I raided and mounted (but didn't modify /etc/fstab), I ran:

mdadm -D /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Mon Dec 19 06:13:44 2011
     Raid Level : raid0
     Array Size : 880730112 (839.93 GiB 901.87 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Mon Dec 19 06:13:44 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 256K

           Name : ip-10-91-18-80:0  (local to host ip-10-91-18-80)
           UUID : 36946c0e:db95eb34:bf22c078:45958378
         Events : 0

    Number   Major   Minor   RaidDevice State
       0     202       16        0      active sync   /dev/xvdb
       1     202       32        1      active sync   /dev/xvdc

Then I rebooted and ran:

mdadm --assemble /dev/md0 /dev/xvdb /dev/xvdc:

mdadm: cannot open device /dev/xvdb: Device or resource busy
mdadm: /dev/xvdb has no superblock - assembly aborted

I also tried:

mdadm --assemble --scan

mdadm: No arrays found in config file or automatically

When I make it verbose:

mdadm --assemble --scan --verbose

mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/xvdb: Device or resource busy
mdadm: /dev/xvdb has wrong uuid.
mdadm: cannot open device /dev/xvdc: Device or resource busy
mdadm: /dev/xvdc has wrong uuid.

Answer 1

Typically the mdadm.conf file will contain the information necessary to reassemble the array on boot. Of particular importance is the UUID contained in that file - which must match that of your array (find it with mdadm -D /dev/md0).

If nothing seems wrong, one option you may pursue is to rename the mdadm.conf file and use dpkg-reconfigure mdadm to regenerate it with the correct information from your array.

This thread on the topic is worth a read.

For your mount to be successful, you typically need to:

1. Load the md module (either with modprobe md (or raid0 instead of md) or by adding it to /etc/modules)
2. Assemble the array e.g.:

mdadm --assemble /dev/md0 /dev/xvdb /dev/xvdc

(Alternatively, use --scan to assemble all, instead of specifying an array)

If all else fails, you can add the above to an init script, to have the array assemble on boot.

Since the mounting is dependent on the existence of the device (md0), you may want add nobootwait to your fstab (or, move the mounting into the script that runs the assemble command).