Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

First, I launch a fresh large instance using the Elestic AMI ami-c162a9a8. Initially, I run:

df -h

/dev/xvda1            9.9G  814M  8.6G   9% /
udev                  3.7G  4.0K  3.7G   1% /dev
tmpfs                 1.5G  156K  1.5G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  3.7G     0  3.7G   0% /run/shm
/dev/xvdb             414G  199M  393G   1% /mnt

cat /etc/fstab

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
/dev/xvdb   /mnt    auto    defaults,nobootwait,comment=cloudconfig 0   2

I then run this script:

#!/bin/bash
apt-get install -y mdadm --no-install-recommends

# Configure Raid - take into account xvdb or sdb
DRIVE=`df -h | grep -v grep | awk 'NR==2{print $1}'`

if [ "$DRIVE" == "/dev/xvda1" ]; then
        DRIVE_1=/dev/xvdb
        DRIVE_2=/dev/xvdc
else
        DRIVE_1=/dev/sdb
        DRIVE_2=/dev/sdc
fi

umount /mnt
dd if=/dev/zero of=$DRIVE_1 bs=4096 count=1024
dd if=/dev/zero of=$DRIVE_2 bs=4096 count=1024
partprobe
mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=2 $DRIVE_1 $DRIVE_2
echo DEVICE $DRIVE_1 $DRIVE_2 | tee /etc/mdadm.conf 
mdadm --detail --scan | tee -a /etc/mdadm.conf
blockdev --setra 65536 /dev/md0
mkfs -t ext3 /dev/md0
mkdir -p /mnt/myraid && mount -t ext3 -o noatime /dev/md0 /mnt/myraid

# Remove xvdb/sdb from fstab
chmod 777 /etc/fstab
sed -i '$ d' /etc/fstab

# Make raid appear on reboot
echo "/dev/md0 /mnt/myraid ext3 noatime 0 0" | tee -a /etc/fstab

When the script finishes, I again run:

df -h

/dev/xvda1            9.9G  815M  8.6G   9% /
udev                  3.7G  8.0K  3.7G   1% /dev
tmpfs                 1.5G  164K  1.5G   1% /run
none                  5.0M     0  5.0M   0% /run/lock
none                  3.7G     0  3.7G   0% /run/shm
/dev/md0              827G  201M  785G   1% /mnt/myraid

cat /etc/fstab

LABEL=cloudimg-rootfs   /    ext4   defaults    0 0
/dev/md0 /mnt/myraid ext3 noatime 0 0

When I reboot the box, I get the error: port 22: Connection refused. When I tried to debug the script, I found that stopping after this command, mkdir -p /mnt/myraid && mount -t ext3 -o noatime /dev/md0 /mnt/myraid, rebooting, then trying to manually mount gives me this error: special device /dev/md0 does not exist. I am in the dark here, what is going on?

EDIT:

After I raided and mounted (but didn't modify /etc/fstab), I ran:

mdadm -D /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Mon Dec 19 06:13:44 2011
     Raid Level : raid0
     Array Size : 880730112 (839.93 GiB 901.87 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Mon Dec 19 06:13:44 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 256K

           Name : ip-10-91-18-80:0  (local to host ip-10-91-18-80)
           UUID : 36946c0e:db95eb34:bf22c078:45958378
         Events : 0

    Number   Major   Minor   RaidDevice State
       0     202       16        0      active sync   /dev/xvdb
       1     202       32        1      active sync   /dev/xvdc

Then I rebooted and ran:

mdadm --assemble /dev/md0 /dev/xvdb /dev/xvdc:

mdadm: cannot open device /dev/xvdb: Device or resource busy
mdadm: /dev/xvdb has no superblock - assembly aborted

I also tried:

mdadm --assemble --scan

mdadm: No arrays found in config file or automatically

When I make it verbose:

mdadm --assemble --scan --verbose

mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/xvdb: Device or resource busy
mdadm: /dev/xvdb has wrong uuid.
mdadm: cannot open device /dev/xvdc: Device or resource busy
mdadm: /dev/xvdc has wrong uuid.
share|improve this question

1 Answer 1

up vote 2 down vote accepted

Typically the mdadm.conf file will contain the information necessary to reassemble the array on boot. Of particular importance is the UUID contained in that file - which must match that of your array (find it with mdadm -D /dev/md0).

If nothing seems wrong, one option you may pursue is to rename the mdadm.conf file and use dpkg-reconfigure mdadm to regenerate it with the correct information from your array.

This thread on the topic is worth a read.

For your mount to be successful, you typically need to:

  1. Load the md module (either with modprobe md (or raid0 instead of md) or by adding it to /etc/modules)
  2. Assemble the array e.g.:
mdadm --assemble /dev/md0 /dev/xvdb /dev/xvdc

(Alternatively, use --scan to assemble all, instead of specifying an array)

If all else fails, you can add the above to an init script, to have the array assemble on boot.

Since the mounting is dependent on the existence of the device (md0), you may want add nobootwait to your fstab (or, move the mounting into the script that runs the assemble command).

share|improve this answer
    
So, I added some updates under "Edits" after trying your suggestions. I've seen various blogs point config to /etc/mdadm.conf, and others point to /etc/mdadm/mdadm.conf...do you think this could be the culprit? I tried both at one point, but not with updated suggestions. –  user52077 Dec 19 '11 at 6:36
    
Your script is using /etc/mdadm.conf (but the one mdadm uses depends on how you set it up) - worst case, try with a copy in both places (your --scan command suggests that it is reading a config file without any arrays listed). Since you don't seem to be able to manually assemble the array, I don't think the problem has to do with your mdadm.conf. Try adding a --verbose to the assemble command to see if you can get more information, and perhaps run it with --examine on /dev/xvdb to see if you can determine why there is no superblock. Ephemeral storage should persist between reboots. –  cyberx86 Dec 19 '11 at 7:03
    
So, everything seems to go OK when raiding. I added a verbose argument after rebooting and not modifying fstab, and got this: sudo mdadm --assemble --scan --verbose mdadm: looking for devices for /dev/md0 mdadm: cannot open device /dev/xvdb: Device or resource busy mdadm: /dev/xvdb has wrong uuid. mdadm: cannot open device /dev/xvdc: Device or resource busy mdadm: /dev/xvdc has wrong uuid. –  user52077 Dec 19 '11 at 23:51
1  
Well, this turned out to be a fun problem - I gave it a try with your AMI. The problem is that on reboot your instance is mapping the raid array to a different block device (i.e. not md0 - in my case it was md127) - find yours with ls /dev/md*. Once you stop mdadm (mdadm --stop /dev/md127 - replace with your device) you can successfully assemble the array - which will map to md0 (should have restarted here, but didn't); Run dpkg-reconfigure mdadm to update your config and make the changes persist. Finally - /etc/mdadm/mdadm.conf is the default, the other (/etc/mdadm.conf) is alternate. –  cyberx86 Dec 20 '11 at 2:13
    
Wow. Really interesting fix. Thanks! –  user52077 Dec 20 '11 at 3:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.