I have created 3 RAID0 arrays using mdadm on 3 Rackspace CloudServers using Rackspace's CloudBlockStorage as their devices. Each array has their own set of CBS volumes. RS-CBS has an upper limit of 1TB for their drives and I needed larger storage. One of the servers is for data storage, the second is a live mirror of that server, and the third is a nightly snapshot. I created the array on each of the servers using this command:
mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/xvdb1 /dev/xvdd1 /dev/xvde1
When it came time to grow the array, and since this was my first time growing one, I started on the least critical server: the snapshot server. I added a new volume and, with a little fiddling, came up with this command:
mdadm --grow /dev/md0 --raid-devices=4 --add /dev/xvdf1
According to the mdadm man page it said you are not able to grow a RAID0 so mdadm converts it to RAID4, grows it, and then converts back. (It never converts back, but that is another issue). The array reshapes, but the size never changes. My assumption is that the new drive just became the parity drive for the array. So to get the space I need, I added another volume:
mdadm --grow /dev/md0 --raid-devices=5 --add /dev/xvdg1
This reshapes and gives me the extra 500GB I was hoping for so I do the
resize2fs /dev/md0 and am good to go; 2TB of space.
So this is where the weirdness happens. I switch to grow the array on the mirror server. Knowing from the previous steps that I will need to add two volumes to get the extra space I need, I use this command:
mdadm --grow /dev/md0 --raid-devices=5 --add /dev/xvdf1 /dev/xvdg1
It reshapes and things look fine, but when I do the
resize2fs /dev/md0 followed by a
df -hal I see that it did grow and is reporting 2.5TB of space, not the 2.0TB I was expecting. It's as if it didn't create the parity drive I assumed it did from the first run and used all available space for the array.
One thing that looked different was the output of
mdadm -D /dev/md0 after the reshape on the mirror server. I saw this for the RAID devices:
Number Major Minor RaidDevice State 0 202 17 0 active sync /dev/xvdb1 1 202 49 1 active sync /dev/xvdd1 2 202 65 2 active sync /dev/xvde1 5 202 97 3 active sync /dev/xvdg1 4 202 81 4 active sync /dev/xvdf1 5 0 0 5 removed
The output of
cat /proc/mdstat shows that it did convert to RAID4:
Personalities : [raid0] [raid6] [raid5] [raid4] md0 : active raid4 xvdg1 xvdf1 xvdd1 xvdb1 xvde1 2621432320 blocks super 1.2 level 4, 512k chunk, algorithm 5 [6/5] [UUUUU_] unused devices: <none>
So my concerns are ... did I grow the first array (snapshot server) incorrectly and it should've gotten 2.5TB, but my error only gave it 2.0TB? Or did I grow the second array (mirror server) incorrectly and don't have the parity drive that I thought I was going to get? What can I do, or what should have I done, to grow these arrays consistently?