Some AWS instances have "ephemeral disks" attached, which are much faster than EBS. But ephemeral disks will be blank and uninitialised when your instance is stopped and started. The data on disk generally survives an instance reboot though.
Question: Should I use a software RAID1 on my AWS instance, built over an ephemeral disk and an EBS volume?
My thinking is that the raid1 will come up in degraded mode with the EBS volume only, and then we can use mdadm commands to add the blank ephemeral disk back into the raid. This will allow the app to start up 5-10 minutes sooner, at the cost of worse performance while the raid1 synchronises.
Background: I have an app that uses ~40 GB of data files. Access times directly corellate with performance, so faster the disk the faster the app works.
Historically we've run something from rc.local to rsync data from an EBS disk to the ephemeral disk, and then started the software. The synch takes 5-10 minutes, better than the 5-20 minutes it took to synch from another instance. In the past we've even used the data files from a ramdisk, which was not as fast as the ephemeral disks.
More info - this is an i3.4xlarge so it has 2x NVME ephemeral drives.
# hdparm -t /dev/md? /dev/nvme?n1 /dev/xvd? /dev/md0: 9510 MB in 3.00 seconds = 3169.78 MB/sec RAID0 of two eph drives /dev/nvme0n1: 4008 MB in 3.00 seconds = 1335.74 MB/sec Eph drive /dev/nvme1n1: 4014 MB in 3.00 seconds = 1337.48 MB/sec Eph drive /dev/xvda: 524 MB in 3.01 seconds = 174.17 MB/sec gp2 16GB, 100 IOPs root /dev/xvdf: 524 MB in 3.01 seconds = 174.23 MB/sec gp2 120GB, 300 IOPs data /dev/xvdz: 874 MB in 3.01 seconds = 290.68 MB/sec gp2 500GB, 1500 IOPs raid-seed disk
I have created a raid1 with
mdadm --create /dev/md1 --raid-devices=3 --verbose --level=1 /dev/nvme?n1 /dev/xvdz
$ cat /proc/mdstat Personalities : [raid0] [raid1] md1 : active raid1 nvme1n1 nvme0n1 xvdz 524155904 blocks super 1.2 [3/3] [UUU] bitmap: 0/4 pages [0KB], 65536KB chunk
Curiously, the raid reads about as fast as the faster drives, and is not limited to the speed of the slowest disk.
/dev/md1: 4042 MB in 3.00 seconds = 1346.67 MB/sec /dev/nvme0n1: 4104 MB in 3.00 seconds = 1367.62 MB/sec /dev/nvme1n1: 4030 MB in 3.00 seconds = 1342.93 MB/sec /dev/xvdz: 668 MB in 3.01 seconds = 222.26 MB/sec
A power-off/on returns a degraded raidset, but the app can still run albeit slower. The cost of waiting 5-10 minutes is avoided, and I can re-add the ephemeral disks to the raid on the fly without an app restart.
So while it seems to work perfectly, is there anything I've missed or not considered?