2

The situation: we have a MySQL server in AWS using three EBS volumes in a striped logical volume to hold the database data. The logical volume is close to full, so we need to expand it somehow. One option would be to relaunch the server and attach new, larger EBS volumes and then restore from an existing snapshot from the old server. (We'll need to do this eventually the next time we have to replace the server for other reasons, which is why I'm skipping over options that change the existing server in-place.)

My question is this: if we have three EBS snapshots of (say) 50GB each that make up the logical volume, can I restore these snapshots to a fresh server with three EBS volumes that are larger (say, 75GB or 100GB or whatever)? Or is this a recipe for disaster? Are there any special steps I'd need to take over our current process? If I wanted to instead add a fourth EBS volume of 50GB, could I restore the three snapshots to the new four-volume group?

4

You can simply increase the size of the current volume (more detail here). I know you said you discounted that option because you'll need a new instance, but you can easily mount these same volumes to a new instance. If you need the old and new instances working at the same time with no interruption (you didn't say) then this won't work.

NB: I'm explicitly not going to copy a large web page to here. AWS documentation is fairly stable, and is updated regularly, I don't want out of date answers to cause problems.

Alternate Solution

Just restore your snapshots to three new volumes, mount them to a new instance, and use them.

  • Does this work even if the logical volume is stretched across three EBS volumes? i.e. I increase the size of each of the three volumes individually and then lvextend? I know this works fine for logical volumes contained on a single EBS disk but have never tried it for a striped volume across multiple EBS disks. – wesley.fok Nov 23 '18 at 19:35
  • I haven't tried it, but I can't see why having software RAID on the volumes would make any difference. They great thing about AWS is you can try it out easily. Clone your volumes (via snapshots most likely), attach them to a cloned instance, and try it on a mirror of your current machine. Record your commands so you can do it on your production machine. – Tim Nov 23 '18 at 19:49
  • Tried this on a test database, and it turns out that you can indeed restore EBS snapshots to volumes larger than the snapshot size, and everything works fine. You will need to run pvresize and lvextend afterwards to get the filesystem to recognize the new space, but overall it was straightforward. Thanks! – wesley.fok Nov 27 '18 at 16:08
3

Because it's LVM the Logical Volume can be extended by simply adding more disks (Physical Volumes = EBS volumes) to the Volume Group.

  1. Create an AMI from your current server so you've got a way back if things go wrong.
  2. Create 3 new EBS volumes (e.g. 3x 50GB)
  3. Create LVM partition on each of them and pvcreate the partition
  4. Add the volumes to Volume Group using vgextend
  5. Extend the MySQL Logical volume with lvresize and then extend the filesystem (depends on your fs type, e.g. resize2fs).
  6. Done.

You can try to expand the current EBS volumes instead of adding 3 new ones but then you'll still have to expand all the partitions, update the VG, expand the LV and resize the filesystem. It's a little more prone to a mistake but should work the same.

The benefit of using multiple EBS volumes is that IOPS bandwidth is per-volume. I.e. more volumes give you more IO bandwidth. It may not be relevant if your MySQL server is only lightly loaded but still good to know.

Or use Amazon Aurora instead

If I were you I would migrate your database to Amazon Aurora which is a MySQL-compatible managed database. There is virtually no need to run self-managed MySQL on EC2 unless you do something very very special. With Aurora you'll get:

  • Automatic disk space management (no more running out of space, no more managing EBS volumes)
  • Automatic OS and MySQL patching
  • Automatic fail-over in case of underlying physical host failure (how do you handle that on your current instance?)
  • Automatic backups with point-in-time recovery (great if you accidentally TRUNCATE wrong_table_oops; ;)
  • All that while providing nearly 100% compatibility with stock standard MySQL

I have migrated many MySQL databases to Aurora and in almost all cases it was as simple as mysqldump from the old database and load that into Aurora (alternatively use AWS DMS) and change the database host name in the application. Simple as that and will save you lots of MySQL management overhead going forward.

Hope that helps :)

  • mysqldump is probably a better option IMHO. I used DMS once, it lost all the indexes, auto-increment values, and such - not sure if I used it incorrectly or if that's a "feature". – Tim Nov 23 '18 at 23:20
  • The concern on my end was that I didn't really know how creating six EBS volumes based on snapshots taken from three EBS volumes would work, or if it would fail. Also that I didn't really need 6 volumes, but I guess that doesn't necessarily do us much harm either except in terms of storage costs (I assume the three new volumes would have to be the same size as the originals for the striping to work, but we don't actually need double the storage just yet.) Your comments on Aurora/RDS have also been on our minds for some time so we may end up moving in that direction. Thanks! – wesley.fok Nov 27 '18 at 16:12

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.