How are admins managing their EC2 EBS's and Snapshots?

Question

My company is moving our hosting to Amazon and I'm working though all the migration issues. It has been quite a challenge transitioning from physical hardware to transient virtual machines.

One of the last challenges is figuring out how to manage our EBS's and snapshots. Since their is no way currently to name them descriptively or add meaning, I'm wondering how admins are managing their resources. It's not too much of a problem with one or two servers, but how are those running multiple servers handling things. Is everyone using third party tools (like RightScale/Scalr) when they get over a handful of servers? Or are you maintaining wikis or other ancillary documentation?

Answer 1

I use ec2-consistent-snapshot ( http://alestic.com/2009/09/ec2-consistent-snapshot ) to take my snapshots (on RHEL/CentOS derived Amazon's Linux) - it is written in Perl and uses the API directly. You specify the volumes you want to snapshot and it allows you to add a description.

(In AWS console you can add tags if you need finer grained control, but for a simple backup that might not be necessary, and isn't yet supported by the Perl module used above (Net::Amazon::EC2)).

The above script does not delete snapshots (only create new ones - consistently (i.e. it can freeze the disk/db before the snapshot). Since it is fairly commonly used and I couldn't find a compatible Perl script to delete old snapshots I wrote my own ( http://www.thatsgeeky.com/2011/06/rotating-ebs-snapshots-ec2-prune-snapshots/ ). It does the job (grandfather-father-son rotation) well and uses the same dependencies and parameters as the above script.

Of course, both are set to run via Cron.

(In theory, it should be easy enough to hook some similar scripts (there are those written in PHP and Ruby for instance) up to a database, and maintain your own log of the volumes and snapshots taken - each snapshot has a unique id so as long as record that, any further in-house organization should be easily possible. [However, for backup purposes, the volume-id and date is often sufficient])

Don't use the CLI tools - they are written in Java and are ridiculously slow compared to any direct API implementations.

Answer 2

Gradually Amazon are adding all the features required to do effective management that is required if Amazon are to appeal to the enterprise market. Tags have been added and enhancements to allow a user to only have access to certain features is coming. In the future surely scripting and deployment will be handled. I am adding these features to my free tool ElastDream.

Answer 3

Short of writing your own tool to index EBS and snapshot IDs against text labels, I've found that using a free RightScale account and assigning nicknames is the best way to manage our EBS volumes and snapshots.

I personally think that the inability to assign an easy-to-remember label to an EC2 instance, AMI or volume is a big hole in the current Amazon offering - it just seems like a no-brainer to me.

Answer 4

I use ElasticFox to manage my snapshots (and all my servers too). It has tagging, which makes it easy to keep track of which one is which.

I also use the boto library when I want to do things programatically. However, if you do that, you'll need to create your own datastore. SQS and Sqlite are good options there.

Answer 5

ElasticFox is surely one of the most convenient tools for (manually) managing EC2. But the key part is - imaging your instances, regularly - set up some automatic initialisation once a new instance is started up: - partition and mount the ephemeral drives - mount the EBS volume once it's available - restore files and databases from either EBS, a shared storage or S3. - start the services (MySQL, Apache, Tomcat, you name it.)

Answer 6

We use chef for managing all of our EC2 resources.

Answer 7

AWS console is too low level for many day-to-day tasks, though it is improving rapidly. If you need this functionality now, you either need to bite the bullet and write your own scripts leveraging the ec2 cli tools to manage the snapshots or use a third party tool. For EBS volume management the ones that have been mentioned are good, another option is the EBS management tool so you can schedule backups, name them, restore them ,etc of BitNami Cloud Hosting (there is a free tier)

Answer 8

I have my own script for making rotating backups (grandfather - father - son ) at my blog http://three99.com/posts/python-grandfather-father-son-snapshots-backup-for-ec2/