6

I'm running an EBS-backed instance which acts as a software development team's build server (running Jenkins and host of other services). The server is running Linux (latest Ubuntu from the official AMIs).

I'd like to take regular, automated snapshots of the instance's associated EBS volume. I only need to keep one latest backup (i.e. old snapshots should be pruned), and a good frequency would be once a day.

It seems that Amazon does not provide such backup service out of the box, so you have to either go with 3rd party scripts or roll your own solution.

My question is, what is the simplest way to achieve this? I'd like a minimal amount of hassle, configuration, and external dependencies. Setting this up as some kind of timed script on the Linux box itself is, to my knowledge, a valid option.

1

5 Answers 5

5

Based on Jonik's concept, I created a python script using boto. You provide it a list of volumes to snapshot, and how many trailing snapshots to keep for each volume:

# Define the snapshots manage. We'll snapshot the specified volume ID, and only keep the X newest ones.
snapshots = [("vol-XXXXXXXX", 30), ("vol-YYYYYYYY", 180)]

import boto.ec2
auth = {"aws_access_key_id": "YOURACCESSKEY", "aws_secret_access_key": "YOURSECRETKEY"}
ec2 = boto.ec2.connect_to_region("YOURREGIONNAME", **auth)
description = "automated backup"
for volume, num_trailing in snapshots:
  snaps = ec2.get_all_snapshots(filters={"volume-id": volume, "description": description})
  print "%s: Creating new snapshot. %s automated snapshots currently exist." % (volume, len(snaps))
  ec2.create_snapshot(volume, description)
  purgeable = sorted(snaps, key=lambda x: x.start_time)[:-num_trailing]
  print "Deleting snapshots for %s > %s: %s" % (volume, num_trailing, purgeable)
  for snap in purgeable:
    ec2.delete_snapshot(snap.id)

I set this up as Jenkins job (via the Python plugin), configured to run daily. If you are using IAM to manage credentials, note that this will require in ec2 policies: DescribeRegions, DescribeVolumes, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CreateTags (because of boto's implementation).

1
  • 2
    Leeeeeerooooooooyyyyyyy
    – pauska
    Jan 14, 2012 at 1:51
4

Okay, for what it's worth, here's what I did. I hope my feeble scripts encourage people to post better solutions!

I wrote two simple bash scripts and automated them using cron. (For now I run these on a local server, as I think (?) it's not recommended to put AWS's certificates in the instances/AMIs/EBSs themselves.)

To create a new snapshot:

# ESB volume associated with the instance we want to back up:
EBS_VOL_ID=vol-xxxxyyyy

ec2-create-snapshot --region eu-west-1 -K pk.pem -C cert.pem -d "Automated backup" $EBS_VOL_ID 

To prune all except latest snapshot:

EBS_VOL_ID=vol-xxxxyyyy

ec2-describe-snapshots --region eu-west-1 -K pk.pem -C cert.pem  | grep "Automated backup" | grep "$EBS_VOL_ID" | awk '{ print $5 "\t" $2 }' | sort > .snapshots

latest_id=$(tail -n1 .snapshots | awk '{ print $2 }')

cat .snapshots | awk '{ print $2 }' > .snapshot_ids
for i in $(cat .snapshot_ids) 
do
    if [ "$i" != "$latest_id" ]
    then
        echo "Deleting snapshot $i"
        ec2-delete-snapshot --region eu-west-1 -K pk.pem -C cert.pem $i
    fi
done

(This parses appropriate snapshot information from ec2-describe-snapshots output and creates a temp file with [timestamp tab snapshot-id] entries (e.g. 2011-06-01T10:24:36+0000 snap-60507609) where the newest snapshot is on the last line.)

Notes:

  • Put your X509 certificate and private key in some place where the scripts can find them.
  • You must explicitly specify --region with all commands. Otherwise e.g. ec2-create-snapshot would fail with volume ID being unknown. (YMMV if you use the default region "us-east-1".)
  • I used a snapshot description ("Automated backup") as a marker to avoid the prune script deleting other snapshots of the volume in question (e.g. snapshots related to AMIs).

Disclaimer: This became partly an exercise in Bash/Unix programming for me, especially the prune script. I readily admit you'd most likely get a much clearer result with e.g. Python, when you need logic like "do something for all but the last item in a list". And even with Bash you could probably do this more elegantly (for instance, you don't really need temp files). So, please feel free to post other solutions!

1
  • 1
    Of course, there are probably scenarios where the approach of keeping only one backup would fail miserably...
    – Jonik
    Jun 1, 2011 at 12:05
1

If you're open to external utilities, check out Skeddly.

Disclosure: I'm the CEO of Eleven41 Software, the company behind Skeddly.

1
1

I expanded on the idea of Jonik's script to allow multiple snapshots to be retained. The code is too long to fit in a comment so I'm adding a new answer. This code assumes all the right environment variables have been set up for the CLI tools. Also, this defaults to taking a snapshot of the current instance.

# Look up our instance ID using the magic URL
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

# The number of previous backups we want to keep
N=3

# get list of locally attached volumes via EC2 API:
VOLUME_LIST=$(ec2-describe-volumes | grep ${INSTANCE_ID} | awk '{ print $2 }')
DATE=$(date '+%Y-%m-%d-%H%M%S')

sync

# actually creating the snapshots
for VOLUME in $(echo $VOLUME_LIST); do
    echo "Processing volume $VOLUME"
    SNAPSHOT_LIST=$(ec2-describe-snapshots | grep completed | grep "Automatic snapshot" | grep $VOLUME | awk '{print $5 "\t" $2}' | sort | head "--lines=-$N" | awk '{print $2}')
    ec2-create-snapshot $VOLUME -d "Automatic snapshot on $DATE"
    for SNAPSHOT in $(echo $SNAPSHOT_LIST); do
        ec2-delete-snapshot $SNAPSHOT
    done
done
0

I have written a script in PHP that will automate EBS Snapshots and delete old ones. It will even email the results of the snapshots to you. You must configure the AWS PHP SDK and PHPMailer for email functionality, but both of these steps are pretty easy. Then you just run the script every night with CRON or Windows Scheduled Tasks. Detailed instructions and code can be found on my blog:

http://www.caleblloyd.com/software/automatically-take-ebs-snapshots-and-delete-old-ones-with-php-script/

1
  • Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.
    – Scott Pack
    Oct 20, 2012 at 17:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .