14

Let's say I've got S3 versioning enabled for my bucket: http://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html

Then, let's say someone (for example, junior employee) messes up the S3 bucket (deletes some files accidentally, etc.)

How can I then restore the entire versioned bucket to a particular point in time? I believe this should be possible given S3's API, but I'd rather not have to write such a script myself, for fear of missing something (I'm not an AWS expert).

Are there are good solution to this problem? I'm using the S3 bucket as an image store for my Rails app, so something Ruby-based that I could use as a rake task would be ideal.

16

You can use s3-pit-restore

S3 Point in Time Restore is a tool you can use exactly to restore a bucket or a subset of a bucket to a given point in time, like this:

s3-pit-restore --bucket my-bucket --dest my-restored-bucket --timestamp "06-17-2016 23:59:50 +2"

What s3-pit-restore actually offers:

  • Restore of all files with timestamp less than the given one
  • Restore of a whole bucket or a bucket prefix
  • Parallel download of multiple files with a great overall speed
  • Customization of parallel workers count to optimize bandwidth usage
  • Restore from s3 bucket versions or from glacier if enabled
  • This doesn't seem to work for me. Destination only creates a local directory despite the description of the tool. – Arthur Jul 30 '18 at 6:04
  • You have saved my day. Thanks for letting us know the amazing tool. – cyberrspiritt Jul 2 at 4:34
2

If I understand the documentation correctly, when you have versioning enabled deleting the file simply reverts the "latest" version back one version number. This however does not give the ability to restore an entire bucket. This makes the previous versions in S3 not suitable for your needs (i.e, recovery from deletion).

Keep a backup someplace else as well just in case. Stack Overflow has a question/answer on this using s3cmd. I'm sure you could find a Ruby-based script somewhere or ask on that site for help if you need it.

  • Correct. You're versioning each individual object in the bucket, not the bucket as a whole. – EEAA Apr 17 '14 at 12:35
  • 1
    Oh, I understand all that. Which is why I realize it's not that simple. I'd probably have to traverse all the files in the bucket, get version info for each file, and then pick the correct item (if it exists) based on the date time I want to "revert" to. Not so simple. I did figure Amazon would have though of something for such a common use case, but alas, no... So I was wondering if someone wrote this tedious script already. I will look into s3cmd, but I do like having versioned snapshots on S3 as well. – elsurudo Apr 17 '14 at 18:43
  • this answer contains incorrect information - a simple delete inserts a delete marker and future requests return a 404, not the previous version. To RESTORE you can copy an old version to a new version or you can delete with a specific version of the current object - then future gets are given the second-to-latest version. docs.aws.amazon.com/AmazonS3/latest/dev/… to be fair, the docu around bucket versioning tends to be vague and lacking... – keen Jan 12 '17 at 15:46
  • @keen Note that this question was answered almost three years ago...it's very possible they updated the documentation. Good catch, though. – Nathan C Jan 12 '17 at 18:30
  • the link referenced for docu is about recovery (the super high level recovery view) and hasn't changed - and to be fair, it's more than a little confusing when it starts talking about deletes. just wanted to make sure no one saw this and thought "just deleting an object from a versioned s3 bucket means the old version will start being returned" - that CAN happen, but you have to specifically delete the current version (.../key?versionId=xyz) for that to happen... – keen Jan 12 '17 at 22:08
2

Here is a handy little php script I used to restore my bucket from a back up bucket with versioning:

To run back up into a bucket with versioning turned on

#CONFIGURATION
export bucket_to_backup='example-of-a-bucket-name';
export backup_bucket='example-of-a-backup-bucket-name'; 
export AWS_ACCESS_KEY_ID=ASDFDSAFADSFDASF;
export AWS_SECRET_ACCESS_KEY=adsfdsaf86adsf5adsf568aadf75ads;
#you will need to download and install the s3cmd tool at the following url:
#http://s3tools.org/
s3cmd --access_key=$AWS_ACCESS_KEY_ID --secret_key=$AWS_SECRET_ACCESS_KEY --recursive sync s3://$bucket_to_backup s3://$backup_bucket;

restore backup from files that were versioned between these two timestamps

export startSnapShotTimeStamp=1427577934;
export endSnapShotTimeStamp=1427577939;
#do the restore with php script
php awsUtil.php sync example-of-a-backup-bucket-name example-of-a-bucket-name $startSnapShotTimeStamp-$endSnapShotTimeStamp

you can download the PHP script here: http://boulderapps.co/post/backing-up-and-restoring-s3-to-a-versioned-bucket

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.