50

Is there any way to recover from accidental deletions of an Amazon S3 Bucket?

We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket itself.

I know I can sync the entire bucket locally, but this isn't too practical if my bucket size is 100GB.

Any ideas on backup strategies?

Improve this question
1

6 Answers 6

Reset to default
25

Another approach is to enable S3 versioning on your bucket. You can then restore deleted files etc. See the S3 documentation for how to enable this

Using third party tools like BucketExplorer makes working with versioning pretty trivial (vs calling the API yourself directly).

You can also enable multi-factor authentication delete for your S3 buckets - which makes "accidental deletion" that little bit harder ;)

More on Multi Factor Authentication Delete
More on Deleting Objects

Improve this answer
1
  • 3
    Question is to achieve this without versioning.
    – Anuruddha
    Jan 10, 2018 at 9:00
15

You could use s3cmd http://s3tools.org/s3cmd

So to backup a bucket called mybucket

s3cmd mb s3://mybucket_backup
s3cmd --recursive cp s3://mybucket s3://mybucket_backup
Improve this answer
2
  • 3
    Is there a faster way to do this? If there are n keys in the bucket, there are at least n requests for copying plus some for listing (and probably checking the results). This may take quite a while for large buckets.
    – Kariem
    Jan 18, 2012 at 0:37
  • 1
    Could you detail the backup operation when mybucket is corrupted and one needs to restore mybucket_backup?
    – Augustin Riedinger
    Sep 13, 2013 at 9:35
8

One possible solution could be to just create a "backup bucket" and duplicate your sensitive info there. In theory your data is safer in S3 than in your hard drive.

Also, I'm not sure if accidental deletions are a real problem because you'll need to accidentally delete all your bucket keys before you could delete the bucket.

Improve this answer
3
  • +1 since it'd be pretty hard to "accidentally" delete everything in a bucket and then subsequently delete the bucket too.
    – DL Redden
    Jun 28, 2009 at 4:13
  • 11
    if you're using a tool like s3cmd, it's no harder than it is to delete an entire directory tree with rm -rf
    – jberryman
    Feb 24, 2010 at 3:13
  • What about Amazon Glacier? Is it an option?
    – Tony
    Sep 9, 2013 at 2:03
7

Another possible solution is to replicate your bucket to the Europe zone in S3. This may persist the bucket after your accidental deletion long enough to recover.

Improve this answer
1
  • 1
    Bucket replication is a great option. For an extra layer of protection use cross account replication to ensure any breach of the source account doesn't result in data loss.
    – Gareth Oakley
    Apr 6, 2018 at 12:34
7

This isn't a cheap solution, but if your buckets really are critical, here's how you do it: boot an Amazon EC2 instance and sync the content there periodically.

Amazon EC2 is their virtualization hosting provider. You can spin up instances of Linux, Windows, etc and run anything you want. You pay by the hour, and you get a pretty big storage space locally for that server. For example, I use the "large" size instance, which comes with 850GB of local disk space.

The cool part is that it's on the same network as S3, and you get unlimited transfers between S3 and EC2. I use the $20 Jungle Disk software on a Windows EC2 instance, which lets me access my S3 buckets as if they were local disk folders. Then I can do scheduled batch files to copy stuff out of S3 and onto my local EC2 disk space. You can automate it to keep hourly backups if you want, or if you want to gamble, set up JungleDisk (or its Linux equivalents) to sync once an hour or so. If someone deletes a file, you've got at least a few minutes to get it back from EC2. I'd recommend the regular scripted backups though - it's easy to keep a few days of backups if you're compressing them onto an 850GB volume.

This is really useful for SQL Server log shipping, but I can see how it'd accomplish your objective too.

Improve this answer
2
  • I guess you could use a micro instance and add as much EBS (Elastic Block Storage) as you needed. May be a cheaper option.
    – Shawn Vader
    Apr 29, 2015 at 7:56
  • Actually you shouldn't, because the dedicated bandwidth to and from S3 depends on the size of the EC2 instance. If you want big throughput, you need a big (= $$$$) instance. My former employer found this out the hard way.
    – John Cowan
    Jan 8, 2019 at 22:05
6

To modify Brent's (excellent) answer a bit; you shouldn't need to keep the instance running. Create an EC2 AMI that pulls your data down, syncs it to an EBS volume, snapshots that volume and shuts itself down.

You could keep the volume running as well by itself, but snapshotting it should be sufficient for a backup. If your custom AMI does all of this (including shutting itself down after it's done) with no interaction, then your 'backup' script just needs to 'ec2run -n 1 -t m1.small ami-' and fire-and-forget.

Improve this answer
1
  • I like this idea that others, this is more reasonable and cheaper solution.
    – BMW
    Apr 28, 2016 at 11:11