Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Is there any way to recover from accidental deletions of an Amazon S3 Bucket?

We've got critical info in our buckets and I need to mitigate the risk of accidental or malicious deletions of the bucket itself.

I know I can sync the entire bucket locally, but this isn't too practical if my bucket size is 100GB.

Any ideas on backup strategies?

share|improve this question

6 Answers 6

One possible solution could be to just create a "backup bucket" and duplicate your sensitive info there. In theory your data is safer in S3 than in your hard drive.

Also, I'm not sure if accidental deletions are a real problem because you'll need to accidentally delete all your bucket keys before you could delete the bucket.

share|improve this answer
    
+1 since it'd be pretty hard to "accidentally" delete everything in a bucket and then subsequently delete the bucket too. –  DL Redden Jun 28 '09 at 4:13
5  
if you're using a tool like s3cmd, it's no harder than it is to delete an entire directory tree with rm -rf –  jberryman Feb 24 '10 at 3:13
    
What about Amazon Glacier? Is it an option? –  Tony Sep 9 '13 at 2:03

Another possible solution is to replicate your bucket to the Europe zone in S3. This may persist the bucket after your accidental deletion long enough to recover.

share|improve this answer

This isn't a cheap solution, but if your buckets really are critical, here's how you do it: boot an Amazon EC2 instance and sync the content there periodically.

Amazon EC2 is their virtualization hosting provider. You can spin up instances of Linux, Windows, etc and run anything you want. You pay by the hour, and you get a pretty big storage space locally for that server. For example, I use the "large" size instance, which comes with 850GB of local disk space.

The cool part is that it's on the same network as S3, and you get unlimited transfers between S3 and EC2. I use the $20 Jungle Disk software on a Windows EC2 instance, which lets me access my S3 buckets as if they were local disk folders. Then I can do scheduled batch files to copy stuff out of S3 and onto my local EC2 disk space. You can automate it to keep hourly backups if you want, or if you want to gamble, set up JungleDisk (or its Linux equivalents) to sync once an hour or so. If someone deletes a file, you've got at least a few minutes to get it back from EC2. I'd recommend the regular scripted backups though - it's easy to keep a few days of backups if you're compressing them onto an 850GB volume.

This is really useful for SQL Server log shipping, but I can see how it'd accomplish your objective too.

share|improve this answer

To modify Brent's (excellent) answer a bit; you shouldn't need to keep the instance running. Create an EC2 AMI that pulls your data down, syncs it to an EBS volume, snapshots that volume and shuts itself down.

You could keep the volume running as well by itself, but snapshotting it should be sufficient for a backup. If your custom AMI does all of this (including shutting itself down after it's done) with no interaction, then your 'backup' script just needs to 'ec2run -n 1 -t m1.small ami-' and fire-and-forget.

share|improve this answer

You could use s3cmd http://s3tools.org/s3cmd

So to backup a bucket called mybucket

s3cmd mb s3://mybucket_backup
s3cmd --recursive cp s3://mybucket s3://mybucket_backup
share|improve this answer
2  
Is there a faster way to do this? If there are n keys in the bucket, there are at least n requests for copying plus some for listing (and probably checking the results). This may take quite a while for large buckets. –  Kariem Jan 18 '12 at 0:37
    
Could you detail the backup operation when mybucket is corrupted and one needs to restore mybucket_backup? –  Augustin Riedinger Sep 13 '13 at 9:35

Another approach is to enable S3 versioning on your bucket. You can then restore deleted files etc. See the S3 documentation for how to enable this

Using third party tools like BucketExplorer makes working with versioning pretty trivial (vs calling the API yourself directly).

You can also enable multi-factor authentication delete for your S3 buckets - which makes "accidental deletion" that little bit harder ;)

More no Multi Factor Authentication Delete
More on Deleting Objects

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.