16

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate somewhere around 200-500GB of data per year. We would like to ensure this data is safe, particularly from accidental deletions and would like to be able to restore files that were deleted regardless of the reason.

I have read about the versioning feature for S3 buckets, but I cannot seem to find if recovery is possible for files with no modification history. See the AWS docs here on versioning:

http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html

In those examples, they don't show the scenario where data is uploaded, but never modified, and then deleted. Are files deleted in this scenario recoverable?

Then, we thought we may just backup the S3 files to Glacier using object lifecycle management:

http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html

But, it seems this will not work for us, as the file object is not copied to Glacier but moved to Glacier (more accurately it seems it is an object attribute that is changed, but anyway...).

So it seems there is no direct way to backup S3 data, and transferring the data from S3 to local servers may be time-consuming and may incur significant transfer costs over time.

Finally, we thought we would create a new bucket every month to serve as a monthly full backup, and copy the original bucket's data to the new one on Day 1. Then using something like duplicity (http://duplicity.nongnu.org/) we would synchronize the backup bucket every night. At the end of the month we would put the backup bucket's contents in Glacier storage, and create a new backup bucket using a new, current copy of the original bucket...and repeat this process. This seems like it would work and minimize the storage / transfer costs, but I'm not sure if duplicity allows bucket-to-bucket transfers directly without bringing data down to the controlling client first.

So, I guess there are a couple questions here. First, does S3 versioning allow recovery of files that were never modified? Is there some way to "copy" files from S3 to Glacier that I have missed? Can duplicity or any other tool transfer files between S3 buckets directly to avoid transfer costs? Finally, am I way off the mark in my approach to backing up S3 data?

Thanks in advance for any insight you could provide!

Update

Amazon recently announced that versioning now works with life cycle rules

1

3 Answers 3

10
+50

I have read about the versioning feature for S3 buckets, but I cannot seem to find if >recovery is possible for files with no modification history. See the AWS docs here on >versioning:

I've just tried this. Yes, you can restore from the original version. When you delete the file it makes a delete marker and you can restore the version before that, i.e: the single, only, revision.

Then, we thought we may just backup the S3 files to Glacier using object lifecycle >management:

But, it seems this will not work for us, as the file object is not copied to Glacier but >moved to Glacier (more accurately it seems it is an object attribute that is changed, but >anyway...).

Glacier is really meant for long term storage, which is very infrequently accessed. It can also get very expensive to retrieve a large portion of your data in one go, as it's not meant for point-in-time restoration of lots of data (percentage wise).

Finally, we thought we would create a new bucket every month to serve as a monthly full >backup, and copy the original bucket's data to the new one on Day 1. Then using something >like duplicity (http://duplicity.nongnu.org/) we would synchronize the backup bucket every >night.

Don't do this, you can only have 100 buckets per account, so in 3 years you'll have taken up a third of your bucket allowance with just backups.

So, I guess there are a couple questions here. First, does S3 versioning allow recovery of >files that were never modified?

Yes

Is there some way to "copy" files from S3 to Glacier that I have missed?

Not that i know of

2
  • Thanks Sirex, one clarification: we intended to only keep one backup bucket at a time (the current month), and archive the last month's backup to Glacier, and then delete the Glacier copies on some general retention policy (each month for 3 months then quarterly for a year). Do you think this is possible, or is there some other AWS limitation preventing this strategy?
    – Fiver
    Nov 12, 2013 at 3:21
  • hmm, i guess that's reasonable. I've not tried that in anger though (or glacier for that matter, we use Sydney and it wasn't available when we looked into it)
    – Sirex
    Nov 12, 2013 at 3:36
3

i dont like to have s3 mounted, because its slow and can hang and performs like old school nfs. better to just up/down as needed.

http://s3tools.org/s3cmd

its easy to script... dont forget your ~/.s3cfg file

2
  • Wow, this looks cool! Nov 12, 2013 at 2:31
  • @nandoP +1 I had thought of using the s3cmd and rolling my own
    – Fiver
    Nov 12, 2013 at 3:15
1

Sounds like you might need S3FS which basically mounts your S3 buckets as if they were local file systems:

http://code.google.com/p/s3fs/wiki/FuseOverAmazon

I prefer to use this forked version since it retains the same folder structure as what is created in the web control panel for Amazon S3:

https://github.com/tongwang/s3fs-c

Then I have some shell scripts in place that basically use rsync to sync the data to my local setups.

In general, I find S3FS works best for reading content placed within Amazon S3 buckets. Writing is not as consistent. But for the purposes of backing up S3 buckets, the Tony Wang fork of S3FS works great.

2
  • 1
    This sounds useful. Does it transfer directly between buckets, without downloading to the client?
    – Fiver
    Nov 7, 2013 at 23:07
  • Possibly yes. I mainly use it to mount on a local server & backup to our local machines. But since it does have write capability, I would not rule using it as a “go between” to shuffle data between buckets. Basically on an intermediary machine, mount one bucket on one mount point. Then mount the other bucket as another mount point. And then just do an rsync. But as I said, the modification times/dates can be weird with Amazon S3 since it is not really a file system. But experiment. Nov 8, 2013 at 3:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .