3

I'm looking for a way to take daily backups of an AWS bucket as incremental backups. These are to be stored offline and away from AWS.

For other storage systems (such as NAS drives) I use a daily rsync for backups. Using rsync's --link-dest switch, I'm able to take a full snapshot every day of the remote file system. Any files which have not changed since the previous backup are hardlinked to the previous backup. This means that full daily snapshots only take the storage space of incremental backups.

I would like to setup something similar for an amazon S3 bucket. There's 20GB in the bucket but only ~50MB changes per day.

Note this is backup the content of an S3 bucket, NOT backup other content to S3 bucket.

I can see how I would use the AWS CLI tools to do full backups. I don't see how I can do incremental backups.

I guess I could (daily) synchronise S3 to a local hard drive, then daily backup the local hard drive. This feels very clunky.

Edit

This was intended as a simple technical question, not a general discussion of backup security. But since I'm being asked "why do you need this", I now see I need to explain basic principles of backups.

Anecdote: I recently witnessed a third party IT provider drop (entirely) an S3 bucket because of a miscommunication. This could have been very costly (~£100K of recent work, ~£1M total work). Luckily we happened to also have copies on our local laptops and for only £1K we rebuilt the content for them.

It has renewed my conviction that the only valid "backup" is on an isolated system stored offsite and offline, and with a media rotation that effectively implements a time lock. Other backups can enhance, providing more rapid recovery etc... but holding all your AWS backups on your own AWS account just isn't safe because ... user error.

  • 1
    s3 sync should only transfer changed files. docs.aws.amazon.com/cli/latest/reference/s3/sync.html "A s3 object will require downloading if the size of the s3 object differs from the size of the local file, the last modified time of the s3 object is newer than the last modified time of the local file, or the s3 object does not exist in the local directory." – ceejayoz Nov 5 '18 at 16:00
  • @ceejayoz The feature you're describing synchronises two file systems. How would you use this feature to do daily incremental backups? – Philip Couling Nov 5 '18 at 16:10
  • Regularly sync the bucket to the server that performs your backups, then perform the incremental backups against that synced folder. – ceejayoz Nov 5 '18 at 16:15
  • Ouch that's messy. Have edited the question to exclude it. Though it does give me a fallback option. – Philip Couling Nov 5 '18 at 16:16
  • 3
    couling I suggest you need to show a little more respect and appreciation to people who are doing their best to help you. Your question was initially incomplete and somewhat ambiguous and couldn't be answered in the way you appeared to want. Your comments on the answer by @MLu could be phrased in a different way to be a little more polite. You also need to answer questions people ask to clarify your needs if you want useful answers – Tim Nov 7 '18 at 1:07
4

There is a way besides aws s3 sync, but it might be just as clunky. You see, it comes down to adding a Lambda hook on the S3 bucket that triggers on PUTs. In theory, this would allow you to build an Add-only replica of the S3 bucket, so any DELETEs don't get replicated. There are tutorials for this, but in essence:

  1. Object is operated on in a bucket.
  2. Lambda is triggered, given the event-data for the operation.
  3. If the operation is a PUT, your written code does things with that object. It ignores DELETEs.

The incremental backup logic would be written by you.

4

Note: this is an answer to the original question before it even mentioned offline backups. Leaving it here as an answer for the original question: How to create an Incremental backup of an AWS S3 bucket.

The first question is Why do you want to back up S3 bucket? What's the issue you're trying to protect against?

  • Remember that S3 durability is somewhere around 99.99999% - you're extremely unlikely to lose objects due to HW failure, so we can rule that out.

  • If you want to make sure that accidentally overwritten objects in S3 can be recovered you can use S3 versioning - that will keep a history of all the older versions of the file and you can recover that way. Same for deletions.

  • Speaking of deletions - you can require use of MFA for S3 Deletions as another layer of protection, e.g. for compliance and auditing reasons. (thx Tim :)

  • If you need a second DR (disaster recovery) bucket in some other region for the unlikely event that your primary region goes off-line you can use S3 Cross Region Replication that will automatically mirror your bucket contents from one region to another with every change.

  • If none of the above still satisfies your needs you may want to have a Lambda function that handles each change in the S3 bucket for you. That way every time you write/update an object in S3 the Lambda will make a backup to your preferred destination. This can be used e.g. to mirror S3 buckets between different AWS Accounts, to other cloud providers or to offline destinations (e.g. to your on-prem server). With Lambda you've got the ultimate flexibility on what to do with the changes. See Using Lambda with Amazon S3.

  • If that's still not enough you can always use aws s3 sync that compares the source and destination buckets and copies only what has changed.

  • (Update) For offline backups

    • You can of course use aws s3 sync as well - that can sync to/from disks, not only between buckets.
    • Or you can develop a more elaborate solution based on catching S3 Events when objects are created/updated, and copying them to your offline storage as soon as that happens. It should be a pretty simple program sitting on your server, listening to the S3 events or SNS messages and reaching out to S3 for the updated objects.

That's plenty to choose from. Hope some of it suits your needs :)

  • 1
    You could also mention MFA delete in your answer :) – Tim Nov 5 '18 at 19:02
  • 2
    Risk analysis for DR is a complex subject and well outside the scope of this question.... This answer is broken because it suggests putting all your eggs in one (Amazon) basket. What happens if someone forgets to pay the bill. What happens if one sysadmin believes the whole service is no longer required.... One single mistake destroys everything. THIS IS NOT A BACKUP! – Philip Couling Nov 6 '18 at 12:13
  • 2
    @couling So to wrap ip up: You ask a misleading question, don't follow up when multiple people ask for clarification and then complain that the answer to your misleading question doesn't answer you unspoken intent. Well done :) – MLu Nov 6 '18 at 20:31
  • Updated the answer to clarify – MLu Nov 6 '18 at 20:31
  • As a stand alone answer, this is still broken because it conflates durability and versioning with backups. You effectively suggested that S3 never needs a backup. That's a view point that many disagree with. My question didn't "mislead" you into this view point. – Philip Couling Nov 7 '18 at 9:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.