I'm looking for a way to take daily backups of an AWS bucket as incremental backups. These are to be stored offline and away from AWS.
For other storage systems (such as NAS drives) I use a daily rsync for backups. Using rsync's
--link-dest switch, I'm able to take a full snapshot every day of the remote file system. Any files which have not changed since the previous backup are hardlinked to the previous backup. This means that full daily snapshots only take the storage space of incremental backups.
I would like to setup something similar for an amazon S3 bucket. There's 20GB in the bucket but only ~50MB changes per day.
Note this is backup the content of an S3 bucket, NOT backup other content to S3 bucket.
I can see how I would use the AWS CLI tools to do full backups. I don't see how I can do incremental backups.
I guess I could (daily) synchronise S3 to a local hard drive, then daily backup the local hard drive. This feels very clunky.
This was intended as a simple technical question, not a general discussion of backup security. But since I'm being asked "why do you need this", I now see I need to explain basic principles of backups.
Anecdote: I recently witnessed a third party IT provider drop (entirely) an S3 bucket because of a miscommunication. This could have been very costly (~£100K of recent work, ~£1M total work). Luckily we happened to also have copies on our local laptops and for only £1K we rebuilt the content for them.
It has renewed my conviction that the only valid "backup" is on an isolated system stored offsite and offline, and with a media rotation that effectively implements a time lock. Other backups can enhance, providing more rapid recovery etc... but holding all your AWS backups on your own AWS account just isn't safe because ... user error.