2

Note: Backing up data stored on Amazon S3 is similar but quite old, and doesn't address common practice.

Our service involves each user upload a couple of files. There are hundreds of thousands of users which will keep increasing.

We use AWS for our infrastructure, and currently all files are uploaded and stored directly in S3. The top-level folder is about 75 GB and growing.

For a long time we've had a script that does a nightly backup by copying each file via the S3 API into another folder. Last few months the storage cost became huge, so we switched to storing just bi-weekly backups.

The reason for keeping backups is that the business depends on the user uploaded data. Think of Dropbox or Flickr or Giphy.

We're not worried about S3 losing the data but rather due to human error, which is more likely no matter how many precautions we take. In such cases, we should be able to restore data to a certain point in time.

However, the S3 backup strategy seems dubious since the backups don't seem to be happening correctly. The size of backups seems to be very low sometimes, implying that S3 copy operations don't behave - even using the S3 web console to copy-paste a folder with a large number of files doesn't work properly and hangs after partially completing, makes us even more suspicious of large copy operations on S3.

We are aware of S3 object versioning but it doesn't seem like the right solution for a backup to mitigate the effects of human error like deleting the bucket.

What backup strategy for files do businesses that depend on user uploaded files take?

  • 1
    Have you tested a restore to a (different, test) location? Backups are pointless without restores. – John Mahowald Jan 7 '18 at 14:38
  • What is your RTO / RPO? Recovery Time Objective / Recovery Point Objective. If you don't understand the terms please Google them, they're widely used. – Tim Jan 7 '18 at 19:19
  • @JohnMahowald yes we have tested restores though not the complete sanity of the data - that is beyond our current priority. – Aditya Jan 8 '18 at 15:49
  • @Tim for us RTO / RPO is not strictly defined because the data is not critical i.e. we basically need an RPO of "about a couple of weeks" and RTO of "a few days" and ideally recovery rate of at least 90% i.e. some data can be lost. Cost is also a factor. – Aditya Jan 8 '18 at 15:54
2

Typical strategies include:

  1. Enabling bucket versioning, and
  2. Enabling MFA-delete.

Enabling bucket versioning will preserve file versions in the case of overwrites and deletes. If this happens, you can restore a previous version of the file.

Enabling MFA-delete would require someone to use an MFA key each time they want to delete files. Accidental delete is very difficult in this case.

Deleting a bucket is not possible as long as the bucket contains data. So in order to delete a bucket, someone would need to delete all the files first.

You could go stronger by adding a bucket policy to your bucket preventing the s3:DeleteObject command. By doing this deleting objects is prohibited at the bucket level; no one would be able to delete objects, even if they had permissions and an MFA device.

One more thing you could do is duplicate your files into another bucket. Using lifecycle rules, your objects can be duplicated into another bucket in another region automatically, daily.

TL;DR, bucket versioning and MFA delete are usually enough to prevent accidental data loss. Only a malicious user really could destroy the data. But you could protect your bucket more with a bucket policy as well.

  • Fantastic, thanks! Better than any other strategy I've come across. Probably an infrequent backup is still nice to have. Somehow, I can't imaging businesses that depend on user data not having any backup at all regardless of how many replicas they maintain. – Aditya Jan 8 '18 at 18:49
1

I would never trust all my business critical information to one provider or location. S3 keeps files in multiple locations with a region, which protects against hardware failure only.

I have a few ideas.

  1. Cross Region Replication. Make sure the bucket in the other region has versioning enabled, and make sure virtually no-one has write / delete access.
  2. S3 sync to a server. This server could be running in another Cloud (Azure / Google), and have it store that data on the other cloud's object store. Alternately it could be an on-premise or hosted server, and that server could run incremental backups.
  3. Use a provider that will sync between clouds for you. I believe Google has a service that will do this, Azure might as well.
  4. There's probably a way to do incremental backups, with storage in EBS, Glacier, or another S3 bucket.

If I had time to think more I could probably come up with more options. Your RTO and RPO will really drive the solution.

  • Thanks for the info. Conceptually, replication is not a solution for deletions due to human error, because the deletion gets replicated. Sync is also not an option for us even if there's an option to prevent deletes, because there needs to be a point-in-time backup. – Aditya Jan 8 '18 at 15:58
  • Sync to a version controlled bucket that no-one has delete access to does mitigate many human error scenarios. You can just undo the delete or fetch data as it was at a point in time. It's also potentially the cheapest solution. Sync lets you get your files to a file system where you can do a point in time backup. – Tim Jan 8 '18 at 18:10
  • Sync is indeed quite likely the cheapest solution. Will explore it further, thanks. – Aditya Jan 9 '18 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.