Backup strategy for user uploaded files

Question

Note: Backing up data stored on Amazon S3 is similar but quite old, and doesn't address common practice.

Our service involves each user upload a couple of files. There are hundreds of thousands of users which will keep increasing.

We use AWS for our infrastructure, and currently all files are uploaded and stored directly in S3. The top-level folder is about 75 GB and growing.

For a long time we've had a script that does a nightly backup by copying each file via the S3 API into another folder. Last few months the storage cost became huge, so we switched to storing just bi-weekly backups.

The reason for keeping backups is that the business depends on the user uploaded data. Think of Dropbox or Flickr or Giphy.

We're not worried about S3 losing the data but rather due to human error, which is more likely no matter how many precautions we take. In such cases, we should be able to restore data to a certain point in time.

However, the S3 backup strategy seems dubious since the backups don't seem to be happening correctly. The size of backups seems to be very low sometimes, implying that S3 copy operations don't behave - even using the S3 web console to copy-paste a folder with a large number of files doesn't work properly and hangs after partially completing, makes us even more suspicious of large copy operations on S3.

We are aware of S3 object versioning but it doesn't seem like the right solution for a backup to mitigate the effects of human error like deleting the bucket.

What backup strategy for files do businesses that depend on user uploaded files take?

Answer 1

Typical strategies include:

1. Enabling bucket versioning, and
2. Enabling MFA-delete.

Enabling bucket versioning will preserve file versions in the case of overwrites and deletes. If this happens, you can restore a previous version of the file.

Enabling MFA-delete would require someone to use an MFA key each time they want to delete files. Accidental delete is very difficult in this case.

Deleting a bucket is not possible as long as the bucket contains data. So in order to delete a bucket, someone would need to delete all the files first.

You could go stronger by adding a bucket policy to your bucket preventing the s3:DeleteObject command. By doing this deleting objects is prohibited at the bucket level; no one would be able to delete objects, even if they had permissions and an MFA device.

One more thing you could do is duplicate your files into another bucket. Using lifecycle rules, your objects can be duplicated into another bucket in another region automatically, daily.

TL;DR, bucket versioning and MFA delete are usually enough to prevent accidental data loss. Only a malicious user really could destroy the data. But you could protect your bucket more with a bucket policy as well.

Answer 2

I would never trust all my business critical information to one provider or location. S3 keeps files in multiple locations with a region, which protects against hardware failure only.

I have a few ideas.

1. Cross Region Replication. Make sure the bucket in the other region has versioning enabled, and make sure virtually no-one has write / delete access.
2. S3 sync to a server. This server could be running in another Cloud (Azure / Google), and have it store that data on the other cloud's object store. Alternately it could be an on-premise or hosted server, and that server could run incremental backups.
3. Use a provider that will sync between clouds for you. I believe Google has a service that will do this, Azure might as well.
4. There's probably a way to do incremental backups, with storage in EBS, Glacier, or another S3 bucket.

If I had time to think more I could probably come up with more options. Your RTO and RPO will really drive the solution.