How to discourage a hacker with root access from deleting remote backups?

Question

I'm currently researching the best backup solution for my CentOS webserver, and I'm thinking of going with either Tarsnap or straight-up Amazon S3.

I am trying to figure out how to discourage a hacker from deleting both my server contents and my remote backups in the worst case where he gains root access to my server and thus has access to the remote backup authentication credentials. (I do fully understand the importance of having a strong password and/or enforcing only key based SSH auth, as well as other general best-practice security. But stuff sometimes happens, there could be a Linux vulnerability or a vulnerability at the VPS level of my host, or something else more or less out of my control.)

I know that both Tarsnap and Amazon S3 have write-only user permissions, but the problem is that I also need automated backup pruning/rotation. Would it be possible to set up either of those services (or possibly some other service) to disallow deletion of backup generations newer than, say, 2 days? That would give me a two day buffer to notice that I was hacked and prevent the hacker from deleting my newest data generations.

Or any other ideas? Thanks a lot!

Answer 1

at a fundamental level, your issue is partly exacerbated by pushing your backups from the host, rather than pulling them off of it.

You can fix that issue, and in addition physically (or logically if needs be) unplug the backup volumes on the central backup host.

I do have machines which push backups to S3, but those S3 buckets use versioning so an attacker pushing a bad backup isn't an issue (and the api keys used don't have rights to delete objects, only add them). I prune old backups with boto as versioning and bucket lifecycle at the same time isn't allowed).

No 'incentive' is going to work, these people don't care about your data.

Answer 2

Well, I ended up going with Duplicity since it has its own Amazon S3 interface. I created an Amazon S3 user via IAM with limited permissions just for GETting and PUTting objects. I'm using the --full-if-older-than Duplicity option to create a new full backup every 7 days. Then I have an automatic S3 lifecycle policy in place that moves objects older than 10 days to Glacier. Then a secondary rule deletes objects that are older that 102 days from Glacier (I added a little extra slop just to make sure I don't get hit with Glacier early deletion fees before 90 days). So this will give me 80+ days of backups in Glacier (once the oldest full backup is deleted its child incrementals will still exist for a few days more but won't be valid anymore), and another 10 days of the newest backups in S3. With differential updates and compression my daily backups are quite small in size, so it should be extremely economical.

Thanks to everyone for their help and suggestions.