Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm currently researching the best backup solution for my CentOS webserver, and I'm thinking of going with either Tarsnap or straight-up Amazon S3.

I am trying to figure out how to discourage a hacker from deleting both my server contents and my remote backups in the worst case where he gains root access to my server and thus has access to the remote backup authentication credentials. (I do fully understand the importance of having a strong password and/or enforcing only key based SSH auth, as well as other general best-practice security. But stuff sometimes happens, there could be a Linux vulnerability or a vulnerability at the VPS level of my host, or something else more or less out of my control.)

I know that both Tarsnap and Amazon S3 have write-only user permissions, but the problem is that I also need automated backup pruning/rotation. Would it be possible to set up either of those services (or possibly some other service) to disallow deletion of backup generations newer than, say, 2 days? That would give me a two day buffer to notice that I was hacked and prevent the hacker from deleting my newest data generations.

Or any other ideas? Thanks a lot!

share|improve this question
Cookies? Money? Positive encouragement? – joeqwerty Dec 15 '13 at 17:54
I'd suggest asking the hacker very nicely not to destroy your remote backups... – ewwhite Dec 15 '13 at 19:46
Write only permissions for the backup user on the S3 bucket? And then set the pruning in S3 by file date. – Fiasco Labs Dec 15 '13 at 23:25
Do your pruning from somewhere else that only has access to your backups but not your main site. – JamesRyan Dec 2 '14 at 12:15
up vote 14 down vote accepted

at a fundamental level, your issue is partly exacerbated by pushing your backups from the host, rather than pulling them off of it.

You can fix that issue, and in addition physically (or logically if needs be) unplug the backup volumes on the central backup host.

I do have machines which push backups to S3, but those S3 buckets use versioning so an attacker pushing a bad backup isn't an issue (and the api keys used don't have rights to delete objects, only add them). I prune old backups with boto as versioning and bucket lifecycle at the same time isn't allowed).

No 'incentive' is going to work, these people don't care about your data.

share|improve this answer
+1 Don't use push backups, use pull. If you can't use pull, the push credentials shouldn't have access to remove data. Pretty simple stuff. – Chris S Dec 15 '13 at 19:45
Definitely there is no 'incentive' for hackers to not destroy your life. By "discourage" I meant "make it a lot more difficult for them", because obviously there's no 100% security. So, pull backups makes sense. Is that possible with Amazon S3? – rahim123 Dec 15 '13 at 22:00
@user12361 - no, but you can use AWS credentials (created via IAM) that only permits those credentials to push backups, and not to change or delete any existing data. – EEAA Dec 15 '13 at 22:37
And then I use different credentials with write/deletion permissions on a different machine that has less attack area to periodically prune backups? – rahim123 Dec 15 '13 at 22:45
yeah. essentially. I have a hardened machine which basically does only that and any other periodic aws jobs like starting/stopping some instances out of business hours to save money. – Sirex Dec 15 '13 at 23:30

Well, I ended up going with Duplicity since it has its own Amazon S3 interface. I created an Amazon S3 user via IAM with limited permissions just for GETting and PUTting objects. I'm using the --full-if-older-than Duplicity option to create a new full backup every 7 days. Then I have an automatic S3 lifecycle policy in place that moves objects older than 10 days to Glacier. Then a secondary rule deletes objects that are older that 102 days from Glacier (I added a little extra slop just to make sure I don't get hit with Glacier early deletion fees before 90 days). So this will give me 80+ days of backups in Glacier (once the oldest full backup is deleted its child incrementals will still exist for a few days more but won't be valid anymore), and another 10 days of the newest backups in S3. With differential updates and compression my daily backups are quite small in size, so it should be extremely economical.

Thanks to everyone for their help and suggestions.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.