I have some (virtual) servers. They backup themselves on Backblaze B2 using an application key.

I want to hire a person to help me administrate these servers.

When that person gets root access to the server (which I want), he also gets access to the B2 application key. This means he can delete the server and the backup.

What is a procedure/setup I could use to guard against this edge case? I do have offline backups, but these are monthly, whereas the B2 backups are daily.

  • 1
    In many ways, this is a byproduct of how the B2 application works. A pull-based backup generally wouldn't have this problem. – Zoredache Jan 30 '20 at 0:17

In some envs, you simply split the job into two teams - one team runs/guards the servers, the other runs/guards the backups.

the other part is, imo even more, important: if the backup is writeable, especially from the system that is being backed up, it is NOT a good backup.

systemic issue in many tools like borg. it is best if you can (in a cloud world) push the backups forward to AWS glacier, while storing them into S3 with a role that can only create them. there's schedules for deletion and noone needs to be able to "quickly" do that.

also, don't forget that there's books about this stuff.

  • Hmmm, I can configure B2 so that an application key isn't allowed to delete files. It does need some more changes in the setup (because right now duplicity deletes old backup files, which will no longer be possible), put I think this is in the right direction, thanks! BTW, can you recommend a book? If not, could you tell me what search term I should be using? – Antonis Christofides Jan 30 '20 at 7:11
  • @AntonisChristofides i can't check the exact title during the next days b/c the book is in the shelf at work and i'm travelling, but generally look for "data protection" instead of "backup" – Florian Heigl Jan 31 '20 at 11:16

Here's how to do it:

  1. On Backblaze B2, create an application key that cannot delete files:

    b2 create-key --bucket MyBucket MyKeyName listBuckets,listFiles,readFiles,writeFiles
  2. Setup the backup so that it uses that key and does not attempt to delete old backup files. For example, in duplicity, do not use remove-older-than, remove-all-but-n-full, or remove-all-inc-of-but-n-full; in duply, do not use purge or purgeIncr.

  3. To delete old backups, set custom lifecycle settings on the bucket; for example, set lines beginning with duplicity-full to be hidden after 360 days and deleted after another 360; and so on for files beginning with duplicity-inc and duplicity-new.


B2 does not actually provide any functionality to "delete a file". Each time you replace the same file, it keeps its history, so a file can have "versions"—the most recent one is normally the one you need. What B2 provides is functionality for "hiding" a file. When you hide a file, you are actually recording in the file's history that the file was "deleted", you kind of add a new file version which is the hidden, or deleted, file.

Except for that, B2 also provides functionality to actually delete file versions. A user who does not have the deleteFiles permission actually does have permission to hide files, but not to delete file versions.

It seems to me that duplicity's remove-... functionality should be implemented by hiding files. (It would then be up to the bucket's configuration to actually delete these hidden file versions after some amount of time.) However duplicity's B2's backend doesn't do this; what it does is to actually delete file versions.


On most public clouds, ssh access to the server does not mean you can delete the server. There is a difference between administering on the server and in the cloud account. So, you can separate these. You can also split out permissions quite a bit, especially on Amazon AWS. You can give someone the permission to reboot the server, but not delete it, for example. And AWS has the delete protect option on VMs.

Best practice, these days, is to use configuration management like Chef/Puppet/Salt/Ansible to do any changes on your servers and not actually log into the servers.

In addition, for AWS, you can snapshot the servers as often as you want and this is the most efficient way to back them up.

  • 1
    When I say delete the server I don't mean delete the VM. You can rm -rf /. In fact I'm using Ansible to administrate the servers, and even if the admin can't ssh into the server, what prevents him from adding an rm -rf / task to the playbook? – Antonis Christofides Jan 30 '20 at 7:04
  • Then you have other problems. Don't give your user root access if you don't trust him/her with that power. Often you can scope out what it is you expect this adminstrator to do and then give isolated power to do only that. I would probably focus on hiring someone I could trust. – Kevin Buchs Jan 30 '20 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.