Secure offsite backup, even in the case of hacker root access

Question

I'm looking for a way to implement a more secure way of doing an offsite backup that will also protect my data against the situation where a malicious hacker has gained root access to my server. Even though the chance of that happening is smaller than other kinds of risks if SSH and password security is properly set up and the system is kept properly up-to-date, the amount of damage that can be permanently done is really high and therefore I'd like to find a solution to limit that.

I've already tried two ways of offsite backups:

• a simple root-writable webdav mount (and configured in fstab) where the backed up data is copied onto. Problem: not really an offsite backup because the connection - and moreover access - to the offsite location is constantly left open as a folder in the filesystem. This is sufficient protection against many kinds of attacks if the mount has limited access privileges (read root only access), but doesn't protect against a malicious person with root access.

• Borg backup through SSH with key authentication. Problem: connection to that offsite server can be done with the key that's stored on the host if the malicious user has root access to the host.

As a solution I'm thinking about these potential ways, but I don't know how and with what:

• Backups can only be written or appended to the destination but not deleted.
• The use of backup software that handles the offsite backups and doesn't support mass deletion of the offsite backups from the first host.

Solutions that aren't really interesting in my situation:

• An extra backup job on the offsite host which transfers them to a location that isn't accessible by the first host (due to technical limitations).

Can anyone give advice on how to implement a proper offsite backup for my case?

Answer 1

All your suggestions currently have one thing in common: the backup source does the backup and has access to the backup destination. Whether you mount the location or use tools like SSH or rsync, the source system somehow has access to the backup. Therefore, a compromise on the server might compromise your backups, too.

What if the backup solution has access to the server, instead? The backup system can do its job with a read-only access, so any compromise on the backup system wouldn't probably compromise the server. Also, the backup system could be dedicated for that purpose alone, making the contents of the backup the only attack vector. That would be very unlikely and need a really sophisticated attack.

To avoid overwriting the backups with tampered or damaged content, do incremental backups that allows you to restore any previous state within the restoration period defined.

Answer 2

Immutable Storage

One good option is to make your backup storage immutable, or at least provide reliable versioning which gives you effectively immutability. To be clear: immutable means unable to be changed, or permanent.

There are multiple services that can do this for you. AWS S3, BackBlaze B2, and I suspect Azure and Google both offer a similar service. You could probably set up a server to do this, but I'm not sure how.

When you have an immutable / version controlled repository you can restore your backup to any point, so if your host is compromised you can still restore as at any point in time.

*AWS S3**

I'm most familiar with AWS S3. S3 provides versioned, encrypted storage, with a high level of durability.

S3 supports versioning, which gives you effective immutability. You can choose to use lifecycle rules to delete old version of files after a time period you can configure. You can also archive versions to cold storage (glacier cold archive), which costs about $1/TB/month.

You can use the intelligent storage tiering class to reduce costs. I choose to use a lifecycle rule to move all of the static data to infrequent access class, which is durable and moderate (hot) performance but doesn't have the scalability or performance of S3 standard.

S3 uses IAM (identity access management, i.e. user management) users and policies. This gives you very granular control of what your backup software can do with your storage. You can give the backup user permission for uploads but deny update and delete. You can also require multi-factor authentication to delete files, or even provide an object lock so that files can't be deleted.

Suggested Software

I create incremental backups using Restic. Restic uploads the new files to your storage location. I believe (but I could be incorrect) that it creates new files, but in general operation it doesn't update or delete any files.

Borg is another option. I used to use Borg, but I found that with my moderate sized backups of hundreds of MB it effectively uploaded all my data each day from EC2 to S3. To me this isn't incremental, so I stopped using it. I did find documentation on this, but don't have the link.

There are dozens of pieces of software that can upload to cloud storage.

Protected Storage

With some backup software you could try giving the IAM user permission to write new files but not update existing files. It's easy to make this restriction with AWS IAM, but as per the comment below Restic will not work with those permissions. You can also have multi-factor authentication required for deleting files from S3.

You could have another IAM user, run from say your PC, which does the periodic clean scrub of the archive, discarding versions as per the policy you set.

Answer 3

Borg Backup supports append-only remote repositories. Any compromise of the server being backed up can result only in creating new backups, not overwriting only old ones.

Answer 4

Solutions that aren't really interesting in my situation:

An extra backup job on the offsite host which transfers them to a location that isn't accessible by the first host.

The fundamental problem is that if you can remotely access your backups then so can the hacker.

(Due to technical limitation)

Technical limitations are made to be overcome.

Can anyone give advice on how to implement a proper offsite backup for my case?

Tape drives have been providing secure, off-site protection against all sorts of disasters -- including hackers -- for almost 70 years.

Answer 5

You can use storage services like AWS S3 (or probably Google's or Azure's equivalent) where you can give your root account PUT permissions to your bucket but not DELETE permissions. That way, you can use a push model and the attacker won't be able to delete the backup.

There are further security measures that you can take with AWS, like requiring MFA to perform DELETEs on the bucket, but allow PUTs and GETs without MFA. That way, you can both back your data up and retrieve it to restore your services without using your MFA device, which might be useful in some extreme (and probably too obscure to even mention) case where accessing the MFA device could compromise it.

Also, an out of scope comment you might find interesting/useful, there are several ways to configure S3 and similar services for automatic failover in case the main data source is offline.

Answer 6

Borg backup through SSH with key authentication. Problem: connection to that offsite server can be done with the key that's store on the host if the malicious user has root access to the host.

You can use option command in your authorized_keys. You fix the command allowed in remote.

how to add commands in ssh authorized_keys

Even if an attacker recovers the login root, he will not be able to do anything other than the defined command.

Answer 7

A technique you could set up is using syncthing between your server and a remote backup server, and letting the remote backup server do snapshots or whatever on its end so that erasure server side doesn't result in erasure offsite.