1

I started running rclone to keep a set of files on my laptop that are synced up to the cloud and encrypted. I wanted to 'control' the encryption because of trust issues. Rclone does this pretty well, and so well that I use it for a lot of my stuff, but that means that the number of files in there is getting bigger and that is costing me money.

I thought about alternatives to this, like syncing just the difference in a zip or an rdiff file or something. It is clear that this could work but it has some disadvantages or hurdles to manage.

First of all, the S3 storage isn't a real rsync server in the sense that it does not only copy the part of the file that changed. If I want to change to a system that sends a blob up to the cloud it needs the intelligence to know what is already up there.

The features I am looking for are:

  • Minimal connection load for backups,
  • Ability to quickly restore when I stupidly delete a file.
  • Low number of files stored on server
  • Full or near full file history
  • Something I can make a script for that takes the load off my brain to figure out all the options I need to include when syncing.
  • Ideally something that works with multiple devices although I don't use multiple devices simultaneously.
  • I should control the encryption (done locally) and keys

I'm posting this here because I thought this question would produce a worthwhile answer for a lot of people with server-type usage requirements.

Thanks in advance

[EDIT: Added that I want full control of encryption keys and encryption should be done locally before transfer]

Improve this question

1 Answer 1

Reset to default
1

I'd recommend to look for backup software that supports S3, and almost all of them are.

Two following I've used in the past or using now - with Google's GCS but they also support S3:

  • Duplicity allows incremental backups, although requires full backup once in a while
  • Duplicacy is a fancy once and allows deduplication across machines. Although it seemed a bit less stable for me, i.e. sometimes my snapshots will be missing a few files.
Improve this answer
3
  • I just added that I wanted to control encryption and do it locally. Looks like Duplicity meets that requirement. Have you found it reliable?
    – Not a chance
    Jan 2, 2019 at 12:30
  • Other comment: Given Amazon S3 won't allow real rsync transfers (i.e. transfering only the part of the file that has changed..) is this really efficient?
    – Not a chance
    Jan 2, 2019 at 12:34
  • Yes, duplicity is quite reliable. It works in different way to rsync: it first uploads a full copy of your data, and an index of what's in it. In subsequent runs it scans the index, and the local filesystem and only uploads the difference in incremental backup. So it's quite efficient, however it requires a full backup once in a while, otherwise restore chain becomes too long.
    – rvs
    Jan 2, 2019 at 13:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .