Does librsync really reduce bandwidth for Amazon S3

Question

I am looking to do some backups to S3 and want to reduce the bandwidth as much as possible. I am looking at a few options. It seems like librsync is the best solution for low bandwidth remote backups.

I've been reading on how librsync works. And it seems like the remote end needs to calculate the checksum on the blocks of the files it is comparing (as well as the local end). I would assume that S3 can't do those checksum calculations since it only serves files.

I've also read that S3 doesn't support separating files into chunks. It can only offer the whole file or nothing.

If both (or either) of these statements are true, would librsync be essentially useless? Can someone shed some light on this for me?

Thanks.

Answer 1

I think librsync is just an implementation of the algorithm. It can be used in multiple ways. The "normal" usage pattern, like used in the original rsync program, does expect recipient to support remote hash generation. Duplicity also uses librsync, but it pre-calculates the hashes instead, and so does not need remote support beyond file storage.

More info: http://en.wikipedia.org/wiki/Rsync. Specifically:

duplicity is a variation on rdiff-backup that allows for backups without cooperation from the storage server, as with simple storage services like Amazon S3. It works by generating the hashes for each block in advance, encrypting them, and storing them on the server, then retrieving them when doing an incremental backup. The rest of the data is also stored encrypted for security purposes.

I'm still researching as well, but minimally if using Duplicity, disabling SSL (s3-unencrypted-connection) and increasing volsize parameter should help conserve bandwidth.