Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am looking to do some backups to S3 and want to reduce the bandwidth as much as possible. I am looking at a few options. It seems like librsync is the best solution for low bandwidth remote backups.

I've been reading on how librsync works. And it seems like the remote end needs to calculate the checksum on the blocks of the files it is comparing (as well as the local end). I would assume that S3 can't do those checksum calculations since it only serves files.

I've also read that S3 doesn't support separating files into chunks. It can only offer the whole file or nothing.

If both (or either) of these statements are true, would librsync be essentially useless? Can someone shed some light on this for me?

Thanks.

share|improve this question
add comment

1 Answer

I think librsync is just an implementation of the algorithm. It can be used in multiple ways. The "normal" usage pattern, like used in the original rsync program, does expect recipient to support remote hash generation. Duplicity also uses librsync, but it pre-calculates the hashes instead, and so does not need remote support beyond file storage.

More info: http://en.wikipedia.org/wiki/Rsync. Specifically:

duplicity is a variation on rdiff-backup that allows for backups without cooperation from the storage server, as with simple storage services like Amazon S3. It works by generating the hashes for each block in advance, encrypting them, and storing them on the server, then retrieving them when doing an incremental backup. The rest of the data is also stored encrypted for security purposes.

I'm still researching as well, but minimally if using Duplicity, disabling SSL (s3-unencrypted-connection) and increasing volsize parameter should help conserve bandwidth.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.