Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 18 down vote favorite
11

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)

Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and cost. I'm considering using rsync with S3 and came across s3rsync. I was just wondering if anybody had any experience using this, or any other utility?

share|improve this question
1  
One thing I noticed about s3rsync is that you are currently limited to 10GB bucket sizes (check the FAQ). You can have multiple buckets, but you have to split your data into 10GB chunks. – dana Jun 30 '11 at 1:37

6 Answers

up vote 7 down vote

Depending on how your Acronis images are created, I'm not sure any kind of rsync would save you bandwidth. Acronis images are single file(s), so rsync wouldn't be able to read inside them to only back up what changed. Also not sure what kind of server images you're creating, but since you said 100GB I'm going to assume full? An incremental image would cut down on the nightly image size greatly, thus saving bandwidth. You could also consider saving the images to an alternate location than S3, such as tape media, and store that off-site.

share|improve this answer
No, rsync doesn't work like that. It works with any file type and doesn't need any knowledge of the internals of the file its syncing. Instead it compares hashes of chunks of the file and transfers only those chunks that differ. en.wikipedia.org/wiki/Rsync – Alan Donnelly Dec 12 '10 at 2:12
and none of the chucks will match because any small change in the files inside the image will cause the entire file to change due to compression. Even with compression turned off I'm not sure it would rsync well because the files inside the image can change order and it matches on a rolling basis, rather than just finding any chunk the same. – JamesRyan Oct 13 '11 at 11:34
up vote 6 down vote

I've also had good luck with S3cmd and S3sync, both of which are free.

share|improve this answer
up vote 6 down vote

I recently stumbled across this thread on Google and it looks like the landscape has changed a bit since the question was asked. Most of the solutions suggested here are either no longer maintained or have turned commercial.

After some frustrations working with FUSE and some of the other solutions out there, I decided to write my own command-line rsync "clone" for S3 and Google Storage using Python.

You can check out the project on GitHub: http://github.com/seedifferently/boto_rsync

Another project which I was recently made aware of is "duplicity." It looks a little more elaborate and it can be found here: http://duplicity.nongnu.org/

Hope this helps.

share|improve this answer
great contribution! thanks and I will give your code a shot soon. Do you have any must-reads for learning python/django? Cheers – iainlbc Jan 22 '12 at 3:13
What advantages / differences does your program have compared to S3cmd and S3sync? – James McMahon May 3 '12 at 3:00
@JamesMcMahon s3cmd/s3sync are more full-featured CLIs for S3 (manage buckets, list contents, etc), while boto-rsync is strictly an attempt at mimicking rsync. – Seth Jun 19 '12 at 17:10
up vote 4 down vote

I Never tried S3rsync.

I'm using duplicity for our off-site backups. It supports incremental backups on S3 though it is not really saving bandwidth due to Amazon S3 storage protocol in which any file modification forces you to upload the whole new file again. Anyway duplicity only uploads differences from the last incremental backup.

With Duplicity you won't need to go through another server as S3sync does, nonetheless if you encrypt your data it should be worth to give S3sync a try.

share|improve this answer
up vote 1 down vote

S3 also has an add-on service called AWS Import/Export that allows you to send a USB drive with your initial 100Gb data set to them and they'll load it on the S3 cloud using some backend tools at their data centers. Once your 100Gb is up there, you can just do differential backups each night to backup everything that's changed.

The site is http://aws.amazon.com/importexport/

If the majority of your data is fairly static then this would be a good option, if the whole 100Gb of data is changing daily then this is not going to help you much.

share|improve this answer
3  
How do you suppose they "load" a 128Gb flash drive? I picture the world's largest usb hub, a floor to ceiling patch panel of USB connectors, 3/4 full of customer supplied flash drives, all going into the back of a single blade server. – user33903 Feb 16 '10 at 19:39
What an image !! In reality probably some poor guy in a dark corner of a data center with your world's largest usb hub connected to his PC :) – monkeymagic Feb 25 '10 at 18:32
up vote 0 down vote

The new Jungle Disk Server Edition (beta) might be useful to you. It has block-level de-duplication, so if your Acronis images have anything in common, this will greatly reduce the amount you need to backup. The features are perfect for server backups. Check out the release notes.

I've been testing the beta for two weeks and, aside from some small issues with the GUI that I'm sure will be fixed in the final, am excited about the product.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.