Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am interested in using Amazon S3 to backup our ~ 100gb server images (created via Acronis backup tools)

Obviously, this uploading to S3 every night would be expensive, in terms of bandwidth and cost. I'm considering using rsync with S3 and came across s3rsync. I was just wondering if anybody had any experience using this, or any other utility?

share|improve this question
One thing I noticed about s3rsync is that you are currently limited to 10GB bucket sizes (check the FAQ). You can have multiple buckets, but you have to split your data into 10GB chunks. – dana Jun 30 '11 at 1:37

I recently stumbled across this thread on Google and it looks like the landscape has changed a bit since the question was asked. Most of the solutions suggested here are either no longer maintained or have turned commercial.

After some frustrations working with FUSE and some of the other solutions out there, I decided to write my own command-line rsync "clone" for S3 and Google Storage using Python.

You can check out the project on GitHub:

Another project which I was recently made aware of is "duplicity." It looks a little more elaborate and it can be found here:

Hope this helps.


The Python team at AWS has been working hard on a boto-based CLI project for their cloud services. Among the tools included is an interface for S3 which duplicates (and in many ways supersedes) most of the functionality provided by boto-rsync:

In particular, the sync command can be configured to function almost exactly like rsync:

share|improve this answer
great contribution! thanks and I will give your code a shot soon. Do you have any must-reads for learning python/django? Cheers – iainlbc Jan 22 '12 at 3:13
What advantages / differences does your program have compared to S3cmd and S3sync? – James McMahon May 3 '12 at 3:00
@JamesMcMahon s3cmd/s3sync are more full-featured CLIs for S3 (manage buckets, list contents, etc), while boto-rsync is strictly an attempt at mimicking rsync. – Seth Jun 19 '12 at 17:10

I've also had good luck with S3cmd and S3sync, both of which are free.

share|improve this answer
+1 for S3cmd -.- – fnkr Jan 24 '14 at 13:11

Depending on how your Acronis images are created, I'm not sure any kind of rsync would save you bandwidth. Acronis images are single file(s), so rsync wouldn't be able to read inside them to only back up what changed. Also not sure what kind of server images you're creating, but since you said 100GB I'm going to assume full? An incremental image would cut down on the nightly image size greatly, thus saving bandwidth. You could also consider saving the images to an alternate location than S3, such as tape media, and store that off-site.

share|improve this answer
No, rsync doesn't work like that. It works with any file type and doesn't need any knowledge of the internals of the file its syncing. Instead it compares hashes of chunks of the file and transfers only those chunks that differ. – Alan Donnelly Dec 12 '10 at 2:12
and none of the chucks will match because any small change in the files inside the image will cause the entire file to change due to compression. Even with compression turned off I'm not sure it would rsync well because the files inside the image can change order and it matches on a rolling basis, rather than just finding any chunk the same. – JamesRyan Oct 13 '11 at 11:34

I Never tried S3rsync.

I'm using duplicity for our off-site backups. It supports incremental backups on S3 though it is not really saving bandwidth due to Amazon S3 storage protocol in which any file modification forces you to upload the whole new file again. Anyway duplicity only uploads differences from the last incremental backup.

With Duplicity you won't need to go through another server as S3sync does, nonetheless if you encrypt your data it should be worth to give S3sync a try.

share|improve this answer

S3 also has an add-on service called AWS Import/Export that allows you to send a USB drive with your initial 100Gb data set to them and they'll load it on the S3 cloud using some backend tools at their data centers. Once your 100Gb is up there, you can just do differential backups each night to backup everything that's changed.

The site is

If the majority of your data is fairly static then this would be a good option, if the whole 100Gb of data is changing daily then this is not going to help you much.

share|improve this answer
How do you suppose they "load" a 128Gb flash drive? I picture the world's largest usb hub, a floor to ceiling patch panel of USB connectors, 3/4 full of customer supplied flash drives, all going into the back of a single blade server. – Paul Feb 16 '10 at 19:39
What an image !! In reality probably some poor guy in a dark corner of a data center with your world's largest usb hub connected to his PC :) – monkeymagic Feb 25 '10 at 18:32

The new Jungle Disk Server Edition (beta) might be useful to you. It has block-level de-duplication, so if your Acronis images have anything in common, this will greatly reduce the amount you need to backup. The features are perfect for server backups. Check out the release notes.

I've been testing the beta for two weeks and, aside from some small issues with the GUI that I'm sure will be fixed in the final, am excited about the product.

share|improve this answer

You can try minio client aka "mc". mc provides minimal tools to work with Amazon S3 compatible cloud storage and filesystems.

mc implements the following commands

  ls        List files and folders.
  mb        Make a bucket or folder.
  cat       Display contents of a file.
  pipe      Write contents of stdin to one or more targets. When no target is specified, it writes to stdout.
  share     Generate URL for sharing.
  cp        Copy one or more objects to a target.
  mirror    Mirror folders recursively from a single source to many destinations.
  diff      Compute differences between two folders.
  rm        Remove file or bucket [WARNING: Use with care].
  access    Manage bucket access permissions.
  session   Manage saved sessions of cp and mirror operations.
  config    Manage configuration file.
  update    Check for a new software update.
  version   Print version.

You can use mirror command to do your operation. "localdir" being local directory & S3[alias for Amazon S3] and "remoteDir" name of your bucket on S3.

$ mc mirror localdir/ S3/remoteDir

You can also write a cronjob for the same. Also in case of network outrage you can anyways use "$mc session" to restart the upload from that particular time.

PS: I contribute to minio project & would love to get your feedback & contribution. Hope it helps.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.