8

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the data backed up.

I can mirror to a local disk with the s3cmd sync function, but that does not scale for very large buckets and is not useful for quick backup restores. I'd rather have my data mirrored to a competitor like Rackspace Cloud Files.

Anyone have some suggestions on a simple and robust way to facilitate this kind of mirroring in an automated way on a Linux box?

  • Am I correct to assume that non of the answers actually solves "1) I don't want all my data only existing with one provider", because they all just mirror buckets within the same provider? – Chris Nov 23 '18 at 9:11
6

You can use the "s3cmd" utility with the "sync" option, although I stumbled on your question because I'm trying to figure out if this syncing mechanism is screwing up my duplicity backups.

5

I was having the same problem so I whipped up a little program specifically designed to mirror one S3 bucket to another; I call it s3s3mirror.

I did try the "s3cmd sync" approach first, but I had a bucket with hundreds of thousands of objects in it, and "s3cmd sync" just sat there, not doing anything but consuming more and more memory until my system died. I designed s3s3mirror to get going immediately, to use 100 concurrent threads (configurable), and to make modest use of CPU and memory. If I do say so myself, it's pretty freakin' fast.

I've made this available on github under an Apache License. If you decide to give it a whirl please let me know what you think and if there's anything that can be improved.

Here's the link: https://github.com/cobbzilla/s3s3mirror

thanks!

  • jonathan.
3

Amazon now has a supported tool to do this, the aws cli.

It can mirror in either direction between local and remote, or between two s3 locations.

Unfortunately it doesn't have any direct support for non-S3 locations like rackspace, but I thought this would be a useful answer for some who find this question. Like me, before I found it.

In particular,

aws s3 sync s3://some/s3/path /some/local/path
2

Check out Jungle Disk Server. It works with both Amazon S3 and Rackspace Cloud Files. You could mount S3 and CF in different locations on your filesystem and then use rsync between the two.

  • Thanks, but I should have mentioned that I'm looking for a solution I can run as a cron job on a Linux box. I've updated the question. – Ben Sep 29 '10 at 12:16
2

You could try mounting the buckets with fuse using s3fs. Once that's done, you can then rsync from the mount point to your local disk.

  • 1
    +1 for s3fs, I just committed a patch to support files up to 5TB in size. – ben lemasurier Aug 30 '11 at 17:58
1

You can setup Minio server, open source and S3 compatible API. You can set it up on EC2 or on any cloud provider as your alternative server and using Minio clients mc mirror command take mirror of S3 periodically. You can automate the same writing a simple cron script.

Hope it helps. Disclaimer: I work for Minio

0

For online synchronization of buckets you can use cross-region replication. This way all of updates to original bucket will be automatically synchronized to S3 bucket in another region: https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html

Keep in mind that it requires versioning enabled for the bucket so if you do updates/deletes of objects in buckets much you'll want to enable lifecycle policy to delete old objects.

protected by Chris S May 21 '13 at 15:50

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.