What is a practical way to mirror an Amazon S3 bucket?

Question

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the data backed up.

I can mirror to a local disk with the s3cmd sync function, but that does not scale for very large buckets and is not useful for quick backup restores. I'd rather have my data mirrored to a competitor like Rackspace Cloud Files.

Anyone have some suggestions on a simple and robust way to facilitate this kind of mirroring in an automated way on a Linux box?

Answer 1

Check out Jungle Disk Server. It works with both Amazon S3 and Rackspace Cloud Files. You could mount S3 and CF in different locations on your filesystem and then use rsync between the two.

Answer 2

You can use the "s3cmd" utility with the "sync" option, although I stumbled on your question because I'm trying to figure out if this syncing mechanism is screwing up my duplicity backups.

Answer 3

You could try mounting the buckets with fuse using s3fs. Once that's done, you can then rsync from the mount point to your local disk.