Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to be no tool to move whole bucket content to different bucket.

There is also problem on how to synchronize the data later on when we switch to EU bucket, the data that will be created meanwhile while the migration was running.

share|improve this question

4 Answers 4

I guess, you can run:

s3cmd --recursive cp s3://oldbucket/ s3://newbucket

It should copy it directly.

share|improve this answer

I don't know of a packaged solution to do this, but it is essentially just a bucket LIST and then PUT-copy each object from the US bucket to the EU bucket. The PUT-copy API supports "if-modified-since" and "if-none-match", so you can run it repeatedly without duplicating effort: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/index.html?RESTObjectCOPY.html

For a live migration, I would suggest the following approach:

  1. Optional first step: If you have a huge amount of data to migrate, consider using AWS Import/Export to capture the first snapshot of the data on a physical device in the US and ship it physically to the EU. This may save you time and money.
    • Set up a script to migrate data from US to EU using the if-modified-since or if-none-match as described above. Make it smart enough not to clobber fresh EU data with stale US data (once you get to step 5 below). Run it continuously in a loop until the two buckets are in roughly the same state.
    • Modify your web app to PUT data to both the US and EU buckets at the same time, while continuing to GET data exclusively from the US.
    • Continue running your migration script until you are confident that the buckets are in exactly the same state (or as close as you can get, given eventual consistency).
    • Modify the web app again to GET data exclusively from the EU bucket.
    • You can now turn off your migration script.
    • Look for eventual consistency anomalies and clean them up.
    • Modify the web app to stop PUTting data to the US bucket.
    • DELETE everything in the US bucket.

You may want to use S3 Reduced Redundancy Storage on your EU bucket during the migration to get cheaper data rates and faster response times, since the data is just a duplicate of the US data.

share|improve this answer

The new official AWS CLI natively supports most of the functionality of s3cmd.

http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

aws s3 sync s3://oldbucket s3://newbucket
share|improve this answer

I've done a fair amount of work with S3 and have never come across a tool that will do this natively. However, it wouldn't be all that difficult to use a couple of s3cmd-sync sync scripts to keep two buckets in sync with each other.

share|improve this answer
    
yeah, that would work, but require us to download everything locally, while S3 API supports moving/copying entries without downloading them. –  skrat Apr 21 '10 at 9:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.