26

We use S3 for storing millions of entries in our webapp, now we move the whole thing to EC2, EU servers, and we also want to move that S3 data to EU. But the bucket we use is in US, and there seem to be no tool to move whole bucket content to different bucket.

There is also problem on how to synchronize the data later on when we switch to EU bucket, the data that will be created meanwhile while the migration was running.

5 Answers 5

24

The new official AWS CLI natively supports most of the functionality of s3cmd.

http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

aws s3 sync s3://oldbucket s3://newbucket --source-region us-west-1 --region us-west-2
3
  • Where can you find a list of region names?
    – bart
    Dec 8, 2016 at 9:15
  • You can find the S3 regions here: docs.aws.amazon.com/general/latest/gr/rande.html#s3_region These are almost identical to the EC2 regions you get from aws ec2 describe-regions, but us-east-1 is also known as US Standard for S3 and is somewhat different for historical reasons.
    – A B
    Dec 9, 2016 at 5:11
  • aws cli v2 drops this region requirement as well Jul 9, 2021 at 15:44
5

I guess, you can run:

s3cmd --recursive cp s3://oldbucket/ s3://newbucket

It should copy it directly.

1
  • CLI says: "command not found"
    – bart
    Dec 9, 2016 at 7:05
3

I don't know of a packaged solution to do this, but it is essentially just a bucket LIST and then PUT-copy each object from the US bucket to the EU bucket. The PUT-copy API supports "if-modified-since" and "if-none-match", so you can run it repeatedly without duplicating effort: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/API/index.html?RESTObjectCOPY.html

For a live migration, I would suggest the following approach:

  1. Optional first step: If you have a huge amount of data to migrate, consider using AWS Import/Export to capture the first snapshot of the data on a physical device in the US and ship it physically to the EU. This may save you time and money.
    • Set up a script to migrate data from US to EU using the if-modified-since or if-none-match as described above. Make it smart enough not to clobber fresh EU data with stale US data (once you get to step 5 below). Run it continuously in a loop until the two buckets are in roughly the same state.
    • Modify your web app to PUT data to both the US and EU buckets at the same time, while continuing to GET data exclusively from the US.
    • Continue running your migration script until you are confident that the buckets are in exactly the same state (or as close as you can get, given eventual consistency).
    • Modify the web app again to GET data exclusively from the EU bucket.
    • You can now turn off your migration script.
    • Look for eventual consistency anomalies and clean them up.
    • Modify the web app to stop PUTting data to the US bucket.
    • DELETE everything in the US bucket.

You may want to use S3 Reduced Redundancy Storage on your EU bucket during the migration to get cheaper data rates and faster response times, since the data is just a duplicate of the US data.

2

You can also consider seting up Cross-Region Replication, which allows you to both copy and keep in sync data between regions https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/

1
  • Please expand your post to be self-sufficient. Apr 14, 2015 at 10:41
0

I've done a fair amount of work with S3 and have never come across a tool that will do this natively. However, it wouldn't be all that difficult to use a couple of s3cmd-sync sync scripts to keep two buckets in sync with each other.

1
  • yeah, that would work, but require us to download everything locally, while S3 API supports moving/copying entries without downloading them.
    – skrat
    Apr 21, 2010 at 9:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .