0

I have a large number of files stored in an AWS S3 bucket. I need to get those files to another S3 bucket owned by another user and in a different region. The files are stored in different locations within my S3 bucket--these keys (e.g. location1 & location2) need to transfer over too. Both S3 buckets are associated with AWS EC2 Ubuntu Servers. Here is an example of the structure of my S3 bucket:

s3://mybucket/location1/some_file
s3://mybucket/location1/some_file2
s3://mybucket/location2/some_file
etc...

What approach to transfer files from S3 to S3 would be most efficient given these constraints?

  • What do you mean by "most efficient"? Fastest? Least human effort required? Least computing resource required? Least network bandwidth required? Cheapest? Or something else? – Mike Scott Jun 14 '16 at 12:15
  • @MikeScott Fastest – Borealis Jun 14 '16 at 12:22
4

https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-s3/

You can copy Amazon S3 objects from one AWS account to another by using the S3 COPY operation. You must give the destination AWS account access to the source AWS account's resources by using Amazon S3 Access Control Lists (ACLs) or bucket polices.

After setting cross-account IAM permissions, it's as simple as aws s3 sync s3://sourcebucket s3://destinationbucket via the CLI. No middleman servers required - everything happens on S3 directly (and will be substantially faster as a result).

(If you don't want the entire bucket, the CLI can copy individual objects just as easily)

-4

You can implement the following scenario:

  1. Deploy a temporary EC2 instance in either source or destination region. Bigger instance you choose, better performance you receive.
  2. Install an S3 client on this instance and transfer your files.
  3. After the operation finished, terminate your EC2 instance.
  • 2
    Vlad, aren't you affiliated with CloudBerry? Affiliations must be disclosed in answers. – Michael - sqlbot Jun 14 '16 at 12:48
  • No, I am not affiliated. In this case CB is just an example. – Vladimir Mukhin Jun 14 '16 at 13:14
  • 3
    @VladimirMukhin Your profile says "Work for CloudBerry Lab". Don't lie. – ceejayoz Jun 14 '16 at 13:20
  • 4
    Vladimir, @ceejayoz is right. You clearly have some affiliation with Cloudberry. This comment is another clear indicator, in case your profile isn't enough. – EEAA Jun 14 '16 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.