1

Alright, so I'm setting up an off-site backup of my S3 data and have set up Cross-Region Replication to a new bucket in another region. However I have a lot of data in the original bucket which needs to be moved as well, >100TB in >20 million files. My first thought was to just run:

aws s3 sync s3://source-bucket s3://destination-bucket

on an EC2 instance. But that's taking way longer than I anticipated, and with all the PUT/LIST requests it's making, costing more than I anticipated.

Reading the AWS docs, it looks like they recommend AWS Snowball for this kind of operation. From the FAQs:

As a rule of thumb, if it takes more than one week to upload your data to AWS using the spare capacity of your existing Internet connection, then you should consider using Snowball.

However it looks like those are intended for either import or export, not both at once. Would I need to do two separate jobs with the same snowball? Won't I still get charged for all those PUT/LIST requests anyway, to get the data onto the snowball? They mention the $0.03/GB for data transfer, but didn't mention API requests.

2

If you're copying data that is already in AWS to another region, Snowball doesn't seem like a good option. Running sync from the command line of an EC2 instance within AWS seems right.

Have you tried configuring max_concurrent_requests? Possibly that will address the performance issue. Also, this article has more suggestions for your situation.

If you have many objects in your S3 bucket (more than 10 million objects), consider using Amazon S3 Inventory reports and Amazon CloudWatch metrics. These reports can help optimize the cost and performance of verifying the copied objects. You can also split sync commands for different prefixes to optimize your S3 bucket performance. For more information about optimizing the performance of your workload, see Request Rate and Performance Guidelines.

How can I copy objects between Amazon S3 buckets?

  • That sounds promising. But I was hoping the cost would be improved too; we're only about halfway through and it's already cost over $7K to transfer. It's starting to look like the Snowball would be cheaper, unless I'm misunderstanding their pricing scheme. – Aaron R. Aug 14 at 16:13
  • 2
    If you have copies of the data on site, snowball may well be less expensive. If not, you'll have to take into account the S3 transfer-out charges. I suspect the needs of "fast" and "inexpensive" are in conflict here. – vjones Aug 14 at 16:30
  • Transfer acceleration is for data in and out of S3 via the REST API. A "sync" operation should be using The PUT+Copy API which sends a request to the target bucket instructing it to fetch the object directly from the source bucket, using internal back-channels and not download+upload. This should be the lowest cost possible, since it relies on cross-region data transfer, which is cheaper than egress. If the data is not already heavily compressed, a process that transports it cross-region, compressed, using EC2, is the way to go. – Michael - sqlbot Aug 14 at 20:31
  • 1
    I removed the transfer acceleration suggestion from the post. – vjones Aug 15 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.