1

We need to transfer our São Paulo buckets to US East so we can make use of glacier.

Those buckets have more than 4Tb each. We moved the first using AWS API using an instance at US East to send those requests and it cost over $800.

Do you know any other form of transfer the data without incurring this cost ?

3

There is no way to avoid the cost entirely, because any form of "outbound" data transfer comes with a cost.

Import/Export "disk" services are not yet available in your region, otherwise you could use a physical external hard drive, ship it to Amazon, they load your data from S3 to the disk, send it to you, you send it back to them at a different location, where they would then transfer the data from the disk to S3. Import/Export "snowball" service would be similarly useful, but it doesn't yet support export -- only import.

Is your data compressed? Data compression might tremendously reduce the amount of actual transfer that needs to occur. Programs like xz (single core) and pixz (multi core/parallel) and zpaq can often reduce your data size to less than 1/10th the original size or more. You could store it compressed, transfer it compressed, and then either decompress it after transfer, or leave it compressed to save long-term storage costs. Generally, the smaller the files are that the compression algorithm takes, the longer it takes to do the compression, but spot instances are a good way to harness cheap computing power to process large compression jobs.

Another possibility appears to be available by funneling your content through CloudFront. (Skeptics: hear me out...) The charge for data transfer from São Paulo to another AWS region is $0.160/GB, but the cost from S3 to CloudFront is quoted as $0.00. The cost for transfer out of CloudFront in the US is $0.085/GB. If you created an EC2 instance in us-east-1 (Virginia) and then accessed CloudFront from there, the request should route to CloudFront in the US, and if your CloudFront distribution is provisioned for the lowest cost tier, requests should always be served out of the US. EC2 to S3 traffic within us-east-1 would not be subject to per-GB bandwidth charges. You would not be able to use the built-in S3 copy mechanisms, but you should be able to cut your transport costs almost in half with this approach, if you write the necessary supporting code to make this work.

Caveat: I have not tested this to verify that the quoted prices actually impact the billing in the expected way -- I am only working from my interpretation of the pricing tables -- but it appears to be one of the few places where there is a small opportunity for "arbitrage" of data transfer prices in AWS. It is premised on the concept that there is no "regional" distinction made with regard to data out, from S3 to CloudFront.

| improve this answer | |
  • it is a nice workaround. I'll try cloudfront. Our data is already compressed. – otaviofcs Feb 3 '16 at 14:33
  • (AWS did actually add Snowball export not long after this answer was posted - see aws.amazon.com/blogs/aws/… - but it's still not an option at all at Sao Paolo). – timday May 14 '16 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.