3

I've been looking into using the Amazon Elastic Transcoder to implement a on-demand videostreaming scenario for quite some time now. The variables around the transcoding cost and restrictions are clear to me. I am bound by the requirement that I need scalability in streaming bandwidth, preferably in increments of 200Mbit/s.

I am, however, struggling to get a grasp on streaming cost. From what I understand from the documentation is that I would transcode using Elastic Transcoder, transfer those files to Amazon S3 and then serve the contents using CloudFront. Transcoding, storage cost and CloudFront Volume costs are easy to determine using the calculators. What I cannot seem to find is information on the bandwidth of CloudFront + S3.

The bandwidthlimit on CloudFront seems to be 40Gbps, according to this document. I find it hard to believe that I wouldn't be charged extra for such bandwidth usage. Also, I doubt the S3 servers/nodes could provide 40Gbps connectivity.

So the real question is: What would be the maximum streaming bandwidth using (Elastic Transcoder +) Amazon S3 + CloudFront? And how and when would that be limited? Or is CloudFront pricing merely determined by Outbound Data transfer volume?

2

I find it hard to believe that I wouldn't be charged extra for such bandwidth usage.

40 Gbit/sec isn't "usage" -- it's available capacity. You aren't charged for available capacity. You are charged for usage, which is the "Regional Data Transfer Out to Internet (per GB)" mentioned in the CloudFront pricing table. There is no charge for the bandwidth from S3 to CloudFront, when CloudFront fetches objects from S3.

Also, I doubt the S3 servers/nodes could provide 40Gbps connectivity.

Well... don't. :) With one of the large EC2 instance types (e.g. hi1.4xlarge, cc2.8xlarge), if the instance is in the same region as the bucket, and you download a large file from S3, you can easily see downloads of individual files with transfer rates in the 1 Gbit/sec range from S3. It has the capacity.

40 Gbps isn't the capacity available for a single download, of course, but on the scale of AWS, this much bandwidth available it is not at all out of the question. None of the components in the architecture you describe is dedicated to you.

It's shared by all -- or a significant subset of -- customers, so there is no need for 40 Gbit/sec of anything to be dedicated to your use. It also does not need to be in one place. There are over 60 distinct, independent locations on the CloudFront edge network and each browser connects to what CloudFront (with help from Route 53, even if you aren't a Route 53 customer) considers to be the optimal edge location for each browser, using their own proprietary geo-routing database, which appears to be adaptive. (My test site shows which edge location your browser is currently using, if my experimental mapping logic can determine it -- it usually can -- and I see it periodically shift even when I hit the site from home.)

With only 40 edge locations, less than the current number, a comically small amount of bandwidth available at each edge would accommodate 40 Gbit/sec of transfer to a globally-dispersed audience.

Each region's S3 infrastructure is also distributed across hardware spanning the region's availability zones.

40 Gbps in fact isn't even a physical limitation...

And how and when would that be limited?

...it's a software-enforced protective control. Just like the limit of 100,000 requests per second. The request would be denied.

LimitExceeded – The request was denied because a CloudFront limit was exceeded.

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html

Contrast this to CapacityExceeded which would be logged in the event of an actual problem in the CloudFront network preventing the request from being processed.

S3 documentation doesn't mention a bandwidth limit, but among the discussion of scaling:

if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate.

http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html

Note that this refers to "a rapid increase," implying that a gradual increase would automatically be handled by S3's automatic scaling and/or sharding of its lexically sorted object key index.

What would be the maximum streaming bandwidth using (Elastic Transcoder +) Amazon S3 + CloudFront?

Everything is of course finite, but in the case of the services in question, there is no way to meaningfully describe the limits. It is effectively unlimited for all practical purposes.

Note, for the sake of absurdly playing with numbers, that actually using 40 Gbit/s, all day, all night, for an entire month would almost certainly never happen, but if it did would cost in excess of $400,000 in outbound data transfer (an extremely round number, the regional pricing and discount tiers make the calculation complex) but makes the fact that this much capacity is technically "available" seem a little more believable -- the capacity is there if you want to pay to use it, but if you don't use it, you don't pay for it.

  • 2
    Thank you for this very elaborate and in-depth answer! I can certainly use this to fill in some of the blanks. – Wouter Pol Jan 5 '17 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.