I am in the process of building a website that will distribute a large amount (>150GB) of small files (~25Mb Avg.) to a large number of people, since both bandwidth and disk space are expensive with my server company, and we are on a low budget, I'm thinking that we should go with Amazon's S3 service to host our content, are there any potential gotchas or specail precautions that should be taken when building a high demand website around Amazon's S3?
S3 is not technically a hosting service. Yes, you can put content up there but seemingly simple things that you may depend upon are not present. E.g.:
- no index file functionality
- no byte-range requests
- no logic behind anything (although there are some ACL options)
Another catch is that, no matter the bandwidth on the client side, the per-file transfer rate will be limited. (E.g.: Here in Europe, I measured 1-2mpbs from the US Amazon S3 service. We had some connection as bad as low 100s of kbps.) That said, cumulative bandwidth is effectively unlimited.
Have a look at CloudFront, which is Amazon's CDN that works with S3.
Note: I'd not call 150GB of data 'large', especially in the scope of Amazon's platform. It is peanuts for them.
The phrase "low budget" seems to be a red flag for me. Last I checked, Amazon S3 didn't have a way to limit the amount of $$$ you could be billed if a zillion people decide to check out your website.
You need to look at Amazon CloudFront.
Amazon CloudFront delivers your content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance.
Try the application from CloudBeryLabs to manage yoyur storage. There is a setting in there for CloudFront.
I am using it to host my deployment apps via Microsoft ClickOnce and I am getting excellent speed for all multiple downloads and updates. We are based in USA.