0

I'm building a website that needs to serve about 300-500 GB of static data from the get-go, with about 5-10 GB added every month. The files are mostly photos.

All of my potential users will be located in Canada, with about 95% of them in a single city (Toronto).

I have many options, but I don't know which is best for my situation.

First of all, I don't think I need to use CDN since the majority of my users will be based in Toronto. I just need to find a database provider based in/near Toronto.

I want to host my website on a VPS in the beginning, since it's easy to set up and should have no problems accommodating my initial users.

The problem is where I should host the 500 GB of static files. Should I host them alongside my website on the VPS? Or should I host them on a cloud platform like BackBlaze B2 or Amazon S3? HDD is sufficient for my static files, although I would like to host website itself on a SSD.

I estimate that each user will be served about 10 MB of data per visit. Assuming I get about 5000 visitors a day, I should be able to serve 50GB of data daily with no visible decrease in performance. The user will spend most of his time on a map (I'm using Google Maps JavaScript API), so it will be highly dynamic with continuous ajax calls.

I'm not too worried about getting perfect availability in the beginning, as long as the site is up most of the time.

9
  • 1
    A CDN has plenty of benefits even if your users are all in one place.
    – ceejayoz
    Dec 22, 2017 at 19:14
  • 1
    To clarify: 300 - 500GB of files, but what volume of downloads - how many GB per day? Do you need very high bandwidth? What's the average size are the files?
    – Tim
    Dec 22, 2017 at 19:21
  • @Tim I estimate that each user will be served about 10 MB of data per visit. My goal is to get 5,000 daily visitors. So I should be able to serve 50GB of data daily with no visible decrease in performance.
    – Valachio
    Dec 22, 2017 at 19:26
  • Do you need high availability? Is price a significant factor - what's your budget? How critical is performance locally, and globally? Is your website largely static or does every user log in, making it largely dynamic?
    – Tim
    Dec 22, 2017 at 19:53
  • @Tim High availability would be nice, something like 99.9% is good enough. Cost is certainly a factor (I'm a poor plebeian student), but having good performance is more important to me than slightly higher costs. The user will spend most of his time on a map (I'm using Google Maps JavaScript API), so it will be highly dynamic with continuous ajax calls.
    – Valachio
    Dec 22, 2017 at 20:12

2 Answers 2

1

I'm not going to go through costs, as you can work them out yourself easily using public pricing. Generally VPS hosting of large amounts of files is most expensive, followed by AWS, followed by B2.

If you want high availability you'll want two servers and a load balancer. This is fairly easy in AWS, Azure, and Google, and some smaller providers can do it as well.

AWS S3

AWS S3 is great, and has redundancy built in - from memory your data is stored in and potentially served from three data centers. Infrequently accessed class storage can reduce the bill, and is fine for low volume file hosting. At your 1.5TB per month will cost something like $30 per month. Good system, reliable, easy to add a CDN. AWS has a region in Canada.

AWS LightSail / VPS

AWS Lightsail VPS has a lot of bandwidth quite cheap, for AWS at least. Digital Ocean and the other decent VPS providers are similar. You should consider just having plenty of storage on your VPS. The downside here is SSD storage will probably be quite expensive, and you have no redundancy - but if you're running from a single VPS you have a single point of failure anyway.

BackBlaze B2

B2 is cheap storage, but more expensive bandwidth. I think they only have one data center, so there's less redundancy there. They'll be somewhere in the USA, so latency and bandwidth should be pretty good, but not as good as AWS. Another single point of failure.

CDN

If you use a VPS then you should consider using a CDN. A CDN can significantly reduce your bandwidth costs if set up correctly.

CloudFlare free tier is fantastic, and you can pay for a better class of service and more features. As well as serving users faster than your server can, it will serve remote users more quickly. Google likes fast websites too, and their crawlers could connect from anywhere.

Recommendation

Your priority seems to be performance, price and reasonable availability. AWS S3 IA class is probably a good trade-off, but if you can put up with it being slightly slower you can try B2.

You can put CloudFlare's free (or paid) services in front of Amazon S3. Instructions here.

4
  • Great answer. Thank you very much. I think I'll probably go with S3 since it's the industry-standard
    – Valachio
    Dec 23, 2017 at 3:06
  • AWS S3 is a safe bet, but I wouldn't say "industry standard" as such. Try the IA storage class first, I think it'll be fine, but be prepared to move to standard class. With a CDN IA should be fine. You could look at Google Cloud as well. I'm not a huge Azure fan, it just doesn't feel as well designed to me. If you have cheap disk for a VPS hosting yourself via a CDN to reduce bandwidth and latency is a decent option, but VPS disk is often quite expensive.
    – Tim
    Dec 23, 2017 at 3:37
  • I just discovered Digital Ocean came out with their own object storage service called Spaces, competing against AWS S3. They are aimed at smaller developers and the primary selling point is their significantly lower bandwidth cost. S3 charges $0.09 for each GB of bandwidth, while Spaces charges a mere $0.01 for each GB. Spaces also offer free requests (S3 charges $0.005 per 1000 requests)
    – Valachio
    Jan 12, 2018 at 15:37
  • 1
    AWS bandwidth is incredibly expensive. What spaces does for $5 is $96 on AWS, $90 of that is bandwidth fees. If you use a LOT of bandwidth it might be worth using. If you're low bandwidth and are ok with infrequently accessed storage then AWS might not be too bad.
    – Tim
    Jan 12, 2018 at 23:14
0

I only add this because it seems to be missing from the excellent lists of suggestions so far put forth, but if you have a lot of data don't overlook the possibility of co-locating your own kit. That might seem rather 1990s, but it's still very much an option.

If you get/build a decent 1U server you can put at least two large, fast discs in it, and RAID-1 them. Your 1.5TB of throughput a month is well within the spec of most colocation offers. And although it's possible that you'll have a couple of days outage when something goes pop, using quality hardware will minimise the likelihood of such an outage, and having a few judiciously-chosen shelf spares (PSU!) will help reduce the duration of such an outage.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .