Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If we are to make something like youtube, then what should be the approach with respect to storing the videos? Any idea about what youtube does ?

share|improve this question

migrated from Nov 12 '09 at 7:26

This question came from our site for professional and enthusiast programmers.

@Darin Great link! Thanks. – jjnguy Nov 12 '09 at 7:20

Google doesn't just store a lot of data on a lot of servers, they have written software that distributes the load and storage automatically, while recovering from hardware faults transparently (like BigTable and Map/Reduce).

The alternative is: a. You use someone else's infrastructure (Amazon S3, Akami) b. You use expensive storage vendor's products (NetApp) c. You write your own system similar to Google's

Facebook recently switched from a combination of A and B to C for their photo storage as it grew to unmanageable proportions.

share|improve this answer

Content Distribution Networks (large clusters of servers at geographically distinct locations) are the key for any data and/or traffic intensive web-scale app.

share|improve this answer

It depends on if you insist on storing and distributing these files yourself.

If you don't need to manage this yourself then there are lots of content distribution networks (akamai etc.) who will hold and resend your files for you, they're usually geographically disperse and have excellent resilience. Your site would then simply need to point at these files to allow your users to access them, your job would then essentially be done.

If you have a need to store and distribute these files yourselves then at the very least you need a big block of fast and resilient storage (such as a large SAN/NAS by EMC/HP/NetApp etc.), some web servers tuned to deliver video (my own personal speciality :) ) and some big caching load-balancers such as Zeus's ZXTM. Of course at this point you have a single point of failure (the data centre) so if up-time is important you should consider one or more additional data centres and distribute your data and load around.

Good luck.

share|improve this answer

Yes - buy many hard disks and servers :P

share|improve this answer

I would probably store them using lots of servers with something like mogileFS. This way each file is stored on more than one server, so a server failing will not lose your data.

Fill them with low-power 2TB drives.

You'll need to be careful when trying to store large files, and not use perlbal as your mogstored, but it'll work. You could also split up the large files into several big chunks like mogilefs likes to do, but just keeping large files around is easier on your client code.

share|improve this answer

I'm running for getting a "+1 funny", so:

Where does youtube store so many videos?

If not in the cloud, then on disks.

share|improve this answer

protected by splattne Nov 25 '10 at 10:24

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?