5

If we are to make something like youtube, then what should be the approach with respect to storing the videos? Any idea about what youtube does ?

1

6 Answers 6

7

Google doesn't just store a lot of data on a lot of servers, they have written software that distributes the load and storage automatically, while recovering from hardware faults transparently (like BigTable and Map/Reduce).

The alternative is: a. You use someone else's infrastructure (Amazon S3, Akami) b. You use expensive storage vendor's products (NetApp) c. You write your own system similar to Google's

Facebook recently switched from a combination of A and B to C for their photo storage as it grew to unmanageable proportions.

2

Content Distribution Networks (large clusters of servers at geographically distinct locations) are the key for any data and/or traffic intensive web-scale app.

2

It depends on if you insist on storing and distributing these files yourself.

If you don't need to manage this yourself then there are lots of content distribution networks (akamai etc.) who will hold and resend your files for you, they're usually geographically disperse and have excellent resilience. Your site would then simply need to point at these files to allow your users to access them, your job would then essentially be done.

If you have a need to store and distribute these files yourselves then at the very least you need a big block of fast and resilient storage (such as a large SAN/NAS by EMC/HP/NetApp etc.), some web servers tuned to deliver video (my own personal speciality :) ) and some big caching load-balancers such as Zeus's ZXTM. Of course at this point you have a single point of failure (the data centre) so if up-time is important you should consider one or more additional data centres and distribute your data and load around.

Good luck.

1

Yes - buy many hard disks and servers :P

1

I would probably store them using lots of servers with something like mogileFS. This way each file is stored on more than one server, so a server failing will not lose your data.

Fill them with low-power 2TB drives.

You'll need to be careful when trying to store large files, and not use perlbal as your mogstored, but it'll work. You could also split up the large files into several big chunks like mogilefs likes to do, but just keeping large files around is easier on your client code.

-1

I'm running for getting a "+1 funny", so:

Where does youtube store so many videos?

If not in the cloud, then on disks.

You must log in to answer this question.