hosting environment for delivering FLVs

Question

What would be the ideal hardware setup for pushing lots of bandwith on a tube site?

We have ever-expanding cloud storage where users upload the movies, then we have these web-delivery machines which cache the FLV files on its local harddrives and deliver them to users. Each cache machine can deliver 1200 mbits/s , if it has SAS 8 harddrives. Such a cache machine costs us $550/month for 8x160gb -- so each machine can cache only 160GB at any given time.

If we want to cache more then 160gb , we need to add another machine..another $550/month..etc. This is very un-economical so I am wondering if we have any experts here who can figure out a better setup.

I've been looking into "gluster FS", but I am not sure if this thing can push a lot of bandwith.

Any ideas highly appreciated.

Thank you!

Answer 1

In general I would question the wiseness of using replication here. WHat I would do is.... ;)

• Dump the files into one central NAS / SAN (possibly replicated as need arises). YOu could possibly use some of the SUperMicro storage cases - 24 x 2.5" drives in 2 rack units. And yes, hsted / rented servers wont fit, you need to colocate your own. SPecial need = special hardware = not the stuff mass hosters offer.

• Put up a database that dynamically assigns the files into streaming server groups, to allow those servers to do some caching. These are identical with separate name server entries that point to all servers in the group and just make sure not every server has to cache every file.

• Create groups of servers (as per point before) that are responsible for streaming groups of files

• Let the OS file caching sort out the rest. Seriously ;) Put in max ram on such a machine and be done with it.

• And while you are at it, who had the super smart idea of running SAS drives... did you check oyu need them? Hint - put in SATA drives, basically... WD Velociraptors. 300gb per drive and I bet a lot cheaper than your SAS drives. Nearly as fast, running 10.000 rpm. I use them on databases and get a very economic IO out of them. And I think my IO requirements are higher than yours (as videos tend to be larger than the data I work with).

Basically you got into the range where central storage system makes sense, and you try to work around it with - sorry - mediocre storage items 88 drives is NOT impressive for one server anymore). The result is tons of hardware cost ;) Now you can either go special cases (look at the supermicro offers I did put out - they also have a large 3.5" disc cage for 48 discs) or special hardware (which, incidéntally, will cost even more). A nice setup with your front end machiens without large discs, and a central storage using proper high end RAID controllers, plenty of discs and one or a couple of 10gb adapters should be good.

Forget anything about clustering file systems - you want something that is plannable. Problem is: you need to plan your bandwith within the system. You can not have too much cross traffic unless you are willing to put 10gb switches into everything. And even then the cross traffic may kill you.

Answer 2

Intelligent Caching!

In your case network bandwidth and i/o will be your current bottle necks.

If you can run something like zfs, then you can use a large SSD as a dynamic cache between physical drivers and memory.

Zfs will manage this cache. If you need better performance from the SSD, just stripe them together for higher performance.

If you don't have zfs, think about identifying 'hot' files on your cluster, and move them on to local SSD's or a dedicated ram disk. That way you can use a slow san to server files, and popular files will be served from fast SSD/RAM. Think of a SSD as extending your cache. It's not a fast as ram, but it's a lot faster than the faster than disk. (lower energy too!).

Max out the physical ram on the servers you have, this will improve caching. If you going down the PC route, I've seen some motherboards with capacity greater than 128GB ! :-).

If you keep growing, then you will soon outgrow your data centre's IP transit. If your users are geographically dispersed then think about putting the infrastructure closer to the users. i.e European users get directed to the data centre in Europe..

Also look at using a optimised web server for streaming the files, http://nginx.org/ springs to mind. You can still keep your existing application servers, but direct users to dedicate video streaming servers.

Memcache is interesting , depending on your works loads this can also help.

as you grow, bottle necks will appear within the infrastructure, make sure you have good performance/error/security log's. You'll be able to capacity plan much better if know what you infrastructure is doing under what users loads. Trend this data over a period of time and you'll be able to show management why you need the new hardware x to solve bottle neck y.

If don't have the in house skills to do all of this, then have a look a content Delivery network, Their infrastructure is tuned to do this. You just pay to use it!

Once you upgrade server to multiple 10Gbit Ethernet, you'll find you run into other bottlenecks, the actual system bus of the machine :-).

Answer 3

I'd second ZFS and a SSD cache drive - Works well. Also max out ram. OpenSolaris based ZFS BTW, the FreeBSD implementation isn't as reliable.

Answer 4

This isn't going to be the answer you want to here I'm sorry. I'm very specifically a Video-over-IP guy, I've built some of the biggest VoD systems out there and what you're doing now seems a huge waste to me, there are so many problems that I don't know where to begin.

What I'd suggest is that you sit down and work out what and where your clients are, what level of service you want to deliver to them, what your asset dataset and turnover is, how you're going to get and deliver your catalogue, how you're going to deal with the transactional aspects of what you're doing and only then revisit your entire deployment design.

As I say, sorry to be negative but anything else I could recommend would just be a very small sticking plaster over a much wider issue.