My prior experience with servers has generally been limited to home file-sharing servers, low-traffic web-servers, and the like. This leaves me with the technical knowledge of how to set up a system, but little experience in terms of scaling said system.

My current project, however, has me as the technical lead in setting up a school for online audio and video streaming. The difficulty I'm running into is that I don't quite have the experience to guess what they'll need, and they don't have the experience to tell me - so I've tried to ask as many pertinent questions about what they want to do with their server, and here's what I found out:

  • About 1000 simultaneous users, and hoping to expand (possibly significantly)
  • Both video and audio streaming, at obviously the highest quality possible
  • Support for both live and playlist-based streaming.
  • Probably only one channel, but as it's an educational opportunity, I imagine letting them have a few more wouldn't hurt. (strike this - I thought that channels was referring to streams, which is odd, given that I'm originally an audio guy!)
  • No word on whether they're locked into Windows or whether Linux is acceptable.
  • Approximate budget - $7000. It may actually be about $2k less than this, because of a mishap with another technology firm (they ordered a $7000 DV tape deck for some reason, and now the company wants them to pay a 30% restocking fee).

The tentative decisions I've already made:

  • I'm planning on using Icecast 2 for my streaming server, fed by VLC Shoutcast encoding.
  • Since the school already has a DMZ set up, I plan on placing the Icecast server in there, and feeding it through their intranet from a simple workstation computer in their studios.
  • This system isn't in any way mission critical - it's an education tool (they're a media magnet school), so I figure redundancy is not worthwhile to them from a cost:benefit perspective.

What I don't know is this:

  • How powerful of a server will I need?
  • What is likely to be my major throttle - bandwidth? How can I mitigate that?
  • Will I need anything special for the encoding workstation other than professional video and audio capture cards and a copy of VLC?
  • Are there any other considerations that I'm simply missing?

Thanks a lot for any help - if there's more information you need, let me know and I'll tell you all I can.

Edit 1:

I'm expecting to go out either in OGG or MP3 - probably MP3 as it's more widely supported (and more familiar to parents of grade-school students.

The content will consist of live/prerecorded standard definition video packages (likely in Quicktime, for the latter). Audio-only broadcasting will likely consist of MP3s, transcoded down to a more manageable quality. DRM is not an issue, nor is copyright (at least, copyright is not my issue). Content is generated whenever the students feel like broadcasting something, but the storage of this media shouldn't be a problem. They're not generating raw movie footage here - they're kids doing small TV packages - I'll throw 4 Terrabyte hard drives in RAID 10 on the workstation and call it a day.

Much of the rest doesn't quite seem to apply. The most relevant portion is the inherent limitations on the network. When I tried to inquire about what those might be, they really had no idea - as they've never really stress-tested it. Their internet connection is used largely for standard internet service for the students and teachers, and serving a website likely built in the 90s. I doubt they'll go through an expensive refitting of their NAT and front-end systems - the kids will simply have to live with what they have.

I don't think my server(s) will have to worry about massive caching, or large chunks of data storage - they will simply receive the stream ready to go from the internal workstation/source client, and have to replicate and distribute the stream to the clients. I expect to run into bandwidth bottlenecks before I find myself clamoring for more memory.

Regarding multicasting - I thought the internet at large didn't support it - and external clients make up the large bulk of our potential subscribers. My big question is a bit more rudimentary - what is a "decent server"/server system for this sort of application? I'm having difficulty envisioning what quantity of machines I will need, as well as what they need to be capable of. Any thoughts? Thanks in advance.

  • Hi, thanks for your updates, looks like your challenges aren't as hard as perhaps I'd painted them, certainly in terms of intellectual property rights, content volume and the mix of audio vs. video. You're right about multicast, the difference for my own scenario is that I work for a ~5M user ISP where we DO have multicast ability - if we do things right anyway, just wanted you to be aware in case all of your users were 'in house'. As for server, well make sure you get a dual-socket xeon-based server, even if you only use one CPU today, get an up-to-date machine as they really are ...cont... – Chopper3 Jan 16 '11 at 12:02
  • ...cont... that much faster, try to make sure you've got a decent number of disk slots so that you can easily add new ones, if content volume isn't an issue I'd be tempted to use 2.5" SAS disks over 3.5" SATA ones if you can as their random-read abilities are much better, stick to RAID 10 as you've said, use a 64-bit OS, oh and a brand-name hardware disk controller (Adaptec etc.) not a system-board based one, use a minimum of 2 x 1GB NICs teamed to deliver your video with one or more extra ones set aside for non-streaming traffic, oh and don't forget backup! ...cont... – Chopper3 Jan 16 '11 at 12:07
  • ...cont... as for software, perhaps consider Darwin Streaming Server? My software is bespoke so I'm less help with that sorry, I know some people on here use Flash server but I have no experience myself sorry. Anyway I'm running out of stuff to help you with at this point, please come back to us with anything else you need help with ok :) Good luck – Chopper3 Jan 16 '11 at 12:09

I build IPTV systems and you're not going to like it but you've got a very tough task ahead of you, especially considering your budget.

Let's go through it step by step (I'm aiming this answer to a broader audience than just you if you don't mind);

  • The first thing anyone building such a system has to do is define its customers in terms of locations, OSs, players and ideally understand the networking aspects between clients and servers - you've done a lot of this already. This is important as it defines the server code, codecs, bit-rates and whether you can use multicasting at all.

  • The second thing to do is understand your content; where does it come from, in what format, how often, copyright issues, DRM requirements, how quickly it needs to be available and how long does it need to be retained for. I think you've got some of this covered but you can probably ask some more questions along these lines.

It's far from a simple task but based on the results of the two blocks of questions above you can start to design your system, again from the point of ingest down - how EXACTLY are you going to reencode (if required) your content, does it need QA'ing, where does it get DRM stamped and how are you going to deal with the risk of temporarily having unencrypted video if that matters. This ingest/staging system needs to be considered up front, you don't want to have to 'hand-crank' this process at all because it'll get repetitive very quickly.

Once you've figured out this system you need to know what's doing your content cataloguing and how the ingest system feeds into this, also how you're going to publish this catalogue and how you removed 'aged' content. This publishing system will then initiate client playback and may require some form of entitlement checking system, which brings its own problems that we could tackle another time.

Now we get onto the part you're directly interested in, the streaming servers.

By this point you'll know your user base data (who, what, where, when, how etc.) and this will help you calculate your peak bandwidth requirements (usually something like 'peak users' x 'highest bit-rate').

"Best quality" is something of a misnomer as I'm sure you don't mean "48-bit 8k HD @60fps" but more like SD quality or so, as a guide I do SD at around 1.5Mbps and HD at about 6Mbps but this will be defined by your codec and requirements. So as an example 1,000 users playing back 1.5Mbps of unicast traffic obviously equals 1.5Gbps, this is where the hard bit comes in. Firstly can your network, the actual trunks and switches themselves consistently deliver data of this nature? You'll need to sit down and work out where your weak-spots are. Can you QoS this data to protect it from someone pulling down a big download and killing an entire network segment's video? There's also the fact that if you're your going over 1Gbps on your NICs you need to ensure that you have them teamed in the right way so that the second and subsequent team member actually gets to carry it's share of the traffic, that or go to 10Gb NICs.

Then we're onto my specialist area, the storage - by now you've worked out how much storage you'll need for content at the start, how quickly it'll grow, it's maximum size and it's 'churn rate' (how much comes on and off the content store over what period). This will tell you how much storage you need, but if you have 1,000 users all looking at a good-size catalogue of content what happens is that you have to pay very close attention to caching and the random-read capability of your storage.

If you're VERY lucky and your server can cache 100% of your content then you're in luck, you just need a decent server, lots of memory and a 64-bit operating system. If you expect users to have access to a larger content store than you can cache then you need to ensure that your storage system can deliver your peak streaming requirements, say 1.5Gbps in the example above, consistently. How you do this depends on the size of content store and if you need more than one streaming server for this (as if you do you'll have to figure out if you're going to shard or share your video).

You can look at SSDs for example but don't look at the 'headline' numbers such as 500MBps, what you need are SSDs or disks that can be RAIDed to ensure that peak delivery speed. There are plenty of SSDs and disks out there that are great for workstation or low concurrency servers but when you hit them with a requirement to keep up with say 1,000 users all pulling little pieces of large files at different point in the files - well basically they don't keep up and neither do most caching algorithms - you have to know that your storage can do the work on its own if it has to.

If it's any consolation the live-streaming bit is much easier but probably deserves it's own machine to handle the capture/encryption/streaming rather than the main content-store based server/s. You're lucky in that you probably have a better understanding of your network as it sounds a closed system, if you can you want to ensure you can multicast this otherwise you're into the same bandwidth management issues I describe above but with a better cache-hit opportunity.

I hope this helps, like I say I've aimed it at a wider audience than just this question and I may come back and add/edit (once I've had my first cup of tea for the day!).

  • That's a lot to chew through - thankfully, I think much of it doesn't apply to my specific circumstance (though the additional information is interesting and serves to broaden my perspective), and the rest is quite helpful. I'll be back after sleeping a bit to answer point-by-point how I think parts of your post apply to what I've already done, and what I anticipate having to do yet. – Brian Bauman Jan 14 '11 at 9:42
  • Ah, the sweet sweet sound of direct professional experience. +1 sir. – mfinni Jan 14 '11 at 17:42
  • Updated original question to reflect your post. – Brian Bauman Jan 14 '11 at 23:12
  • 2
    Thanks for the help. I managed to spend a week putting together a solid setup that would do what they needed, in budget - only to find out after the fact that they were axing my role to go with an online streaming provider, which will only save them money for two years. Sorry I can't report better results, but thanks again. – Brian Bauman Feb 1 '11 at 15:36
  • sorry to hear such bad news - take heart that you did a good job when asked. – Chopper3 Feb 1 '11 at 16:02

Stream bandwidth depends on the quality and the receiving end computing capacity. You could aim for 0.5 Mbit/second per client. This is aprox equivalent of DVD on DivX compression.

You should try finding a solution where server(s) don't need to do any coding or transcoding and the stream is already in the correct format. Especially not per client basis.

High end hardware costs proportionally more, and you end up in trouble if it brakes down. Try a farm of lowend sub-$1k servers with a spare one if needed. Using the same image for system disk for the servers is a good idea.

With only streaming data - no coding/transcoding - the bottleneck comes from the network interfaces (and possibly network infrastructure). You should calculate not to exceed 10% of named bandwidth of the interfaces or the connections will slow down.

I am not sure if your network everywhere is equipped to handle 500 Mbit/s additional traffic (with <10% capacity). If not, you could try doing servers for each location or cut down the bandwidth. You can use five servers with gigabit ethernet but they cannot be connected to a hub with a single gigabit uplink.

Hope this helps.

  • The bit-rate estimate is helpful - I'm sure they're using some sort of fiber out of their DMZ, so 1000 users streaming at .5 Mb/s shouldn't use any more than half their through-put. All the encoding would be done on the encoding workstation, and streamed once through the intranet to the streaming server(s), so that shouldn't be a problem. How do you have a farm with a spare? Isn't the whole point of a farm that they're all spares for each other? Additionally, how would you recommend getting the same stream to each server, and likewise routing requests? – Brian Bauman Jan 14 '11 at 8:46
  • Regarding the 10% comment - we're streaming straight out to the net, so is this a consideration I have to worry about? The only traffic their intranet should have to worry about is the source stream. Thanks for the help. – Brian Bauman Jan 14 '11 at 8:48
  • Sometimes all the computers on the farm are not identical and some are critical. It's nice to have spare hardware and plug in the previous harddrive or boot from iscsi. Live streaming to each server should be easy from the streaming software. The servers would proxy the content forwards. With playlist the distributing servers need to pick up the material from another streaming server, http or NAS. – Antti Rytsölä Circles Consult Jan 14 '11 at 10:27

Contact the guys from the TightwadTech, this is right up their street, and hopefully we get a podcast out of it :-)

  • A nice idea, but unfortunately the school has me on a bit of a crunch. – Brian Bauman Jan 14 '11 at 23:12
  • Shame, but you could always report back with them in 6 months time with bruises gained and lessons learnt :-) The TightwadTechs recently did a feature on video editing and hosting, they might have some instant tips. I'll drop them an email now! – DutchUncle Jan 15 '11 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.