I'm working at a small TV company. They generates 11TB of video files per month.
What can I do?. I just need ordinary SATA disks, in RAID. What solution should I buy?
(They are using a lot of Iomega NAS)
I'm working at a small TV company. They generates 11TB of video files per month.
What can I do?. I just need ordinary SATA disks, in RAID. What solution should I buy?
(They are using a lot of Iomega NAS)
Hmmm
"Expanded ad-infinitum" scares me. Do you really need every second of film online there and then? Can you consider tiered storage? Fast storage for recent/highly used stuff; slower higher capacity storage to stuff you need handy, but not snaps fingers right there; and lastly tape or something similar to archive old data?
As for actual options, what do your budgets look like? I'm going to suggest stuff from the Dell range because that's what I'm most familiar with, but all the top tier server vendors will have their own products at the same points but I'm not as familiar with those and you need advice, not a long list of drive models.
I like Dell's Equallogic range of SAN devices if you want to publish this data through more than one servers, and they come in a number of options that can offer you a fast storage tier and a slower higher capacity tier and of course you can buy a tape library of any size you like from Dell too for the archival tier.
If you only want to publish this stuff through one server then consider the direct attached storage option - banks of disks that can plug into a server. Again from the Dell range I know about you'd be talking about the PowerVault range
As I said earlier, I'm not suggesting you have to buy Dell, I don't work for them and I'm not shilling for them - it's just that we use a lot of their stuff and I know it quite well. You can look at HP, Sun/Oracle, IBM and they'll all have similar ranges of products at the same levels, and there are plenty of companies out there who specialise in storage too. But letting file usage grow without managing it like this is a recipe for disaster imho.
I know you're probably being told that they need everything all instantly available, and that you need to do this cheaply, I understand the pressures. But trying to manage data like that without a management strategy is asking for the moon on a stick and it just won't work.
I work for a broadcast company and we will generate LOTS of video/audio data per month. Our solution was 2 Hitachi SAN, one faster (FC) and one slower (SATA) with lots of cache (256GB cache). We expect for it to exhaust in 3 years. Yeah, HD video is a bitch.
We have a management system in the works that will deal with all content life-cicle and moving it from faster storage (online) to slower storage (near line) and finally to be archived on tape (there's a lot of metadata involved to quickly and easily find the content on the archive later). We have a (expansible) 1000 LTO 4 tape library for archiving.
That's a thing that you should keep in mind. You really just need fast access/space to a determinate time of your programming. Say, the last month. That will be raw footage coming from news teams, recorded studio programmes, live show recordings, plus external productions (syndicated stuff, ads, etc). That is the content that will be going to the playouts and editing and for that must be fast to access (SAN connected to MACs where the editing is done, for example).
After the stuff is aired, you can (and you should) move it to a collection/archive (don't know the correct term in english), probably on tape. If a determinate piece of news, shows or programming is needed later, re-inject it to the faster storage for use and remove it after all is done.
Hope my personal experience helps.
If youre looking for a commercial solution, Isilon is one of the cheaper solutions out there and should suit your needs easily. Their product is a clustering solution. You build a storage cluster out of 3+ nodes. Each node has a bunch of disks in it, and the fault tolerance is customizable to where you can lose a node and/or multiple disks and still be operational. You simply expand by adding a new node or an external disk-pack to the existing nodes. Very simple and easy to work with.
After that I'd look at NetApp as a runner up. Theyre a little bit more expensive, but their products offer more features and performance than Isilon does.
I assume that this is video data, which means compression and de-duplication won't gain you anything. I also assume that you've decided the big, bulletproof commercial SAN/NAS vendors are too pricey. If everything needs to be online and on the fast LAN (and not on a cloud service, which I assume you've looked at):
Your most cost-effective option will be something like these 45-bay SuperMicro disk chassis stuffed with 3 TB drives. That's 135 TB raw in 4U and < 1500 watts. You can keep buying similar as needed, adding them to existing file server front-ends as required. RAID 6 via good LSI hardware controllers instead of software if you can afford it - read up on the "raid 5 write hole" and be scared of software parity raid.
As storage density improves, you can migrate data from older storage boxes to newer ones, keeping your server footprint fairly constant in the long term. Use tape or removable SATA disks for offsite storage as that is the only cost-effective way to back up this much data.
You'll be buying a 135 TB unit (with drives) about every six months accounting for RAID and filesystem overhead. You'll also need an additional front-end file server every once in a while too. And powering and cooling them all 24/7, and replacing failed drives, and rolling your own management software. Maybe you can use OpenStack to make your own internal cloud storage on these boxes, or live dangerously and risk it all on some of the open source clustered file systems. You'll likely need two or more full-time bodies managing all this stuff.
The cloud is looking better and better, isn't it? If the data must be local and online, I would seriously look at data reduction... after a month or so, re-encode all of your streams (presumably MPEG-2 broadcast) into VBR H.264 video at a much lower bitrate. X264 with '--crf 23' is close to transparent, and half or less the bitrate of most broadcast MPEG2 streams. When presented with the costs, I suspect management will simply live with data older than X days being offline on removable SATA disks or tape.
If you've got the time to set something up yourself, the guys at Backblaze have a good writeup on how they build their storage pods: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
Check GlusterFS: many-petabyte expandability, low cost per gigabyte, management console, appliance-like storage bricks, configurable replication level, as fast as your network allows, open source, etc. The whole enchilada.