up vote 1 down vote favorite
2
share [g+] share [fb]

I would like to create a cdn-like storage system, where my clients can upload/download files (average 300-400MB video files, divx files upto 2 GB) and even stream videos from them.. And i got scalability issue which i am thinking someone could help me to fix.

Here are the hardware specs of my servers:

Storage servers : 1TB with 100mbit bandwidth (to each other and to external networks) I got only 2 and after the demand increases, I would like to add more storage servers. (I would use raid10) generally dual core with at least 2GB ram.

Cache server: One super computer wtih about 12GB ram, 2-4 TB hdd, 1GBIT inbound and outbound connection speed to both storage servers and external networks. 4 cores.

All servers will be in the same switch.

So what softwares/file systems/ do you recommend me to give a life for a such service? Some files would ve very high demanded, and some files are very few demanded. I am thinking of to install Varnish to cache the video files in main cache server and mogileFS for the storage servers.

After too much traffic, I can easily add more storage servers and add another cache server and add a load balancer for two cache servers. (of course i can add more cache servers too)

So what do you recommend me to do? I dont know whether varnish servers can handle upto and their limits in terms of performance. And about storage servers, what do you recommend me to install?

Thanks

link|improve this question
Honestly, if you don't know what your doing already on of the established CDNs is the way to go. Building and supporting your own small scale CDN is going to end up bing more expensive than outsourcing it. – Chris S♦ Mar 7 '11 at 13:46
But i dont have full control over the objects in CDNs, example i want to only let my clients to access those files, and i want to count their bandwidth usage. thats why i am trying to find a good solution. (please dont tell me i can do access control to objects with aws, because we dont.) – TORr0t Mar 8 '11 at 10:42
feedback

migrated from stackoverflow.com Mar 7 '11 at 12:24

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 4 down vote

I'd like to point you to an interesting article that explains in detail how to build your own CDN using Varnish:

http://blog.unixy.net/2010/07/how-to-build-your-own-cdn-using-bind-geoip-nginx-and-varnish/

Note: Not sure whether that is what you need, though. You ask for how to create your own CDN, which the above linked article explains. In your text, however, you speak of "All servers will be in the same switch", which, well, contradicts the very idea of a CDN.

link|improve this answer
1  
Interesting article - but I agree I think the term "CDN" is being misused in the original question. – Coops Mar 7 '11 at 12:27
Yes you are right, but i am more likely trying to implement a private CDN network. Lets say, private cloud. Just want to be sure that the stream flow flow will be ebough so that the visitor wont have to stop and wait to load of the video. – TORr0t Mar 7 '11 at 12:37
feedback
up vote 0 down vote

Assuming you're looking for distributed storage and load balancing, you can use MogileFS and perlbal to serve the files from the available disk with the least IO load, and keep hot files in cache.

Alternatively, if you already run varhisn/nginx, you can use nginx with the MogileFS plugin (which acts as a full MogileFS client) to serve your files. You can also have nginx take care of the cache/reverse proxy part, or you can hand the files over to varnish.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.