Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I would like to create a cdn-like storage system, where my clients can upload/download files (average 300-400MB video files, divx files upto 2 GB) and even stream videos from them.. And i got scalability issue which i am thinking someone could help me to fix.

Here are the hardware specs of my servers:

Storage servers : 1TB with 100mbit bandwidth (to each other and to external networks) I got only 2 and after the demand increases, I would like to add more storage servers. (I would use raid10) generally dual core with at least 2GB ram.

Cache server: One super computer wtih about 12GB ram, 2-4 TB hdd, 1GBIT inbound and outbound connection speed to both storage servers and external networks. 4 cores.

All servers will be in the same switch.

So what softwares/file systems/ do you recommend me to give a life for a such service? Some files would ve very high demanded, and some files are very few demanded. I am thinking of to install Varnish to cache the video files in main cache server and mogileFS for the storage servers.

After too much traffic, I can easily add more storage servers and add another cache server and add a load balancer for two cache servers. (of course i can add more cache servers too)

So what do you recommend me to do? I dont know whether varnish servers can handle upto and their limits in terms of performance. And about storage servers, what do you recommend me to install?



migration rejected from stackoverflow.com Feb 18 at 15:33

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as too broad by TomTom, Falcon Momot, Jenny D, Ward, Katherine Villyard Feb 18 at 15:33

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

Honestly, if you don't know what your doing already on of the established CDNs is the way to go. Building and supporting your own small scale CDN is going to end up bing more expensive than outsourcing it. –  Chris S Mar 7 '11 at 13:46
But i dont have full control over the objects in CDNs, example i want to only let my clients to access those files, and i want to count their bandwidth usage. thats why i am trying to find a good solution. (please dont tell me i can do access control to objects with aws, because we dont.) –  TORr0t Mar 8 '11 at 10:42
The next one to close - no service recommendations, do your own homework. –  TomTom Feb 16 at 21:36

1 Answer 1

Assuming you're looking for distributed storage and load balancing, you can use MogileFS and perlbal to serve the files from the available disk with the least IO load, and keep hot files in cache.

Alternatively, if you already run varhisn/nginx, you can use nginx with the MogileFS plugin (which acts as a full MogileFS client) to serve your files. You can also have nginx take care of the cache/reverse proxy part, or you can hand the files over to varnish.


Not the answer you're looking for? Browse other questions tagged or ask your own question.