I would like to create a cdn-like storage system, where my clients can upload/download files (average 300-400MB video files, divx files upto 2 GB) and even stream videos from them.. And i got scalability issue which i am thinking someone could help me to fix.
Here are the hardware specs of my servers:
Storage servers : 1TB with 100mbit bandwidth (to each other and to external networks) I got only 2 and after the demand increases, I would like to add more storage servers. (I would use raid10) generally dual core with at least 2GB ram.
Cache server: One super computer wtih about 12GB ram, 2-4 TB hdd, 1GBIT inbound and outbound connection speed to both storage servers and external networks. 4 cores.
All servers will be in the same switch.
So what softwares/file systems/ do you recommend me to give a life for a such service? Some files would ve very high demanded, and some files are very few demanded. I am thinking of to install Varnish to cache the video files in main cache server and mogileFS for the storage servers.
After too much traffic, I can easily add more storage servers and add another cache server and add a load balancer for two cache servers. (of course i can add more cache servers too)
So what do you recommend me to do? I dont know whether varnish servers can handle upto and their limits in terms of performance. And about storage servers, what do you recommend me to install?