I am looking for a solution for a project i am working on.

We are developing a websystem where people can upload their files and other people can download it. (similar to rapidshare.com model)

Problem is, some files can be demanded much more than other files. The scenerio is like: I have uploaded my birthday video and shared it with all of my friend, I have uploaded it to myproject.com and it was stored in one of the cluster which has 100mbit connection.

Problem is, once all of my friends want to download the file, they cant download it since the bottleneck here is 100mbit which is 15MB per second, but i got 1000 friends and they can only download 15KB per second. I am not taking into account that the hdd is serving same files.

My network infrastrucre is as follows: 1 gbit server(client) and connected to 4 Nodes of storage servers that have 100mbit connection. 1gbit server can handle the 1000 users traffic if one of storage node can stream more than 15MB per second to my 1gbit (client) server and visitor will stream directly from client server instead of storage nodes. I can do it by replicating the file into 2 nodes. But i dont want to replicate all files uploadded to my network since it is costing much more.

So i need a cloud based system, which will push the files into replicated nodes automatically when demanded to those files are high, and when the demand is low, they will delete from other nodes and it will stay in only 1 node.

I have looked to gluster and asked in their irc channel that, gluster cant do such a thing. It is only able to replicate all the files or none of the files. But i need it the cluster software to do it automatically.

Any solutions ? (instead of recommending me amazon s3)



I'd fix the design first rather than look to add needless complexity to your existing design.

Specifically why does one server have a 1Gbps link but the other servers only 100Mbps? Put them all on the same non-blocking switch in front of your FW/LB/Proxies - that'll solve your replication problem in one go.

Oh and I'd love to know how to get 15MBps down a 100Mbps link, are you working in a 7-bit world? :)

  • 1
    +1, Bad design = Problems; Throwing resources at problems doesn't fix the design. (I want 15MBps too! I'm suck around 9.5 after various overheads.) – Chris S May 24 '10 at 15:22
  • Problem is , 1gbps server is very expensive, but not the 100mbit ones. Thus, i will use 1gbps server to setup client and from that server, i will stream the files to visitors. I dont have access to switches or something. I will rent dedicated servers from a provider. Yes 100Mbps is about 12.5 MB per second which i averaged in 15MB :) So what is your recommendadion for the software usage ? Thanks – TORr0t May 24 '10 at 17:03
  • I genuinely believe you need to seriously rethink this design, I build content delivery systems and I worry for your system. – Chopper3 May 24 '10 at 18:25
  • System wouldnt be streaming videos or something, just i want the visitors able to download files from my website. Content delivery system is very expensive, and there is not any company that provides that system for free and i cant do as well. I cant replicate a file in many servers, many different DC. I just have 1 DC house to rent my dedicated servers, and i want to copy the content to different nodes, if those files are demanded first. May you please tell me why you believe since you had more experience in that field than me. Thanks – TORr0t May 24 '10 at 20:41
  • 1
    I didn't suggest that I have more experience than you, since you mention it I probably have as I've been building large IT platforms for over twenty years but my comments are more based around the fact that you appear to want to fix a data-path design issue with a replicating file system - I think your just going about it the wrong way - sorry if you disagree but why come here if you're not interested in listening to others experience? – Chopper3 May 24 '10 at 22:26

It sounds like you are describing a reverse proxy setup. Take a look at the Wikipedia article and perhaps the nginx server - you can also get a similar result with squid or apache-httpd.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy