Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like to serve videos from a pool of servers and have a minimum of 2 copy of each to prevent server crash and lose data. The goal is to have a kind of cloud where I can add as many server as I want. I checked openstack, glusterfs, moosefs but their solutions seems not to be my needs as some need proxies, other chunks the datas, etc..

Actually I have 5 servers on wich I check at every video upload the space left on disk and then distribute the video on at least 2 servers, then save theses servers names in a db and generate 2 urls that I randomly call for balancing. (ie, )

This is working fine except for 2 things:

  • delete a video > I have to log to each server and then delete the video, I can make a script to do that anyway
  • move content, if I want to move content from one server to another it is a bit pain as I have to change the server url in db and to move the video. I can do a script for that also.

I can continue like this and complete the missing scripts but I'm sure there are existing systems better than mine, more robust that I would ever do.

I do not want to use services like openstack as it need a lot of servers (actually for my configuration it would mean my 5 servers and 5 more servers for proxies) or maybe I'm wrong.

Any help is welcome.

share|improve this question

closed as not constructive by Michael Hampton, faker, Ward, mdpc, Khaled Dec 27 '12 at 14:38

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

I think we should add a section the FAQ that points out the inappropriateness of questions like "I need to get a large distributed service started but I don't know much about these things; someone help me." – mfinni Dec 26 '12 at 17:59

since videos are static.. why not just use somehting like s3 and cloudfront as your cdn to serve the videos? Seems to make more sense then spinning up servers and trying to replicate the data that way. Using servers just creates more risk and work then using s3 with cloudfront.

share|improve this answer
I do not use S3 for many reasons. The cost is very expensive because videos uses a lot of bandwidth, actually I have 600 Mbps and it is growing. They can cut your account for any reason so I will never trust this kind of company I prefer to have my pool of server on differrents datacenters. – kollo Dec 26 '12 at 17:42
The cost is cheaper then running your own servers and worrying about backups. I run a photo sharing website and converted all images to s3 and we cut our expenses by 65% getting rid of our image servers. Since we didn't worry about raided drives and backups. Scale is not cheap.. it never will be cheap or easy. You have to pay for it – Mike Dec 26 '12 at 19:00
  1. You do want distributed storage; don't try to roll it yourself. That will handle the "delete all the copies" issue, plus a lot of other problems you haven't yet considered. Open source, Isilon, something - depends on your exact needs and budget.
  2. You do need some sort of front-end proxy for a service like this - that could just be Apache or what have you. Further, whatever your front-end is should be a load-balanced cluster as well.
  3. I'm not intimately familiar with OpenStack, but I'm pretty certain that you're reading it wrong.

Those answers are suited for essentially any commercial service. If you're doing this as a learning exercise, but aren't interested in learning the professional practices that go with setting up something like this, you're on the wrong site. In fact, this question is so open-ended (as I mention in my comment), that it's not really well-suited for this site in the first place.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.