I am having real problems trying to wrap my head around my problem. I have decided on the middle and end solution to my problem but I can't seem to work out the the beginning of it.
So as a bit of a project I'm building a Youtube style site. My solution to the problem of uploading videos so far consists of:
- Using S3 to house 5 different encodings of each video
- Using an EC2 standard instance with MongoDB driven queueing system determined on free disk space on that instance compared to the file size of the next video in queue. So basically if there is 5x the space of the video on the instance it will start encoding that video (Maybe have 5 CPU heavy servers picking from a queue encoding one a a time, easy to scale). I have decided I need this temp space since the encoding itself cannot be performed on S3 (?) so I must bring the video from the S3 bucket down to the encoding server (EC2 instance) and then run 4 encodings paralel to each other and then upload those encoded files to S3 and then delete the temp encoded files on the EC2 instance.
The problem I got is how to handle the initial upload to server. So when a user goes onto the upload page how to initially get that video:
- On to S3 without losing space on my app server while storing information about that video (even possible? don't think so)
- Get it into queue
Now the main problem is space. I am trying to work out how to upload the file always knowing there will be enough space to initially boost that onto S3 (Thinking of many uploads per minute here). My worry is that I have to upload initially either to a super storage server like S3 but not CDN or I upload to the app server and hope for the best. Neither really seem super scalable in terms of both expansion and money so I am wondering how best is it to get the video uploaded initially?
If I must use my app server how best should I set it up to deal with these uploads? Should I just get a disk heavy app server?
If I were to do it another way outside of the app server how would I do it?
Sorry if my question seems a bit vague, this has given me a bit of a headache thinking about.
EIDT: I decided to use zencoder with S3. I think it's actually a really good idea I won't try and create this part myself as I don't really intend to properly host this site so it would just end up with me paying lots of money for something that might be better done third party.
Thanks for the help guys :D