I am having real problems trying to wrap my head around my problem. I have decided on the middle and end solution to my problem but I can't seem to work out the the beginning of it.

So as a bit of a project I'm building a Youtube style site. My solution to the problem of uploading videos so far consists of:

  • Using S3 to house 5 different encodings of each video
  • Using an EC2 standard instance with MongoDB driven queueing system determined on free disk space on that instance compared to the file size of the next video in queue. So basically if there is 5x the space of the video on the instance it will start encoding that video (Maybe have 5 CPU heavy servers picking from a queue encoding one a a time, easy to scale). I have decided I need this temp space since the encoding itself cannot be performed on S3 (?) so I must bring the video from the S3 bucket down to the encoding server (EC2 instance) and then run 4 encodings paralel to each other and then upload those encoded files to S3 and then delete the temp encoded files on the EC2 instance.

The problem I got is how to handle the initial upload to server. So when a user goes onto the upload page how to initially get that video:

  • On to S3 without losing space on my app server while storing information about that video (even possible? don't think so)
  • Get it into queue

Now the main problem is space. I am trying to work out how to upload the file always knowing there will be enough space to initially boost that onto S3 (Thinking of many uploads per minute here). My worry is that I have to upload initially either to a super storage server like S3 but not CDN or I upload to the app server and hope for the best. Neither really seem super scalable in terms of both expansion and money so I am wondering how best is it to get the video uploaded initially?

If I must use my app server how best should I set it up to deal with these uploads? Should I just get a disk heavy app server?

If I were to do it another way outside of the app server how would I do it?

Sorry if my question seems a bit vague, this has given me a bit of a headache thinking about.


EIDT: I decided to use zencoder with S3. I think it's actually a really good idea I won't try and create this part myself as I don't really intend to properly host this site so it would just end up with me paying lots of money for something that might be better done third party.

Thanks for the help guys :D

  • Have you considered just using a hosted service, something like Zencoder.com? It interfaces nicely with S3. – ceejayoz Apr 16 '12 at 17:54
  • Its not so much the encoding I got that sorted, its the uploading part that's got me, how to upload initially without expending all my disk space on say my app server, should I upload straight to S3 (if possible?) or should I upload some other way? – Sammaye Apr 16 '12 at 17:57
  • I mean I would rather do it myself cos it's a project, I am learning so I am trying to avoid hosted solutions. – Sammaye Apr 16 '12 at 17:58
  • 2
    Somehow I think this will get expensive... – voretaq7 Apr 16 '12 at 18:01
  • Indeed either way hosting videos is mega expensive. But how would you go about the initial push to S3 or another CDN? – Sammaye Apr 16 '12 at 18:04

You can allow users to upload directly to your S3 bucket, see Browser Uploads to S3 using HTML POST Forms for an example.

| improve this answer | |
  • Ah so I can, I didn't see that. Thanks! Though if people can upload like that couldn't that make the form easily abusable? – Sammaye Apr 16 '12 at 18:07
  • No more than having a submission form going to EC2 instances... – ceejayoz Apr 16 '12 at 18:10
  • True though I can't house PHP to drop the file should it be malicious can I? I see only the ability to redirect saying the upload is done. Which means if it gets abused the file will upload and save no matter what and I must come bakc later, hmmmm maybe that's what youtube do, they just have a very quick queue that gets it fast and removes it quickly. – Sammaye Apr 16 '12 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.