Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am approaching the need to scale a video-processign application that runs on EC2.

So far the setup is one machine:

Backbonejs frontend Rails 3.2 Postgresql Resque + S3 for storage

The flow of the app is as follows:

1) Request from frontend. Upload a video. 2) Storing video 3) Quering external APIs. 4) Processing / encoding videos. 5) Post to frontend.

I can separate the backend and frontend without any problems, but when it comes to distributing the backend between several servers I am a bit puzzled. I can probably come up with a temporary solution (like just duplicating apps making several instances), but since I don't really have expertise in backend system administration, there can be some fundamental mistakes.. Also I would rather have something that is scalable. I wonder if anyone can give some feedback on the following plan:

A) Frontend machine. Just frontend, talks to backend via REST Api of sorts. B) Backend server (BS), main database. Gets request from 1), posts to 2) saves uploads to 3) C) S3 storage. D) Server for quering APIs. Basically just a Resque workers, that post info back to 2) E) Server for video encoding. Processes videos uploaded on 3) and uploads them back.

So I will have:

A)frontend
          \       
           \      
            B)MAIN_APP/DB ----- C)S3 Storage (Files)
             /          \              /
            /            \            /
D)ExternalAPI_queries    E)Video_Processing
   (redundant DB)           (redundant DB)

All this will supposedly talk to each other via HTTP requests.

My reason for this is that Video Processing part is really the most resource-intensive and I would just run barebones application that accepts requests and starts processing them.

Questions:

1) In this setup I will have the main database at B) and all other servers will communicate with it via HTTP requests (and store duplicates of databases also I guess..for safety reasons). Is it the right approach or should I have 1 database that everyone connects to (how then?)

2) Is it a good idea to separate API queries from Video Processing part? Logically they are very close (processing is determined by the result of API queries), but resource-wise Video Processing is waaay more intensive.

3) what should I use to distribute calls between backend apps based on load?

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.