Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For a video converter [using FFMPEG on linux [ ubuntu ] ] , and media files grabber [ from youtube and other video hosting websites] .. What is the best architecture to use in EC2?

By architecture I mean , the types/numbers of instances to use .. Should I host the application logic on multiple small instances , or use an extra large one ? should the data go to a RDS instance or I can host it with application logic?

If it is the first option , how to do it , communications between instances ..etc ?

This architecture should start small , and scales out or up as necessary

Please help!

share|improve this question

closed as not a real question by sysadmin1138 Nov 14 '11 at 22:27

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 3 down vote accepted

Scalability is a hard thing to give a best-practices document for. Each application is different. Some workflows lend themselves very well to massive parallelization, others have unavoidable single-points-of-processing that slow the entire set down. In order to recommend such a 'best' solution, the following needs to be known:

  • A high-level overview of the processing stages data follows.
    • Each stage in the process needs to be analyzed for:
      • Parallization
      • Failure tolerance
      • Dependencies with other stages
  • A strong understanding of your uptime requirements, how long you can afford to be down, how much data-loss is tolerable (if any).
  • A strong understanding of how failure occurs in your system and how it responds
    • How much data gets lost when something goes wrong?
    • How long does it take to re-process the data that experienced the failure?
  • How tolerant you are of cost. For EC2, the variables:
    • How much data gets transferred into and out of the EC2 system during the lifecycle of processing.
    • Need for multiple availability-zones or multiple regions
      • Data transfer required between regions
    • How many instance-hours hours each unit of data costs for processing.
    • Costs for any specialized infrastructure (load-balancers, elastic search, etc.)

And so on. None of these we can tell you. Or even give you guidance on since they're unique to the system you are building, and to reveal most of the above will requiring revealing the secret-sauce for your system (something I think you'd rather not do).

share|improve this answer
Thank you , but is there any guide / tutorial to help me decide ? – Naughty.Coder Nov 14 '11 at 22:51
"Scalability is a hard thing to give a best-practices document for." – Bart Silverstrim Nov 14 '11 at 22:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.