I know it's possible to balance traffic using ELB/EC2 instances but is there any way to distribute CPU calculations over a cluster of instances.

I'm wanting to process lots of video/audio using ffmpeg, but if the server gets too slow, I'd like to be able to spawn another instance with the same setup to take some of the load from the main server. Think of it as a mini processing farm

Is what I'm looking to do possible with amazon, if it is, what should I be looking for in terms of documentation and AWS services? Thanks


In this regard, Amazon's EC2 instances are no different than any other hardware.

Yes, there are frameworks that help with cluster computing, but it's not simple, and there is surely no "magic" package you can install to make it work.

Your application will need to be written with parallelization/clustering in mind for this to have any chance of working, and whether or not it will work at all depends highly on your specific workload.

  • 1
    So would the best way of going about this (for a simple-ish setup) be to just spawn a few identical instances and pick one at random to process on? Aug 14 '12 at 13:57
  • Yes. Additionally, you need to figure out where your bottleneck is and spin up an appropriately-provisioned instance to take the load.
    – EEAA
    Aug 14 '12 at 13:57
  • Well I already know where the bottleneck is, does EC2/Cloudwatch have anything to spawn new instances when cpu load is high? Aug 14 '12 at 14:00
  • @JamieTaylor - yes, they have Auto Scaling groups. You'll still need to write the application logic to configure and hand out tasks to each node, though.
    – EEAA
    Aug 14 '12 at 14:13
  • That's fine, Thanks for your help, you've saved me many hours of googling. Aug 14 '12 at 14:14

OP has already accepted @ErikA's answer, but I'm going to post a couple other thoughts. While it is true that EC2 instances offer nothing different from other hardware, the same is not true of AWS as a whole. There are several services specifically built for distributing computation and services in mini (or maxi-) processing farms.

There are some great resources available describing how to do video processing using the suite of AWS services. I would suggest starting with the Encoding.com Case Study and then digging in a bit deeper in some of the compute areas of the Architecture Center.

The combination of services available is unique: EC2 and Elastic Beanstalk for compute resources, S3 for large-scale blob storage, SimpleDB for small blob storage (metadata), and SQS/SWS to support messaging and task distribution. You can certainly design and deploy systems that do any or all of these things, but with an AWS account you can fire up the entire suite of tasks this afternoon.

  • Can't believe I haven't found the architecture centre before, seems to have all the info I'm looking for, especially this, Thanks Aug 15 '12 at 7:57

A new answer to AWS distributed computing is AWS Lambda. It essentially lets you run JavaScript functions on demand, on different cloud machines, without you having to do any of the setup/teardown provisioning. You only pay for the the time the code is executed, and don't have to set up a cluster of server to wait for jobs.

AWS Lambda automatically runs your code on high availability compute infrastructure and performs all the administration of the underlying compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code and security patch deployment, and monitoring and logging. All you need to do is write the code.


  • I'm not sure Lambda would be an effective solution for ffmpeg video encoding. AWS Elastic Transcoder would probably wind up cheaper/easier.
    – ceejayoz
    Mar 18 '15 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.