Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I know it's possible to balance traffic using ELB/EC2 instances but is there any way to distribute CPU calculations over a cluster of instances.

I'm wanting to process lots of video/audio using ffmpeg, but if the server gets too slow, I'd like to be able to spawn another instance with the same setup to take some of the load from the main server. Think of it as a mini processing farm

Is what I'm looking to do possible with amazon, if it is, what should I be looking for in terms of documentation and AWS services? Thanks

share|improve this question
Take a look at Amazon's HPC information. It will give you some idea of the software others use to solve these sorts of problems. –  Michael Hampton Aug 14 '12 at 18:43

2 Answers 2

up vote 1 down vote accepted

In this regard, Amazon's EC2 instances are no different than any other hardware.

Yes, there are frameworks that help with cluster computing, but it's not simple, and there is surely no "magic" package you can install to make it work.

Your application will need to be written with parallelization/clustering in mind for this to have any chance of working, and whether or not it will work at all depends highly on your specific workload.

share|improve this answer
So would the best way of going about this (for a simple-ish setup) be to just spawn a few identical instances and pick one at random to process on? –  Jamie Taylor Aug 14 '12 at 13:57
Yes. Additionally, you need to figure out where your bottleneck is and spin up an appropriately-provisioned instance to take the load. –  EEAA Aug 14 '12 at 13:57
Well I already know where the bottleneck is, does EC2/Cloudwatch have anything to spawn new instances when cpu load is high? –  Jamie Taylor Aug 14 '12 at 14:00
@JamieTaylor - yes, they have Auto Scaling groups. You'll still need to write the application logic to configure and hand out tasks to each node, though. –  EEAA Aug 14 '12 at 14:13
That's fine, Thanks for your help, you've saved me many hours of googling. –  Jamie Taylor Aug 14 '12 at 14:14

OP has already accepted @ErikA's answer, but I'm going to post a couple other thoughts. While it is true that EC2 instances offer nothing different from other hardware, the same is not true of AWS as a whole. There are several services specifically built for distributing computation and services in mini (or maxi-) processing farms.

There are some great resources available describing how to do video processing using the suite of AWS services. I would suggest starting with the Encoding.com Case Study and then digging in a bit deeper in some of the compute areas of the Architecture Center.

The combination of services available is unique: EC2 and Elastic Beanstalk for compute resources, S3 for large-scale blob storage, SimpleDB for small blob storage (metadata), and SQS/SWS to support messaging and task distribution. You can certainly design and deploy systems that do any or all of these things, but with an AWS account you can fire up the entire suite of tasks this afternoon.

share|improve this answer
Can't believe I haven't found the architecture centre before, seems to have all the info I'm looking for, especially this, Thanks –  Jamie Taylor Aug 15 '12 at 7:57

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.