Sign up ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I know it's possible to balance traffic using ELB/EC2 instances but is there any way to distribute CPU calculations over a cluster of instances.

I'm wanting to process lots of video/audio using ffmpeg, but if the server gets too slow, I'd like to be able to spawn another instance with the same setup to take some of the load from the main server. Think of it as a mini processing farm

Is what I'm looking to do possible with amazon, if it is, what should I be looking for in terms of documentation and AWS services? Thanks

share|improve this question
Take a look at Amazon's HPC information. It will give you some idea of the software others use to solve these sorts of problems. – Michael Hampton Aug 14 '12 at 18:43
See also – ceejayoz Mar 18 at 18:47

3 Answers 3

up vote 2 down vote accepted

In this regard, Amazon's EC2 instances are no different than any other hardware.

Yes, there are frameworks that help with cluster computing, but it's not simple, and there is surely no "magic" package you can install to make it work.

Your application will need to be written with parallelization/clustering in mind for this to have any chance of working, and whether or not it will work at all depends highly on your specific workload.

share|improve this answer
So would the best way of going about this (for a simple-ish setup) be to just spawn a few identical instances and pick one at random to process on? – Jamie Taylor Aug 14 '12 at 13:57
Yes. Additionally, you need to figure out where your bottleneck is and spin up an appropriately-provisioned instance to take the load. – EEAA Aug 14 '12 at 13:57
Well I already know where the bottleneck is, does EC2/Cloudwatch have anything to spawn new instances when cpu load is high? – Jamie Taylor Aug 14 '12 at 14:00
@JamieTaylor - yes, they have Auto Scaling groups. You'll still need to write the application logic to configure and hand out tasks to each node, though. – EEAA Aug 14 '12 at 14:13
That's fine, Thanks for your help, you've saved me many hours of googling. – Jamie Taylor Aug 14 '12 at 14:14

OP has already accepted @ErikA's answer, but I'm going to post a couple other thoughts. While it is true that EC2 instances offer nothing different from other hardware, the same is not true of AWS as a whole. There are several services specifically built for distributing computation and services in mini (or maxi-) processing farms.

There are some great resources available describing how to do video processing using the suite of AWS services. I would suggest starting with the Case Study and then digging in a bit deeper in some of the compute areas of the Architecture Center.

The combination of services available is unique: EC2 and Elastic Beanstalk for compute resources, S3 for large-scale blob storage, SimpleDB for small blob storage (metadata), and SQS/SWS to support messaging and task distribution. You can certainly design and deploy systems that do any or all of these things, but with an AWS account you can fire up the entire suite of tasks this afternoon.

share|improve this answer
Can't believe I haven't found the architecture centre before, seems to have all the info I'm looking for, especially this, Thanks – Jamie Taylor Aug 15 '12 at 7:57

A new answer to AWS distributed computing is AWS Lambda. It essentially lets you run JavaScript functions on demand, on different cloud machines, without you having to do any of the setup/teardown provisioning. You only pay for the the time the code is executed, and don't have to set up a cluster of server to wait for jobs.

AWS Lambda automatically runs your code on high availability compute infrastructure and performs all the administration of the underlying compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code and security patch deployment, and monitoring and logging. All you need to do is write the code.

share|improve this answer
I'm not sure Lambda would be an effective solution for ffmpeg video encoding. AWS Elastic Transcoder would probably wind up cheaper/easier. – ceejayoz Mar 18 at 18:53

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.