Looking for software suite to delegate CPU intensive tasks onto EC2 on-demand instances

Question

I am looking for a project which would simplify EC2 on-demand instance management for script execution. Here is my scenario.

• script waits for new job
• my script reads execution parameters
• it executes external python script for long-term processing
• script handles output from python script
• loops to the start (goto 10)

Instead of executing cpu intensive python script on the same server, i'd like to delegate it onto an on-demand EC2 instance. I am looking for a project to help me with the following:

1. provide script built on top of ec2 for instance management
2. i suppose, once instance is up, it might ssh into it and set up necessary environment, such as upload data file
3. perform application execution in this new instance
4. collect results form instance and terminate instance

I know that this can be all done through some insane scripting, but I'd prefer to use existing software for this. Please suggest a software packages / libraries or tools I can use.

Answer 1

There are several libraries available for different programming languages that may help you to interact with AWS:

• boto (Python)
• right_aws (Ruby)
• fog (Ruby)

HTH

Answer 2

I've had good luck using boto (a python AWS-control library) for this kind of thing.

Answer 3

As one building block, have a look at the AWS SQS queue system. Your scripts can add "jobs" to it and your EC2 instances will read it from there.