Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am looking for a project which would simplify EC2 on-demand instance management for script execution. Here is my scenario.

  • script waits for new job
  • my script reads execution parameters
  • it executes external python script for long-term processing
  • script handles output from python script
  • loops to the start (goto 10)

Instead of executing cpu intensive python script on the same server, i'd like to delegate it onto an on-demand EC2 instance. I am looking for a project to help me with the following:

  1. provide script built on top of ec2 for instance management
  2. i suppose, once instance is up, it might ssh into it and set up necessary environment, such as upload data file
  3. perform application execution in this new instance
  4. collect results form instance and terminate instance

I know that this can be all done through some insane scripting, but I'd prefer to use existing software for this. Please suggest a software packages / libraries or tools I can use.

share|improve this question

closed as not constructive by Chris S Mar 25 '13 at 13:33

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

Shopping Questions are Off-Topic on any of the Stack Exchange sites. See Q&A is hard, lets go Shopping and the FAQ for more details. – Chris S Mar 25 '13 at 13:33
up vote 2 down vote accepted

There are several libraries available for different programming languages that may help you to interact with AWS:


share|improve this answer

I've had good luck using boto (a python AWS-control library) for this kind of thing.

share|improve this answer

As one building block, have a look at the AWS SQS queue system. Your scripts can add "jobs" to it and your EC2 instances will read it from there.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.