Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I don't have much experience as a sysadmin or with amazon aws, so I hope someone can explain in simple terms or refer me to a good guide on how to achieve the below.

I have a system running on ec2 and amazon rds getting data in and saving it to the db.

I need to run a script once a day (at the end of the day) to process all that data and prepare a daily report. This process will take approximately an hour to run. It needs to run on a high memory instance..

From what i've read so far, I guess the best way to do it is to have a high memory spot instance run every day, set it up to execute the script on startup and and shut down when done.

Is that the right way to do it? If so, how to do it? how to tell the spot instance to run every day? through a cron job on the other server or is there a better way? How to set it up to run the script on startup? through cloudinit?

Any help would be appreciated.

One last thing, the job is not very time sensitive as long as it runs every day..


share|improve this question
up vote 3 down vote accepted

Spot Instances may not be the best mechanism to use in your case. You require the process to execute every day, and most likely to complete execution once started.

Spot instances may not run when you want them to. If your bid price is too low, or there isn't capacity, then you won't get your instance. Also, once your instance has started, if the bid price rises above your maximum price, then the instance will be terminated before it's had time to complete.

You can use a high maximum bid price to try to avoid some of these issues, but there's no guarantees.

Depending on costs, you can take a look at reserved instances (in your case, a light reserved instance may suffice) or simply an on-demand instance that you only have running at the end of your day.

Amazon does not have any scheduling of instance start/stops, so you'll need to use something like cron or other automated scheduling tools.

share|improve this answer
Alright, so using a cron job and aws' API i startup an instance, what's the best way to run the script and let it shut down? do i do it on the script's server or from the cron server? thanks for your help. – chaft Jun 10 '12 at 10:24
The cron server is simply going to start your other instance. Everything else (including shutdown) would happen on the other instance. How you run your processing app is going to depend on the OS that's running on that instance. Once your processing is done, either use your OS's shutdown mechanism, or use the AWS API to stop the instance. – Matt Houser Jun 13 '12 at 14:21

Autoscaling with a schedule can be used to schedule a daily processing task. A guide is available here Running EC2 Instances on a Recurring Schedule with Auto Scaling

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.