How to automate an Amazon EC2 instance startup, execution of some commands and shutdown?

Question

I need to download 100 GB of files (it’s in about 150 files) within a 7 day period before they expire. The download is rate-limited by the host so it takes MUCH longer than the theoretical transfer rate based on normal Internet speeds. I have a script of curl http://curl.haxx.se/docs/manpage.html commands - one line per file.

I had the idea of automatically spinning up n EC2 instances, executing the command and FTPing the files to a central location, then shutting down the machines. How would I do this?

I don't care whether it's Linux or Windows.

Answer 1

I would recommend using one of Canonical's AMIs in conjuction with cloud-init

https://help.ubuntu.com/community/CloudInit

Look for the section about "User-Data Scripts" where you can use the --user-data-file to auto-launch a script on startup. Thus, you can just put all the commands you want to run in the script and put "shutdown -h now" at the end of the script.

Answer 2

I've used boto to script such things before. Easiest to spin up linux images then ssh into them to have them do work.

Answer 3

Not sure of the best solution. But if you are doing this on a regular basis and need something to automatically start and stop your machines, then you may want to look at this product: http://LabSlice.com (with the disclaimer that I am the CTO).

Answer 4

Vagrant with the AWS provider does that easily, and is multi-platform.

Once the Vagrantfile is created, then you launch the instance with vagrant up, login to it with vagrant ssh, and destroy it with vagrant destroy. Super-easy.