8

I need to download 100 GB of files (it’s in about 150 files) within a 7 day period before they expire. The download is rate-limited by the host so it takes MUCH longer than the theoretical transfer rate based on normal Internet speeds. I have a script of curl http://curl.haxx.se/docs/manpage.html commands - one line per file.

I had the idea of automatically spinning up n EC2 instances, executing the command and FTPing the files to a central location, then shutting down the machines. How would I do this?

I don't care whether it's Linux or Windows.

7

I would recommend using one of Canonical's AMIs in conjuction with cloud-init

https://help.ubuntu.com/community/CloudInit

Look for the section about "User-Data Scripts" where you can use the --user-data-file to auto-launch a script on startup. Thus, you can just put all the commands you want to run in the script and put "shutdown -h now" at the end of the script.

1

I've used boto to script such things before. Easiest to spin up linux images then ssh into them to have them do work.

0

Not sure of the best solution. But if you are doing this on a regular basis and need something to automatically start and stop your machines, then you may want to look at this product: http://LabSlice.com (with the disclaimer that I am the CTO).

  • your product looks great and your site looks terrific too. I accepted the other answer though because it was more specific about the scripts. – Howiecamp Feb 2 '11 at 16:13
0

Vagrant with the AWS provider does that easily, and is multi-platform.

Once the Vagrantfile is created, then you launch the instance with vagrant up, login to it with vagrant ssh, and destroy it with vagrant destroy. Super-easy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.