Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I need to download 100 GB of files (it’s in about 150 files) within a 7 day period before they expire. The download is rate-limited by the host so it takes MUCH longer than the theoretical transfer rate based on normal Internet speeds. I have a script of curl http://curl.haxx.se/docs/manpage.html commands - one line per file.

I had the idea of automatically spinning up n EC2 instances, executing the command and FTPing the files to a central location, then shutting down the machines. How would I do this?

I don't care whether it's Linux or Windows.

share|improve this question

4 Answers 4

up vote 6 down vote accepted

I would recommend using one of Canonical's AMIs in conjuction with cloud-init

https://help.ubuntu.com/community/CloudInit

Look for the section about "User-Data Scripts" where you can use the --user-data-file to auto-launch a script on startup. Thus, you can just put all the commands you want to run in the script and put "shutdown -h now" at the end of the script.

share|improve this answer
    
It looks like the official Amazon Linux AMI also supports CloudInit: aws.typepad.com/aws/2010/09/introducing-amazon-linux-ami.html –  dialt0ne Feb 5 '11 at 3:57

I've used boto to script such things before. Easiest to spin up linux images then ssh into them to have them do work.

share|improve this answer

Not sure of the best solution. But if you are doing this on a regular basis and need something to automatically start and stop your machines, then you may want to look at this product: http://LabSlice.com (with the disclaimer that I am the CTO).

share|improve this answer
    
your product looks great and your site looks terrific too. I accepted the other answer though because it was more specific about the scripts. –  Howiecamp Feb 2 '11 at 16:13

Vagrant with the AWS provider does that easily, and is multiplatform.

http://www.vagrantup.com/

https://github.com/mitchellh/vagrant-aws

Once the Vagrantfile is created, then you launch the instance with "vagrant up", login to it with "vagrant ssh", and destroy it with "vagrant destroy". Super-easy.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.