I need to download 100 GB of files (it’s in about 150 files) within a 7 day period before they expire. The download is rate-limited by the host so it takes MUCH longer than the theoretical transfer rate based on normal Internet speeds. I have a script of curl http://curl.haxx.se/docs/manpage.html commands - one line per file.
I had the idea of automatically spinning up n EC2 instances, executing the command and FTPing the files to a central location, then shutting down the machines. How would I do this?
I don't care whether it's Linux or Windows.