How do you configure Amazon EC2 auto-scaling to create instances programmatically based on a database event?

I have an application that processes several dozen large files (100MB each) on a monthly basis. Currently, once a month I'll manually create an instance per file, and then run a shell script to kick off the processing task on each instance and kill the instances once all processing is complete.

I want to automate this by configuring an EC2 auto-scale group to launch a new instance when a new database record is created, signifying a new file is ready to be processed. That instance will be seeded with code that will automatically connect to the database, find the new file, mark it as "in progress", and begin processing it.

I control when the files get added to the system, so I don't have to worry about some user adding 1000 files and exceeding my EC2 quota. However, I don't see any obvious way to configure auto-scale to work like this. It seems to only trigger instance creation based on CPU or ELB usage.


Your observations are correct - you will not be able to use Amazon's own AutoScaling controls for this type of event. You can however script it via Amazon's CLI tools.

You could poll for the file in any number of ways (e.g. inotify/incron, periodic SQL query to the database, or even an ls on the folder) and make an Amazon API request to increase the capacity of your AutoScaling Group by changing the Desired Capacity when the event is triggered.

Once the job is over, you can get the instance to auto-terminate itself - either by getting the Instance ID from the metadata URL and feeding it directly to the Amazon CLI, or by setting the instance to terminate on shutdown and issuing a shutdown -h now once your processing job is complete.

I don't know the detail of your implementation, so YMMV wildly here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.