Build solution for Jenkins with Amazon EC2 - How to keep state between builds

Question

I'm trying to move my build environment to EC2 using Jenkins. I have a lot of source to check out each time, so I want to preserve the source persistently between builds.

This would be easy if Jenkins stopped/started the EC2 instances when it needed them (as I could just associate the AMI with a root EBS volume). But Jenkins terminates the instances when not being used.

I have tried making a separate EBS volume and attaching it to the EC2 instance using the startup commands with Jenkins and the problem there is it's possible for the EBS volume and EC2 instance to be in different availability zones. And Jenkins cannot control the availability zone when starting the instance.

So is there a way to keep the state between builds using the Jenkins/EC2 interface. Snapshots perhaps?

See this non-question from me for a bit more context.

Answer 1

I took a look at the source code of Jenkins and that's a reasonable fix to it. It won't take much hacking to add the option of selecting which availability zone in a region one wants or to change termination to a simple stop. I think that's the best approach here.

If sufficiently motivated, I might post a patch. Source code is at https://github.com/jenkinsci/ec2-plugin/ if you're handy with Java.

Answer 2

I would suggest looking at a deployment management tool for this. A growing number of people in the pallet community use it in conjunction with Jenkins to do exactly the type of thing you are describing.

Pallet is a clojure framework similar to Chef/Puppet that uses JClouds to interface with Amazon, Rackspace or other cloud providers. Using Maven Pallet can be invoked by Jenkins to deploy an entire cluster if needed to run a test suite. Doing what you describe would be entirely possible, but at this time would require some specific knowledge of the workings of JClouds and Pallet to implement.

If all you need to do is exactly what you described, ec2-plugin modification is probably the best way to go. If you need to push a little further and you want maintainable instances, you might take a look at Pallet + Jenkins.