I currently build and test a large Java product based on Eclipse an Ubuntu machine using Jenkins (Hudson) and tested on both Ubuntu and Windows, the testing also involves all of the major databases. I want to move all of this to the cloud and want to get some feedback as the best way to do this. The build process has a number of Hudson jobs that must go in sequence, and the test has several jobs that can run in parallel.
I'm leaning toward using Amazing EC2/S3/EBS to do all of this, but I'm open to a better solution.
There is the following sorts of data involved with this process:
- The system software like Java, Mercurial, databases, etc. This would be part of the AMI.
- The software required for build and test execution (mainly a particular version of Eclipse), this needs to be read by all jobs. I'm thinking of using S3 with s3fs for this, as this will be accessed by several instances in parallel. This is effectively read-only. This would be updated by non-EC2 instances, e.g. my development machine.
- A small amount of data indicating the built timestamp which is generated by the first step in the process and shared by all downstream jobs. I'm thinking of using something like Ubuntu One for this (just accessing it though some command).
- The location of the build output artifacts, written by each build step and consumed on a read-only basis by the tests. I'm thinking of S3 with s3fs for this. Also, I would want these to be available to non-EC2 instances.
- The work area for each test run. This will be heavy reading and writing, so s3fs will not be suitable (due to its consistency issues). This also does not need to be shared, though it will rarely be accessed from my (non-EC2) development machine when something goes wrong (which means I will need to start an EC2 instance to mount this and get the files that I need since I don't think you can mount EBS from anything but an EC2 instance). So the choice here would be a persistent EBS store.
As far as tools to manage this, I'm just using the browser interface to AWS, and possibly the Eclipse tools (though they don't look too much better than the browser interface). Are there better tools I should consider using?
I imagine I will probably need to create a custom AMI to get what I need for at least the build steps, and hoping that I can use an out of the box AMI for each of the databases (rather than a single AMI where I have to install all of the databases and manage that myself).
I don't think any of this is going to cost a huge amount of money, even if several builds are done a day, so I'm not particularly cost sensitive (if there is a better but more expensive way do to this, I'm interested).
I'm open to any suggestions and also confirmation that I'm on the right track if that's the case.
