Best practices for software location in building custom AMIs

Question

I've been experimenting with AWS, and learning about AMIs, how they run, and how to build them. I'm curious if anyone here has any thoughts on the following:

Is it better to build an AMI with all your needed software pre-installed in the image, or provide that software via EBS? I'm thinking that it might be better to have a central location for things like apache, or mysql binaries, assuming you would want multiple web/database servers available to be easily brought up. I'm old-school enough that I'm thinking of using EBS to construct something that looks a bit like an NFS exported /usr/local for all my AMIs.

That seems like a good use of EBS - and a way to keep AMI size down. But maybe there's a reason not to do it this way.

Answer 1

Your question is one that comes up fairly often it seems.

It seems the ultimate answer is everyone's environment is different and has different requirements etc. However, here is how I have proceeded and what has/hasn't worked.

1. Go with EBS backed instances unless you truly will never need the instance back, it makes many things such as bundling and data transfer much simpler.

2. Assuming you are going to be using fairly static software versions (I.E. you are not using the nightly bleeding edge release of the software) install the software on the image but not configurations.

3. Version control your configurations and have these be updated as the instance comes up. In linux and windows it is fairly easy to have a script run on startup, contact a central repository and update your code, configs etc. This can even be extended to bringing up new servers, just have them pull a base config from the repository and modify it in place. This makes it easier to manage smaller server changes without having to rebundle everything and retest.

4. Store data on an attached EBS volume. On linux, the main OS should fit easily onto the default 8GB EBS root volume with room for some simple scripts etc. Things such as large code collections, static files, databases etc should go onto an EBS store. What is stored on the attached volumes should be server agnostic. For example. if you have 3 web servers, the data device attached to each of them should able to be identical and not tied to a specific instance. This makes backup and recovery from failure much easier.

5. Unless your are in the VPC and have static internal IP addresses you should set up a private DNS server and have scripts manage updating this (as well as security groups) as the instances are turned on/off. Due to the nature of EC2 and non static IPs the individual servers should avoid referring to these as much as possible since they will be different every time they start.

TL:DR The AMIs should only contain the info that is not instance specific or a large data set, the rest should be assumed to be changeable and managed accordingly.