Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am always interested in how IT teams approach planning Production changes. Typically we use runbooks to layout the key steps of a change as well as backout plans. I am keen to see whether we can learn from others and how best to document runbooks.

share|improve this question
    
WTF is a runbook? –  nray Jul 10 '09 at 22:13

4 Answers 4

What I do:

All server configuration that needs to change from the installed OS baseline is managed by Chef, which is stored in modules (called cookbooks), which are then stored in version control via Git.

Most configuration was done manually on a test system (often a VM image or simply an EC2 instance), and then a configuration recipe written to cover all the various components of the changes. Updating the environment workflow goes something like this:

  • Create a ticket in the appropriate system that a change needs to be made.
  • Document all the whys and whats of the change.
  • Edit the configuration recipe(s), templates, files, etc, required to make the change occur on the target system or systems.
  • Commit the change to the local repository and push out to the master version control server.
  • Update the ticket for peer review of the changes.
  • Changes are signed off, and the change is deployed to the Chef server so it knows about the updated bits.
  • Run Chef manually on clients, or let it run automatically depending on the requirement of the change. (I wouldn't run on say more than a half dozen systems by hand).

Chef's mode of operation is to fail if there's an issue running the client, like if a package doesn't exist, or a template file isn't found, or many other issues. Fix the issue, document in the ticket, and then rerun the client.

The person with the business requirement for the change verifies that it was successful, and they close the ticket.

Chef specific because that's what I use. Subsitute the appropriate tool for your environment, and if you're not using a configuration management tool, you need to look at something, because it makes this whole process a lot more robust and reliable. Not to mention scalable.

share|improve this answer
    
Sounds like a very sensible approach - Chef appears a very powerful toolset –  Julie Washington Jul 13 '09 at 21:07

I prefer scripts with comments and prints.
They have a double advantage of documenting and automation.

But, taking a more holistic approach, there is usually a lot of things to track,
scripting is only for things that need to be done in a sequence.

When there is a lot of notes involved I prefer a locally hosted Wiki (personal or group).
It can be used to

  1. refer your tools with links and write quick notes
  2. enumerate contacts and escalation references
  3. document emergency steps based on keywords
  4. backup locations and recovery sequences
  5. host a searchable record of regular requirements and solutions; so people come to you after they have checked that out

But, just keep the location safe -- you don't want the data to be inaccessibly in an emergency.

Here is an old Microsoft Technet SQL Server runbook page for catching generic ideas.

share|improve this answer

For really critical and sensitive changes, I will typically have a text file with the actual commands I'll use with #comments explaining what's going on. That way, I can cut-n-paste them into a terminal quickly.

share|improve this answer
    
Sounds like a shell script to me. :) –  Ernie Jul 10 '09 at 22:26
    
Touché! I didn't even notice that. It's a slowly executed shell script. :) –  Geoff Fritz Jul 10 '09 at 22:53
    
I tend to do the same thing but these things end up documented in our wiki. –  3dinfluence Jul 11 '09 at 5:16

I'd second the basic idea behing jtimberman's post, with puppet being my tool of choice.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.