How do you avoid that server documentation gets out of sync with the actual setup?

Question

We have have reasonably good documentation for our environment (in AsciiDoc format) which recently allowed another person to recreate the entire setup from scratch in less than 30 minutes.
However, I noticed that after the initial setup, it easily happens that small changes done to the system (say: inetd gets disabbled, my IMAP server listens on an additional port for ManageSieve connections, a new router is added to the exim configuration) don't end up in the documentation immediately (if at all).

My idea was to avoid this problem by (partially?) generating the documentation out of the configuration files and the comments therein - one way to implement this may be to put /etc and /usr/local/etc into some source code management system (say - git) and then run a script which regenerates the documentation on every commit. However, I'm not sure whether that would be overkill and/or too difficult to get right (after all, I don't want complete copies of the source files in my documentation but rather just the diffs).

How do other people avoid that the server documentation gets outdated - is there a good way to keep them in sync automatically, or do you just have the discipline to update the documentation the same time you modify the system?

Answer 1

You're never going to get away from some documentation but as you intimated there are systems that can be integrated into your change process to cover a lot of it.

• Use a config management tool (like puppet or chef).
• Store your config in a change controlled manner. (like git or SVN)
• Make sure config is readable/accessible by humans (i.e. plain text, searchable db)

This way the lower level documentation that we all normally miss (or don't bother with) is enforced by storing that deploy information in config items or code as a part of the system your making changes to. This also has an additional bonus of the process becoming more repeatable in the future.

External documentation does still need to be updated but it becomes very high level with pointers to "deploy x" or "deploy y" instead of long command/file listings. This additionally makes documentation changes both less frequent and easier which also means it will be more likely to get done.

Also before you go home brew, with puppet someone has probably already written something to manage what you want.

Answer 2

If you only admin one or two small systems, setting up a large configuration management system like puppet or chef seems like overkill. (Though, if you plan to have more systems in the future, do it now!)

For a small setup like this, I'd recommend using something like etckeeper, a program which puts /etc into a git repository and provides a few useful functions, like doing an automatic commit whenever you install, upgrade or remove a package.

Answer 3

You just have to update your documentation each time you make a change on the system. AKA Change Management.

The fact that most companies implement change management in such a ridiculous fashion as to make it worse than nothing should not detract from the utility of the basic concept or prevent you from doing it right.

I used to use html or some sort of wiki to track all my configs. Now I work in a Windows shop with (shudder) SharePoint, so now I use Word document "templates" I created to track every system I have and every config change I make, which isn't as bad as it sounds, given that many systems are just cookie-cutter copies of other ones that can all be lumped together into the same document. (And I keep local copies of all my stuff docs my hard drive, actually organized in a sensible fashion, in addition to throwing them onto the unorganized heap that is anyone's SharePoint site.)

The biggest challenge is really making the time to document, which I do by adding the documentation time as part of the time for making the change. So, not really that hard, especially if you're a bit of an ass and don't mind telling people to screw off and wait in line because you're too busy for their problem at the moment.