Document the heck out of everything.
There was a thread on Slashdot recently about starting documentation, which inspired me to write down my thoughts about documentation.
My key points were:
Principle #1: It Is Never Done
Documentation is an on-going effort that will always lag behind what is in production. Changes are made ad-hoc, things moved around or discontinued or put into service at random. Documentation will never catch up.
You have to sell the people paying the bills on the value of spending time (and therefore, money) on keeping the running documentation up to date. Frequently those conversations go like this: "remember when I had to spend $TIME figuring out how $THING was broken? Well when I was finished, there was this tech note detailing $THING, so that the next guy to come along won't have to figure it all out."
You have to do it, even though you will never finish.
Principle #2: The Only Thing Worse Than No Documentation Is Wrong Documentation
This is more of a truism than a principle. Documentation can lull you into the false sense that something is in a known state and that if something goes wrong you can therefore have a running start at fixing it.
It is important to acknowledge this problem.
Principle #3: You Are Writing Documentation For Your Successor
Odds are 95% of anything you do document you will never have to refer to again. Documentation is a collection of wisdom for the future, not for you. So you have to assume that your audience knows little or nothing about the specifics of how things are the way they are.
And there will be a successor. I don't know about you, but I don't plan to be in these specific environments for the rest of my life. Opportunities come and go, and when they come, sometimes you go. But life goes on behind you, and the smoother you can make life for your successor the better. Otherwise you might have a collection of former customers who quietly say unflattering things about you. I like to say that it is the same 50 guys working everywhere in IT in Ottawa because you keep running into them everywhere. Helping your successor might open doors for you in the future.
Now to a certain extent there is always a degree of "blame the previous guy" when trouble comes up. That is part of the business. I've done it myself. But on several occasions when I had blasted the previous guy as some kind of moron, I have learned otherwise that he really had his act together and knew more about what was going on than I did at the time.
Principle #4: "Why" is often more important than "How"
When looking at a system most of us start thinking thinks like why the hell is this like this? There are almost always very specific reasons for the configuration choices made. In these circumstances, the "Why" dictates the "How", and you have to make sure that the reader understands the specific problems being solved when examining the smoking remains of your solution.
Principle #5: It has to be easy or you won't do it
This means you have to be very aware of your tools as well as those who are going to use your tools.
Keeping things up-to-date has to be easy. If you have to make any kind of effort, then you will find excuses to avoid doing it when it is best done, which is immediately after a change.
If your tools are not easy for others to use, then they won't use it. This can be especially crippling in a team environment, since the larger the team gets the more likely you will encounter a team member who does not like your choice of tools.
Personally, I like a wiki for documents. However the problem is that a wiki does not force a structure on you, so the structure must be imposed from outside. This always leads to conflict somewhere as somebody else has a better/different idea.
In some places I've used Word and Visio documents "published" to PDF, with the "latest" PDF being considered authoritative. This is good in that you then have a collection that you can hand to your employer/successor. The PDFs, if properly dated, can provide a historical record of what happened, although one which is not easy to navigate through. It is bad in that I don't like Word or Visio and have been forced to get a basic understanding of these tools in order to effectively communicate the ideas.
My current employer is toying with the idea of Word documents in a Sharepoint portal. We'll just have to see how far we get there