As a professional system administrator, what configuration items do you consider it essential to track to perform proper configuration or change management?
For example, in Windows, do you track registry changes in addition to hardware or software? In Linux, do you diff config files?
EDIT: For clarification, I'm looking for what specifically to track (or not to track) -- if you want to recommend tools in addition to the items you feel are worth tracking, go ahead.
[Primarily Unix/Linux bias, I don't work on Windows systems :-)]
Anything that, once modified, alters the state of the running system needs to be tracked. Configuration must be stored in a repository that supports full change history, for example Git or Subversion, and it must be deployed in an automated manner such as with Opscode's Chef or Reductive Labs' Puppet.
A few examples of items that alter the state of the system:
Such tracking needs to be done with automated tools that provide sufficient logging for auditing changes. A software repository history will show, uh, the history of changes in the configuration files stored in said repository. In a declarative configuration tool like Chef or Puppet, each config file represents a set of resources such as users, packages or services, and the repository history will indicate changes in these.
From an application perspective, it is critical to document
Note - in response to romandas' comment in the question, I know that's not exactly one item, but I consider the overall document one item in this case
For windows you shouldn't need to "track" patch levels of the OS as that can be queried at any time via WMI. If you are following change management you can do a global query before and after a change to verify the results of an OS patch. (make sure the MSI provider is installed)
Driver updates should be tracked (wmi can be used to query them but it's not worth the effort to try to catch them all) Software application updates should be kept in the CMDB but (except for initially) should come from change management If you really wanted to track registry changes set a group policy to enable file and object access and choose which keys to audit (http://support.microsoft.com/kb/324739). This isn't for a CMDB but will allow you to track who is making changes.
So that's what folks typically want to track in a CMDB and why it's sometimes not a good idea so what should go into a CMDB. The answer is it depends. An ITIL defined CMDB may or may not care about machine configs unless that particular machine config pertains to a service. ACMDB might also contain an asset tracking solution (for things like hardware config location and warranty info) but is more about relationships of a particular Configuration Item to other CI. Relationships include things like "is responsible for", "is connected to", "depends on" and (more difficult but in many cases more important) "required for SLA tier_". In short, record what's required to recreate the service - not the server.
For example. If email was the service I'd list things like:
Hardware: 64 bit CPU, 3gb of ram (based on # of users on @today) 7gb of space for server install, 100 gb of storage for exchange dbs and recovery (based on usage as of @today), dvd rom drive.
Software: Server 2003 R2, Exchange 2007 SP2, MPIO drivers for the SAN
etc...
CMDBs are not typically for server admin usage. Admins will be far happier with an asset management solution.
OS patch level. Which could mean a lot of things. Ideally, you track which individual patches are on which machines, in a matrix. The poor man's version of this is a "last patch date" for each server (which assumes you just pull in all updates each time, wholesale).
For UNIX/Linux, I divide config files into two groups: Those that are provided by the OS and those that are part of our custom application(s).
All application config files are kept in our source repo. I have different standard copies for each installation (dev, QA, production, pre-production, demo...). When we determine that we need to make a change in any of these, the change happens first in the source repo copy and is then deployed out to the necessary servers. We also open a corresponding change request in the issue tracker.
For OS-type config files, we may or may not have them in the source repo. If they are, then we use the process above. If not, we'll put them in the repo if the number of changes reaches a certain arbitrary threshold. Until then, we at least open a ticket in the issue tracker to track that change.
Also, a great feature of our issue tracker is the ability to index source repo commits. It ties your tickets to your config changes quite nicely.
For network devices, such as Cisco switches, I like to collect and track their startup and running configs in a version control system. If possible, I store the username that did the edit as well.
I pull this via a Perl script usually, unless I have the option to automatically monitor the logs and trigger the config collection from log data.
For network devices RANCID (the automatic config grabber) is ideal.
For servers a combination of something like etckeeper (keeping /etc/ in a VCS) and puppet (with its config in a VCS).
Windows:
