As a professional system administrator, what configuration items do you consider it essential to track to perform proper configuration or change management?

For example, in Windows, do you track registry changes in addition to hardware or software? In Linux, do you diff config files?

EDIT: For clarification, I'm looking for what specifically to track (or not to track) -- if you want to recommend tools in addition to the items you feel are worth tracking, go ahead.

  • In true SF and SO style, can we try and have one item per answer and vote up the best ones? – romandas May 26 '09 at 23:02
  • Great idea, I've split mine up. – Adam D'Amico May 27 '09 at 0:01
  • Just a note - registry changes are impossible to track. Go get a copy of Process Monitor ( technet.microsoft.com/en-us/sysinternals/bb896645.aspx ) and use it to track registry writes for just a few minutes, and you'll understand what I mean here. – quux May 27 '09 at 0:25
  • @quux - just for sake of discussion (since I haven't figured out how to track registry changes well), any chance of tracking just the changes initiated by application installs/config changes.. as opposed to the torrent of OS writes? Not thinking every change, but certain ones perhaps? – romandas May 27 '09 at 2:10

[Primarily Unix/Linux bias, I don't work on Windows systems :-)]

Anything that, once modified, alters the state of the running system needs to be tracked. Configuration must be stored in a repository that supports full change history, for example Git or Subversion, and it must be deployed in an automated manner such as with Opscode's Chef or Reductive Labs' Puppet.

A few examples of items that alter the state of the system:

  • Installed package / patch versions.
  • Deployed applications, application libraries.
  • Administrative and application users.
  • Hardware configuration and device drivers/modules.
  • Scheduled maintenance jobs (cron on Unix/Linux).
  • Services and components that are monitored by external tools.

Such tracking needs to be done with automated tools that provide sufficient logging for auditing changes. A software repository history will show, uh, the history of changes in the configuration files stored in said repository. In a declarative configuration tool like Chef or Puppet, each config file represents a set of resources such as users, packages or services, and the repository history will indicate changes in these.

| improve this answer | |
  • 1
    +1 - I agree with 'Anything that, once modified, alters the state of the running system'. On a side note (not specifically directed at you, jtimberman), how does one go about tracking all the state-affecting Windows components? e.g. local policies, NTFS permissions, etc – romandas Jun 3 '09 at 22:43
  • No idea on Windows myself, I use it for media center and games :-) – jtimberman Jun 4 '09 at 2:52
  • Really? Someone uses Media Center? – romandas Jun 4 '09 at 21:59
  • Definitely off topic, but yeah. I use MC and really like it. Its the killer app for Vista. – jtimberman Jun 5 '09 at 0:17

From an application perspective, it is critical to document

  • OS and patch level
  • Which layered products are installed and which patches have been applied
  • Which custom and / or third party products and libraries are installed including version
  • Where to lay your hands on install source for said custom / third party items
  • Product keys / SN's for all of the above
  • Application level server configuration (ie- web, mail, etc)
  • A contact list if there is a separate development team responsible at the application level
  • Backup configuration

Note - in response to romandas' comment in the question, I know that's not exactly one item, but I consider the overall document one item in this case

| improve this answer | |

For windows you shouldn't need to "track" patch levels of the OS as that can be queried at any time via WMI. If you are following change management you can do a global query before and after a change to verify the results of an OS patch. (make sure the MSI provider is installed)

Driver updates should be tracked (wmi can be used to query them but it's not worth the effort to try to catch them all) Software application updates should be kept in the CMDB but (except for initially) should come from change management If you really wanted to track registry changes set a group policy to enable file and object access and choose which keys to audit (http://support.microsoft.com/kb/324739). This isn't for a CMDB but will allow you to track who is making changes.

So that's what folks typically want to track in a CMDB and why it's sometimes not a good idea so what should go into a CMDB. The answer is it depends. An ITIL defined CMDB may or may not care about machine configs unless that particular machine config pertains to a service. ACMDB might also contain an asset tracking solution (for things like hardware config location and warranty info) but is more about relationships of a particular Configuration Item to other CI. Relationships include things like "is responsible for", "is connected to", "depends on" and (more difficult but in many cases more important) "required for SLA tier_". In short, record what's required to recreate the service - not the server.

For example. If email was the service I'd list things like:

Hardware: 64 bit CPU, 3gb of ram (based on # of users on @today) 7gb of space for server install, 100 gb of storage for exchange dbs and recovery (based on usage as of @today), dvd rom drive.

Software: Server 2003 R2, Exchange 2007 SP2, MPIO drivers for the SAN


CMDBs are not typically for server admin usage. Admins will be far happier with an asset management solution.

| improve this answer | |
  • So, would it be correct or incorrect to say that a CMDB is meant to provide a current snapshot of how the server and network environment looks? If correct, how is that not useful for a server or network admin? – romandas May 27 '09 at 13:35
  • 1
    it would be incorrect. CMDB comes from ITIL. THe CMDB tracks CI relationships and their Configs. The Config piece is not necessarily server specs but usually software specs. Asset management is part of the cmdb. From an ITIL perspective hardware specs are the least important since it has little to do with the service delivery It would be NICE if it, did but not needed. It depends on what you decide a baseline item is and if the windows already tracks it do you need to duplicate it? Far more valuable would be software versions and can you easily compare them to change management. – Jim B May 27 '09 at 20:55
  • I get the impression that my idea of configuration management and its purpose doesn't track along ITIL lines.. :) – romandas May 28 '09 at 2:44
  • Right :) It sound like what you're really asking for is change management. Along with the CMDB you get a granular snapshot, as the CMDB would be populated from the completed change management entries. Even if you captured the baseline configs just before a change you would either have or be able to get any baseline config. – Jim B May 28 '09 at 21:04
  • Hmm, should I change the initial question to 'proper change management' then? – romandas Jun 3 '09 at 22:44

OS patch level. Which could mean a lot of things. Ideally, you track which individual patches are on which machines, in a matrix. The poor man's version of this is a "last patch date" for each server (which assumes you just pull in all updates each time, wholesale).

| improve this answer | |

For UNIX/Linux, I divide config files into two groups: Those that are provided by the OS and those that are part of our custom application(s).

All application config files are kept in our source repo. I have different standard copies for each installation (dev, QA, production, pre-production, demo...). When we determine that we need to make a change in any of these, the change happens first in the source repo copy and is then deployed out to the necessary servers. We also open a corresponding change request in the issue tracker.

For OS-type config files, we may or may not have them in the source repo. If they are, then we use the process above. If not, we'll put them in the repo if the number of changes reaches a certain arbitrary threshold. Until then, we at least open a ticket in the issue tracker to track that change.

Also, a great feature of our issue tracker is the ability to index source repo commits. It ties your tickets to your config changes quite nicely.

| improve this answer | |

For network devices, such as Cisco switches, I like to collect and track their startup and running configs in a version control system. If possible, I store the username that did the edit as well.

I pull this via a Perl script usually, unless I have the option to automatically monitor the logs and trigger the config collection from log data.

| improve this answer | |

For network devices RANCID (the automatic config grabber) is ideal.

For servers a combination of something like etckeeper (keeping /etc/ in a VCS) and puppet (with its config in a VCS).

| improve this answer | |
  • So are you saying the items to track are the configs from your network devices and the files in /etc, or are you making a recommendation on tools? I was looking for the items you would track; tool recommendations can be included as a nice addition, but aren't specifically what I was looking for. – romandas May 27 '09 at 13:08
  • Bit of both, those are my starting points, along with a general wiki for things that can't be easily covered should get you 90% there. – LapTop006 May 27 '09 at 13:53


  • OS Version
  • OS Service Pack
  • Security Patches
  • Audit policy (should be enforced via GPO)
  • Members of Administrators, Power Users, and Remote Desktop Users
  • Installed application list
  • Network Shares
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.