Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

What software or system do you guys out on server fault use to remind you to do routine maintenance? How do you checklist and log the various items you are supposed to check? Do you have an internal process document? Do you have cron mail you every week with reminders to check system logs?

Also, do you work on a team to do system maintenance, and if so, how do you coordinate who will do what maintenance?

If you use a bug/issue tracking system to enter tasks, do you have a cron job enter recurring tasks?

share|improve this question

5 Answers 5

I'm currently using Request Tracker (http://www.bestpractical.com/rt)
All maintenance events get an associated ticket in the "systems" queue. Notes on problems encountered, who did what work when, etc. are all entered into the ticket, along with necessary approvals.

At the moment our recurring tasks (quarterly patching, etc.) are manually created, but they could be automated easily enough (cron job + email).

Coordinating who is doing what work is relatively easy for us as there's only 2 people in our admin group, but as we scale up the plan is to create a master ticket for maintenance events & use child tickets assigned to the responsible parties to delegate the work.


Daily stuff (log checks, etc.) is another matter: I have all of that farmed out to automated processes:

  • InterMapper keeps an eye on the servers' overall status (SNMP queries looking for high load, low disk space, etc.), functionality of our web interfaces, and sundry other things that could indicate trouble.
  • Syslog-NG collects logs from our hosts & feeds them through a bunch of scripts that check for obvious badness. I cast my eye over the logs occasionally to sanity-check the scripts, but it's not regularly scheduled.
share|improve this answer

Outlook and OneNote

share|improve this answer

For project-work, it's driven out of the Project Management app (email & calendar integrated with the ability to document detailed work and schedule it for particular people).

For maintenance, upgrades, fixes, etc. we have a ticketing system that more or less integrates with our Change Management process to handle requests and scheduling.

For completely internally-driven work and work on long cycles (quarterly, yearly, etc.) :

Reminders to do things are calendared. Informal/Semi-formal documentation exists ("wiki") for what the general schedule might be.

Some amount of "how to" and procedural documentation exists on how to carry out tasks and is accessible to the team at large but people have their own admin "black books" and logs with notes & recipes.

share|improve this answer

A monitoring system can help with these things:

  • We document each round of monthly maintenance in a word doc file with checkboxes. Each month we save the report into a folder on our NAS. We monitor the folder's minimum file age. If the minimum file age is above 40 days, we get an alarm.

  • One part of our routine maintenance is to reboot selected servers and appliances once a month. We use "system uptime" sensors (SNMP/WMI) on our monitoring software and if the uptime is above 40 days we get an alarm.

  • For backups we monitor the minimum file age in each server's backup folder on our NAS. If the minimum file age is above 10 days, we get an alarm.

share|improve this answer

Properly implemented automation does away with the need for task and check lists altogether. Why are you manually wanting to check things when you have computers which can do the job far more effectively and efficiently?

Anything that needs periodic checking is checked by the monitoring system. Routine tasks are automated whenever practical and reminders sent for those few tasks that need to be done manually. Documentation is another matter but done right your computers can mostly create their own documentation.

Stop looking for better manual ways and start looking for better automated ways to do any job. The computers are there to work for us, not us to work for them.

share|improve this answer
    
Good rule of thumb: A sysadmin should always be both competent and lazy. The desire not to do work will lead good sysadmins to implement good automation. –  voretaq7 Mar 24 '10 at 17:55
    
Let me give a specific example: I need to monitor for security patches for Apache, then generate a new build and test it when a patch does come out. The routine part is monitoring for a new Apache release. Can't just update directly from the (main) repository because it won't have the correct modules compiled in. Also, need to audit to make sure that releases have been checked for. Does that make more sense? –  Zak Mar 24 '10 at 22:35
    
Also, I don't want to just roll the latest batch of whatever software until the build has passed QA. Much of QA is automated, but not all of it. –  Zak Mar 24 '10 at 22:38
    
And is there a reason that can't all be scripted? Automated checks for updates, sending you an alert when some are available, followed by a scripted compile and install, ready for you to test. Let the machine do the bulk of the work and tell you when your attention is required. –  John Gardeniers Mar 25 '10 at 0:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.