Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What are some common mistakes made by System Administrators, and how can we avoid them?

Please justify your answer and give examples where applicable.

share

locked by voretaq7 Aug 4 '12 at 22:30

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by voretaq7 Aug 4 '12 at 22:29

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

12  
Should be community wiki –  Adam Gibbins May 10 '09 at 20:59
1  
This should definitely be a community wiki –  Matt Simmons Jun 1 '09 at 20:17

48 Answers 48

"When you hear hoofbeats, think horses, not zebras"

This is the best rule of troubleshooting issues I've ever heard. I try to live by it. It means to look at what the system is actually telling you first.

Too many folks dismiss the error messages or symptoms they see and attempt to fix what they think is the problem. Don't assume.

If the error message indicates a permissions problem, investigate permissions first. "Access Denied" is not a network issue.

share

Almost all the common mistakes - as opposed to strategic errors like not having proper backups - are caused by the operator (rm -rf / being a case in point). The way around this is to have your procedures automated. Use GPOs, puppet, setup scripts, whatever. That way your work will be higher quality and you will work faster, giving you more time to post on serverfault...

share

Fatigue. Usually prompted by heroics. Answer: avoid heroics

Fileglobbing. Test globs e.g. before you rm *20090527.log check that it gets what you think it gets with ls *20090527.log

I very much agree with owenevans. Automate where possible.

Test on dev systems.

Always ask your self the question "What can go wrong? And how can I prevent/fix it?".

share
  • Underestimating Murphy
  • Not using checklists
  • Taking shortcuts
  • Assuming that almost complete is "good enough"
  • "Who would want to steal /our/ data?"
  • Implementation without proper testing
share

overconfidence

share

Motivating Staff

Get a contractor in to do their job, which you haven’t given them enough time to do, and asked them to do during a time when they’re swamped with other work. And then complain that you’re running over budget.

RAID

When trying to rebuild a mirrored RAID array, select the new disk as the place to mirror from….

Working with Crucial Services

  • Make sure that the only environment a crucial service can run on (the one that takes payments) is an old PC that used to belong to an Ex Developer.
  • Make sure that this box has faulty RAM.
  • Regularly play with the innards of this box.
  • When people ask what happens if the box goes down, tell them it’s a low priority.
  • When the box actually does go down, tell people that it’s ok, there are no issues, and that you’ll have it fixed in 5 minutes…

Staff Motivation #2

Break your development team’s dev environments, and delete half of the work they’ve done that week. A week later, ask why they didn’t do unpaid overtime to keep up to speed

Be the person to break things, then go on holiday

This morning, the first day our DBA is away on holiday for a week, we find that he’s changed the servers to master-master replication. We’re learning fast about howto fix replication issues, but even we know that ignoring replication issues by default is a bad idea

All things that have happened to myself in the last... 4 months. Courtesy of http://www.stopyouredoingitwrong.com (my website, and the reason for this question!)

share

From Deep Thoughts by SysAdmins...some of my favorites

THe Mack Truck Scenario: If no one will be able to figure this out if you get hit by a Mack Truck then you are doing something wrong.

If you havent thought of at least one potential negative outcome of hitting 'enter' after the command you just typed then you dont understand the command well enough to use it on a production system.

If you do it more than once, automate it. If you cant automate it then document it. Document it anyway.

AND MY FAVORITE - if it seems like someone else may have encountered this problem before, they probably have. Google for the answer.

share

Two things:

Make sure you have a solid backup plan for your data.

Also, it is critical to have a solid support team around you when you get stuck. Being the loan ranger doesn't work.

share

Most common pitfalls that I've ever seen and fallen into myself, ranked on order of importance/criticality.

1.) Assumption. Example: "I assumed that the problem had to be with the network card and had been troubleshooting the device for an hour before it had occurred to me to check the cable." This is one of the number one killers I've ever seen. Never assume anything and remember Mr. Holmes's lesson, `When you take away everything that cannot be the problem, what you're left with, no matter how impossible, must be the answer.'

2.) Arrogance. Example: "I'm the freakin' senior admin, what does the junior think he/she can positively contribute to the troubleshooting?". During an ITR, I've had a web developer point out a very small, yet critical problem in a router configuration that would have saved me hours of troubleshooting. Another set of eyes on a problem can't hurt and many times even is beneficial for training.

3.) Lack of RTFM. Example: "I've been working with Brocade fiberchannel switches for years. I know how to zone a fabric, ok?". The tech in question ended up creating a a zone for a tape library that consisted of a massive amount of devices all trying to talk to the tape library at once, instead of a one-to-one zoning plan. Without a quick consult with `El Manuel', the tech didn't know he was far outside best-practices. The one-to-one example was in the first three pages.

4.) Poor change-management/lack of communication and documentation. Example: The umteenth email sent out to a group asking, "Did anyone mess around with the webserver over the weekend, because it's down, we're out of clues and corporate wants it back up ASAP." This is another huge killer. No matter how good of an admin you are, if you didn't document or communicate what you did to fix that 20-hour router outage, and another one goes down three hours after you've finally gone home to get some sleep, you're only a.) looking like a fool and b.) doing yourself harm.

5.) Bad management/dysfunctional team. Example: Fear of looking stupid or assassination from co-workers causes things to be `swept under the rug', etc. etc. A good team is a reflection of it's leader and vice versa. A team's leader is responsible for a.) Ensuring the entire team gets credit when someone does a stellar job, and rewarding the stellar worker. b.) Shielding the team (AND responsible party) from the wrath of others when someone screws up, taking full responsibility for the problem, privately counseling the responsible one and taking positive steps to ensure it never happens again. A good manager will also remove all obstacles in the path of his or her team.

Finally, A good leader/manager, especially in tech, is NEVER the smartest guy on the team. Good leaders surround themselves with smarter advisers. A leader's job is to enable the team, not become bogged down and responsible for every little detail, in effect carrying a team who can't get the job done. It becomes a self-defeating fallacy.

HTH.

share

Typing too fast. I've made more errors that way.

I now have a three-step process when logged in as root or using sudo:

  1. Type a command, without hitting enter.
  2. Sit on my hands and look at it without distraction. (Talking to anybody counts as distraction.)
  3. Hit enter.
share

Many of the "fiasco" scenarios that I've been called in to resolve come down to admins not applying consistent and scientific troubleshooting technique.

When you're troubleshooting a problem in a "black box" (read: closed source software/hardware, 3rd party system, etc), you should change one thing at a time (and document your changes) and exercise a consistent test case with each change. If your hypothesis doesn't bear out, return things back to their original state and start again.

Lather, rinse, repeat.

What I see, more often than not, are frazzled admins running around making random changes without documenting what has changed, and without testing whether or not their change made a difference. Before long, the initial conditions are lost. When the issue is finally resolved, no root cause analysis can ever be done because no one is sure what fixed the problem.

We make a bad name for our trade when we act that way.

share

Plan for the worst, hope for the best - anything else is wrong in my book.

share

People being sys admins? One of my biggest issue is over-training or excessive research. I seem to spend more time reading journals and feeds than I do actually working. It's a tough habit to break.

share

Assuming the user will have enough knowledge of how the software should work and avoid dealing with error handling that comes back to bite you in the butt.

share

Here are my 9 rules to Failure.

1 - DON'T Think for yourself... lack of confidence. 2 - DON'T Keep it simple... the best way, the hard way. 3 - DON'T Control inner your chaos... AVOID to become a stress Zen. 4 - IGNORE your enviroment... you're the only one! 5 - DON'T Keep Backups... why bother disk space? 6 - DON'T Test Backups... AVOID backups, skip this waste of time. 7 - AVOID exploring new paths of knowledge... better walk in familiar paths. 8 - DON'T Gain extra motivation... winners suck... ordinary fits best! 9 - AVOID Social Media, Trobleshooting, reading manuals... only trust in your experience.

share

Forgetting that there are hundreds or thousands of people who will be affected by the consequences of your actions.

Failing to get the basics adequately covered in your blind rush to the exciting stuff.

share

Assumptions are some of the commons mistakes I see. Assumptions are evil. One of my sysadmin mantras is "Assume nothing, diagnose everything."

This can be caused by several things, but mainly by badly configured alerts (misleading severity, bad tresholds, etc). So, that's a thing to definitely keep an eye on.

share

Document first, then execute

share

Not the answer you're looking for? Browse other questions tagged or ask your own question.