Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What are some common mistakes made by System Administrators, and how can we avoid them?

Please justify your answer and give examples where applicable.

share
12  
Should be community wiki –  Adam Gibbins May 10 '09 at 20:59
1  
This should definitely be a community wiki –  Matt Simmons Jun 1 '09 at 20:17
show 1 more comment

locked by voretaq7 Aug 4 '12 at 22:30

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by voretaq7 Aug 4 '12 at 22:29

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

48 Answers

Assumptions are some of the commons mistakes I see. Assumptions are evil. One of my sysadmin mantras is "Assume nothing, diagnose everything."

This can be caused by several things, but mainly by badly configured alerts (misleading severity, bad tresholds, etc). So, that's a thing to definitely keep an eye on.

share
add comment

I've been on both sides of these in the past:

Not letting people know when you're making changes that affect them.
I think this one's pretty obvious.

Not realising that most people aren't aware of the same issues you are, and penalising them for it.
As has been noted around this site many times, people generally want to do well at their own job, not both theirs and yours. You might have to spell out what you're thinking occasionally.

Not knowing when to call in help.
Don't try to be Superman! A few hundred (or even thousand) of your local currency spent on brains can really save time and money for the business. And you get to learn from it.

Not researching a problem before disturbing someone.
We tend to take half-decent researching skills for granted, so it irritates when others don't take the time to find stuff out for themselves.

share
1  
Good points, especially letting people know about changes before making them. –  dss_so May 11 '09 at 7:48
show 2 more comments

I have had a good 4years+ experience as a sys admin and have tried several methods to show people the method to what they perceive as 'madness'.

I've been feeding users a short brief background as to why some things can and can't be done and they've found it very useful. Users started perceiving ITS in general as a more productive entity that really wants to help users!

Never seen this before! Sure there's always the odd troublesome users which fit the cliche' "10% giving 90% of trouble".

I guess what I'm trying to say is, one of my biggest mistakes was underestimating the effect of a short brief summary as to why we do what we do. It sure has granted a much healthier relationship with IT in general.

share
add comment
  1. When scripting always echo the commands that you intend to process from iterated variables. Review the commands and ensure that they look exactly as you anticipated.

  2. Don't leave commands sitting on the shell prompt. It only takes another task or person to distract you and you've forgotten that it was "primed". Can be dangerously compounded by idle screen blanking or laggy connections.

share
add comment

Typing too fast. I've made more errors that way.

I now have a three-step process when logged in as root or using sudo:

  1. Type a command, without hitting enter.
  2. Sit on my hands and look at it without distraction. (Talking to anybody counts as distraction.)
  3. Hit enter.
share
add comment
  • Non-existent change-management procedures. Solution; any steps towards change-management are good steps.

  • Inability to trust/listen to others. Solution; hire the right tool for the right job. Pass over the tech who's got borderline Aspergers. Focus on someone who will be a good fit, but who learns quickly and shows aptitude.

  • Not troubleshooting the underlying problem, instead focusing on the symptom. Solution; don't get caught up in the details. Look at it in a meta-sense. 10K-foot view.

  • Not being honest/ethical. Solution; your integrity is the only thing of value that you have. Leadership will recognize someone like that and reward them.

  • No Documentation. Solution; write.. it.. down. Email it if you must, to yourself if you want to. Find a group of people, email it to them. Google docs, whatever.

share
add comment

Most common pitfalls that I've ever seen and fallen into myself, ranked on order of importance/criticality.

1.) Assumption. Example: "I assumed that the problem had to be with the network card and had been troubleshooting the device for an hour before it had occurred to me to check the cable." This is one of the number one killers I've ever seen. Never assume anything and remember Mr. Holmes's lesson, `When you take away everything that cannot be the problem, what you're left with, no matter how impossible, must be the answer.'

2.) Arrogance. Example: "I'm the freakin' senior admin, what does the junior think he/she can positively contribute to the troubleshooting?". During an ITR, I've had a web developer point out a very small, yet critical problem in a router configuration that would have saved me hours of troubleshooting. Another set of eyes on a problem can't hurt and many times even is beneficial for training.

3.) Lack of RTFM. Example: "I've been working with Brocade fiberchannel switches for years. I know how to zone a fabric, ok?". The tech in question ended up creating a a zone for a tape library that consisted of a massive amount of devices all trying to talk to the tape library at once, instead of a one-to-one zoning plan. Without a quick consult with `El Manuel', the tech didn't know he was far outside best-practices. The one-to-one example was in the first three pages.

4.) Poor change-management/lack of communication and documentation. Example: The umteenth email sent out to a group asking, "Did anyone mess around with the webserver over the weekend, because it's down, we're out of clues and corporate wants it back up ASAP." This is another huge killer. No matter how good of an admin you are, if you didn't document or communicate what you did to fix that 20-hour router outage, and another one goes down three hours after you've finally gone home to get some sleep, you're only a.) looking like a fool and b.) doing yourself harm.

5.) Bad management/dysfunctional team. Example: Fear of looking stupid or assassination from co-workers causes things to be `swept under the rug', etc. etc. A good team is a reflection of it's leader and vice versa. A team's leader is responsible for a.) Ensuring the entire team gets credit when someone does a stellar job, and rewarding the stellar worker. b.) Shielding the team (AND responsible party) from the wrath of others when someone screws up, taking full responsibility for the problem, privately counseling the responsible one and taking positive steps to ensure it never happens again. A good manager will also remove all obstacles in the path of his or her team.

Finally, A good leader/manager, especially in tech, is NEVER the smartest guy on the team. Good leaders surround themselves with smarter advisers. A leader's job is to enable the team, not become bogged down and responsible for every little detail, in effect carrying a team who can't get the job done. It becomes a self-defeating fallacy.

HTH.

share
add comment

Forgetting that there are hundreds or thousands of people who will be affected by the consequences of your actions.

Failing to get the basics adequately covered in your blind rush to the exciting stuff.

share
add comment

Here are my 9 rules to Failure.

1 - DON'T Think for yourself... lack of confidence. 2 - DON'T Keep it simple... the best way, the hard way. 3 - DON'T Control inner your chaos... AVOID to become a stress Zen. 4 - IGNORE your enviroment... you're the only one! 5 - DON'T Keep Backups... why bother disk space? 6 - DON'T Test Backups... AVOID backups, skip this waste of time. 7 - AVOID exploring new paths of knowledge... better walk in familiar paths. 8 - DON'T Gain extra motivation... winners suck... ordinary fits best! 9 - AVOID Social Media, Trobleshooting, reading manuals... only trust in your experience.

share
add comment

Assuming the user will have enough knowledge of how the software should work and avoid dealing with error handling that comes back to bite you in the butt.

share
add comment

Two things:

Make sure you have a solid backup plan for your data.

Also, it is critical to have a solid support team around you when you get stuck. Being the loan ranger doesn't work.

share
add comment

People being sys admins? One of my biggest issue is over-training or excessive research. I seem to spend more time reading journals and feeds than I do actually working. It's a tough habit to break.

share
show 2 more comments

From Deep Thoughts by SysAdmins...some of my favorites

THe Mack Truck Scenario: If no one will be able to figure this out if you get hit by a Mack Truck then you are doing something wrong.

If you havent thought of at least one potential negative outcome of hitting 'enter' after the command you just typed then you dont understand the command well enough to use it on a production system.

If you do it more than once, automate it. If you cant automate it then document it. Document it anyway.

AND MY FAVORITE - if it seems like someone else may have encountered this problem before, they probably have. Google for the answer.

share
add comment

My two golden rules

  1. Don't install from source on a binary distribution. It breaks your upgrade path and security patches.

  2. Run the same versions of all packages on development and production.

share
2  
I would rephrase 1. as: don't install anything by bypassing your package management system. You can install software from sources, but use system tools to create native packages (dpkg-buildpackage on Debian, for example). It will make tracking that software easier in the future. –  drybjed Jun 2 '09 at 0:38
add comment

Plan for the worst, hope for the best - anything else is wrong in my book.

share
add comment

Motivating Staff

Get a contractor in to do their job, which you haven’t given them enough time to do, and asked them to do during a time when they’re swamped with other work. And then complain that you’re running over budget.

RAID

When trying to rebuild a mirrored RAID array, select the new disk as the place to mirror from….

Working with Crucial Services

  • Make sure that the only environment a crucial service can run on (the one that takes payments) is an old PC that used to belong to an Ex Developer.
  • Make sure that this box has faulty RAM.
  • Regularly play with the innards of this box.
  • When people ask what happens if the box goes down, tell them it’s a low priority.
  • When the box actually does go down, tell people that it’s ok, there are no issues, and that you’ll have it fixed in 5 minutes…

Staff Motivation #2

Break your development team’s dev environments, and delete half of the work they’ve done that week. A week later, ask why they didn’t do unpaid overtime to keep up to speed

Be the person to break things, then go on holiday

This morning, the first day our DBA is away on holiday for a week, we find that he’s changed the servers to master-master replication. We’re learning fast about howto fix replication issues, but even we know that ignoring replication issues by default is a bad idea

All things that have happened to myself in the last... 4 months. Courtesy of http://www.stopyouredoingitwrong.com (my website, and the reason for this question!)

share
add comment

Trusting your end users to tell the truth. Sometimes they may think that if they tell you what really happened that they would suffer some sort of repercussions, or perhaps they don't know which information is relevant. But, at the end of the day, it's best to be skeptical and to ask as many questions as possible.

share
add comment

overconfidence

share
add comment

Many of the "fiasco" scenarios that I've been called in to resolve come down to admins not applying consistent and scientific troubleshooting technique.

When you're troubleshooting a problem in a "black box" (read: closed source software/hardware, 3rd party system, etc), you should change one thing at a time (and document your changes) and exercise a consistent test case with each change. If your hypothesis doesn't bear out, return things back to their original state and start again.

Lather, rinse, repeat.

What I see, more often than not, are frazzled admins running around making random changes without documenting what has changed, and without testing whether or not their change made a difference. Before long, the initial conditions are lost. When the issue is finally resolved, no root cause analysis can ever be done because no one is sure what fixed the problem.

We make a bad name for our trade when we act that way.

share
add comment

This question is pretty close to the post: "Common mistakes made by System Administrators and how we can avoid them"

I'd like to add that one thing to do wrong is to NOT document your work.

share
add comment

Document first, then execute

share
add comment
  • Underestimating Murphy
  • Not using checklists
  • Taking shortcuts
  • Assuming that almost complete is "good enough"
  • "Who would want to steal /our/ data?"
  • Implementation without proper testing
share
add comment

Making assumptions on 'what seems sensible'.

E.g. A computer isn't turning on, so you think the power supply is shot because that's the sensible thing. Instead the user is calling on their mobile in the middle of a power cut.

share
add comment

Rushing to conclusions...

Stop and think. Make sure brain is in gear before engaging mouth. Quit leading me on wild goose chases when troubleshooting problems and wasting my time reporting things that are really by-design behavior.

share
add comment

just a few

  • not listening to others [ this starts with requirements gathering.. ]
  • not reading the f... manuals
  • assuming everything will go well [ who needs backups anyway ? ]
  • assuming that if things work now they will in the future [ who needs monitoring ? ]
  • relying blindly on 3rd parties... we have 99.999 internet access sla. [ things will work fine. or not.. in worst case you'll pay 10% less of your monthly bill in exchange for 10h downtime, convince your 'the business' to be satisfied with this ]
  • this leads us to not explaining clearly to non-techies what are consequences, costs, risks and opportunities. and double checking they actually understood.
share
add comment

Running systems on luck rather than on work.

I've seen many, many instances of the luck-based approach to systems administration:

  • If you don't do the work to document anything or share information, you're hoping to get lucky and have something break while you're at work, rather than when you're sick, on vacation, asleep, out of cell phone range or after you've left the company.
  • If you don't take backups off site, you're counting on luck to save you from a fire, flood, burglary, damaged building, etc.
  • If you don't put redundancy into your backup system, you're counting on the good luck of a backup tape never going bad when the director of your division overwrites a file that's due RIGHT THEN.
  • If you don't patch your PCs or their applications, you're hoping luck holds out and a worm never gets into the network and owns all your machines at the same time.
  • If you don't communicate with your users and assume they know not to save information on their local drives, you're hoping to get lucky and have an administrative assistant's PC die, rather than the CEO, CIO, CTO, COO or some Vice President's computer.
  • If you don't backup all of your servers because some are "not important" or they're "the developer's problem", you're gambling that you can rebuild those servers without the data or any configuration information.

Do the work and you take luck out of the equation. For the rest of you, good luck!

share
show 1 more comment

A common mistake (that I have made in the past) is becoming dependent on particular tools, shortcuts, or the like. When they aren't there, and you're accustomed to them, it can really throw you for a loop. Be sure that you understand how all your tools work, and that you can replicate their functionality with nothing more than the command line-- you never know when you're going to have to do something critical on a machine you had completely forgotten about, or weren't in control of until recently.

share
add comment

Fatigue. Usually prompted by heroics. Answer: avoid heroics

Fileglobbing. Test globs e.g. before you rm *20090527.log check that it gets what you think it gets with ls *20090527.log

I very much agree with owenevans. Automate where possible.

Test on dev systems.

Always ask your self the question "What can go wrong? And how can I prevent/fix it?".

share
add comment

A very big mistake I see is not immediately owning up to the mistake. It is understandable to cringe and try to hide (or even try to resolve yourself) but very likely this makes the problem worse.

share
add comment

Almost all the common mistakes - as opposed to strategic errors like not having proper backups - are caused by the operator (rm -rf / being a case in point). The way around this is to have your procedures automated. Use GPOs, puppet, setup scripts, whatever. That way your work will be higher quality and you will work faster, giving you more time to post on serverfault...

share
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.