Common mistakes made by System Administrators and how can we avoid them? [closed]

Question

What are some common mistakes made by System Administrators, and how can we avoid them?

Please justify your answer and give examples where applicable.

Answer 1

Creating backups but not testing the restore procedure properly.

Justification: It's easy to lose oneself in backup strategies without considering restore strategies.

Examples: I'm sure none are required :)

Answer 2

I've been on both sides of these in the past:

Not letting people know when you're making changes that affect them.
I think this one's pretty obvious.

Not realising that most people aren't aware of the same issues you are, and penalising them for it.
As has been noted around this site many times, people generally want to do well at their own job, not both theirs and yours. You might have to spell out what you're thinking occasionally.

Not knowing when to call in help.
Don't try to be Superman! A few hundred (or even thousand) of your local currency spent on brains can really save time and money for the business. And you get to learn from it.

Not researching a problem before disturbing someone.
We tend to take half-decent researching skills for granted, so it irritates when others don't take the time to find stuff out for themselves.

Answer 3

Biggest mistake? Not understanding that the network you're administering is used by real people who are trying to get real work done. Real people who do not understand what you do or why, and don't need to. And you are not their boss. :)

Not understanding this leads to anger and resentment on both sides. It leads to sysadmins making assumptions about what they can and should do.

For example, forcing reboots of systems overnight to apply patches will anger the users who didn't save their work before they left home. Bad sysadmin thinks "this is for security, we must do it, people ignore the reboot reminders, they get what they deserve". Good sysadmin thinks "people have weird workflows that work for them, they don't understand security, I have to find ways to help them out and still keep things safe". What that help may be is dependent on the culture.

Real people!

Answer 4

Getting sucked into solving non-technical problems with technical means. For example, if people browsing the internet for personal use is a problem, the answer is not always to install a web filter, monitoring application, and check logs daily. If you have supervisors doing their job and supervising, then people hanging out on Facebook all day won't be nearly as much of a problem. Sometimes reminding people of the company policy is all it takes. And it costs a hell of a lot less to write a policy saying "no facebook, myspace, ebay, and other personal websites" than it does to do the research on a web filter, get a quote, buy it, learn about it, install it, configure and deploy it, and finally provide continued support for it.

Now, don't get me wrong, there are certainly lots of great uses for hardware/software web filters, and other sorts of great stuff, but sometimes they are impractical for the situation. For example, a small office of 10 people doesn't need a Websense filter. It needs to have a manager or other kind of supervisor who makes sure to assert their presence in the office so that people know they can't get away with, not only personal browsing, but personal phone calls, and other kinds of slacking that don't involve a computer or other piece of technology (sudoku during work hours, etc).

Also, don't ever underestimate the importance of making rounds. Go around the office and ask around with everyone if everything's running swell on their machine. If it isn't, either fix it right there, or schedule some time for it later in the day. It might seem like you're creating work for yourself at first, but overall, if your users are open about the problems they are having, and tell you about it when making rounds, you can stop big problems early and save yourself a lot of time.

Answer 5

The biggest "common" mistake is when sysadmins don't document anything.

Answer 6

Believing that you're only making a small change. Nobody will notice. It'll be fine.

Avoid by having proper change management procedures.

Answer 7

Trying to resolve a problem with complicated approaches instead of taking a few steps back and try to see if there's a simple solution.

In other words, starting at the end instead of the beginning. I don't know if that makes sense :)

Answer 8

Changing things and then not being able to repeat those changes.

For example, let's say you're trying to get a network service set up on Linux. You read the man page and it says you should edit a certain file. You edit the file, but the service doesn't work. You read the manual again and see some mention of a second file. Edit that file, still no luck. Repeat until the service finally works.

Why does the service now work? Is it the last change you made? The first and the last? All the changes together? Do you even remember what files you changed?

Now you have to set up the same service again on a different computer (or on the same computer again, after having to restore something from a backup). Can you get it working again?

This sort of problem occurs a lot when admins are not thoroughly familiar with a particular product. And given the number of products out there there is no way for everyone to be familiar with everything they set up.

This problem is worse on Windows if you're clicking stuff in a GUI becuase you have no idea what is changed and it's much harder to do version control.

Answer 9

My two golden rules

1. Don't install from source on a binary distribution. It breaks your upgrade path and security patches.

2. Run the same versions of all packages on development and production.

Answer 10

Running systems on luck rather than on work.

I've seen many, many instances of the luck-based approach to systems administration:

• If you don't do the work to document anything or share information, you're hoping to get lucky and have something break while you're at work, rather than when you're sick, on vacation, asleep, out of cell phone range or after you've left the company.
• If you don't take backups off site, you're counting on luck to save you from a fire, flood, burglary, damaged building, etc.
• If you don't put redundancy into your backup system, you're counting on the good luck of a backup tape never going bad when the director of your division overwrites a file that's due RIGHT THEN.
• If you don't patch your PCs or their applications, you're hoping luck holds out and a worm never gets into the network and owns all your machines at the same time.
• If you don't communicate with your users and assume they know not to save information on their local drives, you're hoping to get lucky and have an administrative assistant's PC die, rather than the CEO, CIO, CTO, COO or some Vice President's computer.
• If you don't backup all of your servers because some are "not important" or they're "the developer's problem", you're gambling that you can rebuild those servers without the data or any configuration information.

Do the work and you take luck out of the equation. For the rest of you, good luck!

Answer 11

Old phrase from carpentry: measure twice, cut once.

When you're pounding out a command on a CLI of some kind, double-check your entry. A single missing option, typo, or misplaced delimiter, can spell success or disaster. Re-read your command before pressing that Enter key.

Example

rm -rf ./*

vs.

rm -rf /*

The first one will remove files from a directory; the next one will remove all files from the / directory, which in turn, is everything.

Likewise, for those Windows admins out there:

robocopy \\sourceserver\sourceshare \\destserver\destshare /e /zb /move

vs.

robocopy \\sourceserver\sourceshare \\destserver\destshare /e /zb /mov

The first moves files and directories, then deletes afterwards; the second moves files and directories, but only removes the files.

Answer 12

See also What makes a “Good” or “Great” Administrator?. In my experience, avoiding common mistakes is not achieved by having a list of common mistakes in your head the whole time. I think it's one of the qualities that makes a good or great administrator; for example, a good administrator will devise appropriate policies, have them approved, and then follow them -- for example, a change management policy -- which will prevent the most common mistakes caused by changes.

It's really about thinking about the bigger picture, and not being caught up in the technical details of whatever you're working on at the moment.

I repeat my recommendation of The Practice of System and Network Administration by Tom Limoncelli, Christina Hogan and Strata Chalup; it's the 1000-page answer to your question, but it's absolutely worth it. Plus it's got lots of good stories of common mistakes.

Answer 13

I have had a good 4years+ experience as a sys admin and have tried several methods to show people the method to what they perceive as 'madness'.

I've been feeding users a short brief background as to why some things can and can't be done and they've found it very useful. Users started perceiving ITS in general as a more productive entity that really wants to help users!

Never seen this before! Sure there's always the odd troublesome users which fit the cliche' "10% giving 90% of trouble".

I guess what I'm trying to say is, one of my biggest mistakes was underestimating the effect of a short brief summary as to why we do what we do. It sure has granted a much healthier relationship with IT in general.

Answer 14

Believing in unrealistic deadlines. Deadlines, especially unrealistic ones, tend to make life worse, rarely better.

For instance, I knew two guys who were responsible for encrypting entire laptop hard drives. These were laptops belonging to sales reps in the field. Their boss made the statement "That'll be done next week." Deadline rolls around, and it's not done. CIO gets concerned and asks these two what the problem is. They state "If we get this remote encryption process wrong we'll turn those laptops into bricks, and require the sales reps to drive to home office to get the problem resolved. We're still testing everything to make sure we know what's going to happen." CIO turns to the manager and says "Why didn't you tell me this last week?"

You should never sacrifice quality for quantity, especially if it's just to meet unrealistic deadlines.

Answer 15

Not doing backups at all and relying on things like RAID or SAN mirroring to protect data.

Answer 16

Relying on rebooting a machine to fix a problem. Stop providing band-aid fixes to problems because it's quick and has worked in the past. Don't just fix it, fix it!

Answer 17

admitting that you're the sysadmin, and giving people your phone number ;-)

Answer 18

Shutting down a remote server when i meant to restart it. Easily done at 3am in the morning. To stop it i set the group policy to only let the administrator shut a server down and ensure everyone logs on using their own account. If you do then ever need to shut the server down you have to log in as the administrator which you should rarely have to do.

Answer 19

Not having support from upper management for the things you do. User policy? Useless without their approval. Backup solutions, disaster recovery and efficient virtualization projects? Useless without management on your side. Wasting time on things that won't matter, thinking "it's neater" or "it's cheaper".

It's one thing to properly mark the cables in a rack or redo a cabling job mess to be able to work with it in the future, but it's another to order a refit of all horizontal cabling because of policy, for an office that is to be abandoned in two months where a few floor-runs and mini-switches would do the job fine 'til then.

Oh and a completely different small pet peeve of mine: using the wrong monitoring solution - like a Hobbit or Nagios system for a mostly windows-based network, or the other way around, using a Microsoft monitoring suite for a mostly Linux-based network (not that I've seen that one yet, but the first one is way too common to even make sense). When the Exchange team themselves releases management packs with thousands of rules and written solutions to these problems, it seems wierd to try and redo their work one step at a time. You want to monitor entire services, like "the e-mail system" or "that line of business application" or "that office in Canada" - not individual processes and windows services. And you want to teach the system how to fix a problem once it has occurred, if it doesn't already know how, to prevent a occurrence somewhere else from making too much of a problem.

But then again, this is probably just me being a prick again ^^

Answer 20

Backing up to the same machine on which the data sits, is a problem I've encountered at two separate customers. Both DBAs were fired when the RAID died, and rightfully so.

Off-site backups are your best answer, and although I am echoing previous posts, it's important to ensure that you test your backups.

Answer 21

A very big mistake I see is not immediately owning up to the mistake. It is understandable to cringe and try to hide (or even try to resolve yourself) but very likely this makes the problem worse.

Answer 22

A common mistake (that I have made in the past) is becoming dependent on particular tools, shortcuts, or the like. When they aren't there, and you're accustomed to them, it can really throw you for a loop. Be sure that you understand how all your tools work, and that you can replicate their functionality with nothing more than the command line-- you never know when you're going to have to do something critical on a machine you had completely forgotten about, or weren't in control of until recently.

Answer 23

just a few

• not listening to others [ this starts with requirements gathering.. ]
• not reading the f... manuals
• assuming everything will go well [ who needs backups anyway ? ]
• assuming that if things work now they will in the future [ who needs monitoring ? ]
• relying blindly on 3rd parties... we have 99.999 internet access sla. [ things will work fine. or not.. in worst case you'll pay 10% less of your monthly bill in exchange for 10h downtime, convince your 'the business' to be satisfied with this ]
• this leads us to not explaining clearly to non-techies what are consequences, costs, risks and opportunities. and double checking they actually understood.

Answer 24

Making assumptions on 'what seems sensible'.

E.g. A computer isn't turning on, so you think the power supply is shot because that's the sensible thing. Instead the user is calling on their mobile in the middle of a power cut.

Answer 25

This question is pretty close to the post: "Common mistakes made by System Administrators and how we can avoid them"

I'd like to add that one thing to do wrong is to NOT document your work.

Answer 26

1. When scripting always echo the commands that you intend to process from iterated variables. Review the commands and ensure that they look exactly as you anticipated.

2. Don't leave commands sitting on the shell prompt. It only takes another task or person to distract you and you've forgotten that it was "primed". Can be dangerously compounded by idle screen blanking or laggy connections.

Answer 27

Rushing to conclusions...

Stop and think. Make sure brain is in gear before engaging mouth. Quit leading me on wild goose chases when troubleshooting problems and wasting my time reporting things that are really by-design behavior.

Answer 28

Trusting your end users to tell the truth. Sometimes they may think that if they tell you what really happened that they would suffer some sort of repercussions, or perhaps they don't know which information is relevant. But, at the end of the day, it's best to be skeptical and to ask as many questions as possible.

Answer 29

• Non-existent change-management procedures. Solution; any steps towards change-management are good steps.

• Inability to trust/listen to others. Solution; hire the right tool for the right job. Pass over the tech who's got borderline Aspergers. Focus on someone who will be a good fit, but who learns quickly and shows aptitude.

• Not troubleshooting the underlying problem, instead focusing on the symptom. Solution; don't get caught up in the details. Look at it in a meta-sense. 10K-foot view.

• Not being honest/ethical. Solution; your integrity is the only thing of value that you have. Leadership will recognize someone like that and reward them.

• No Documentation. Solution; write.. it.. down. Email it if you must, to yourself if you want to. Find a group of people, email it to them. Google docs, whatever.

Answer 30

That 'management' don't want to pay for anything, often they just need to see a good business-case or to understand the return on their investment. It's as much 'IT's responsibility to manage upwards as it is for management to manage downwards.