# What do you do to prevent stupid mistakes?

I saw this comment:

"Been there, done that. We aliased the killall-command on the solaris-boxes after that: alias killall='echo ORLLY?' =) – Commander Keen May 28 at 12:03"

It made me wonder, what do sysadmins do to prevent stupid mistakes, either for themselves or others?

In a previous organization where every change had to go through change management (even adding a host to /etc/hosts), we did copy/paste instructions for all change records. If it required additional commands/procedures that weren't in the record, a new ticket was opened.

-

## locked by Chris S♦Dec 16 '12 at 6:30

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

Perhaps this should be community wiki ... –  tomjedrz Jun 5 '09 at 19:20
indeed, and so it is. –  jtimberman Jun 9 '09 at 0:34

Surprised no-one has added... go home when you're tired! Brain does not do its best work when you're 10 or 12 hours into the day, go home, grab a beer, get some shut-eye and hit the ground running in the morning!

I also find "peer review" useful... "hey bob, I'm just going to bargle the frargle - you see anything up with that?" just saying it out loud can solidify what you are doing in your mind.

Now we return you to "technical solutions for a single, tired brain" ;)

-
I agree with not doing something critical when you tired... better do it the next day. –  Hapkido Jun 6 '09 at 1:44
We have a policy at our office, that no major changes get made after 3pm, because we're all starting to look forward to hometime and start making mistakes. –  Mark Henderson Jun 7 '09 at 7:07
I like that - an official "3pm" policy - we can add that to "never make a big change on a friday" (As it costs more to rectify mistakes on saturday!) –  Tom Newton Jun 7 '09 at 12:43

I don't do anything to prevent them, but rather, plan around the expectation that I'm bound to make horrific ones.

-

Just don't make a mistake that was obviously noted or explained in the documentation... which is a good advice: READ THE DOCUMENTATION FIRST

-
Amen! Read and make sure there's no word or phrase that you don't understand. That can always throw you off. –  Hondalex Jun 5 '09 at 20:50
AKA RTFM....... –  Wayne Koorts Jun 9 '09 at 0:49
And read the entire log message, end to end, when you see an error occur. –  jtimberman Jul 7 '09 at 1:46

I take a maxim from my carpenter freinds...

Measure twice cut once.

Before doing things that may result in me standing in a unemployment queue...

Think twice run once.

-
Peer review is good for this! Adding comment so I remember to +1 when I get votes back. –  jtimberman Jun 5 '09 at 19:38
No matter how many times I cut this board its still too short. Kinda like no matter how many times I reboot this computer it won't start. –  SpaceManSpiff Jun 6 '09 at 3:40

I'm strongly against protective aliases like rm="rm -i".

Once you retrain your brain to expect rm to be safe, you become very dangerous on any machine without those protections. I'd much rather train my fingers to type "rm -i" or just use mv instead of rm, since those aren't likely to get me into trouble in a new environment.

-
So what do you do proactively to prevent stupid mistakes of your own, or of other admins? –  jtimberman Jun 6 '09 at 7:05
Get in the habit of not pressing enter quickly when doing destructive commands. –  Matt Simmons Jun 6 '09 at 12:14

Among others, these might prove valuable:

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
set -o noclobber


Also, if you often do a lot of browsing your database, but don't often have to make a lot of changes, create a separate user that only has SELECT privileges on tables.

-
commenting so I remember to +1 when I get votes back tomorrow. –  jtimberman Jun 5 '09 at 19:37

Many commands have an option that just shows the output as if the command were run, but doesn't actually do it. (E.g. rsync --dry-run) Look for them, and use them.

-

Automate whatever you can. Whenever you rely on yourself doing something manually, you allow the possibility for mistakes.

Use various techniques to write robust shell scripts.

When preparing a batch job (for loop, clusterssh job, etc.), prepend the commands that do stuff with echo to make sure that they look sane.

-

Checklists and scripting

For every complex task, there is a checklist or a script that will save your butt.

If it's good enough for surgeons and airline pilots, it's good enough for us.

-

When it really matters, I sit down a week before, and write the entire thing down in a Wiki page. The intent is to cut and paste the entire action without a single live edit. Basically, write a script, but with a human able to abort and restart any action.

The next day, I read it and fix it.

The next day, I read it again and fix it.

The next day, I read it again and fix it.

2-3 days before the real execution, I run it once on a machine that I can mess up. Scratch that, a machine that I will mess up. Then I fix it the wiki page.

The next day, I read it again and fix it.

On the actually execution date, I run it on the first production system. Then I fix the wiki page.

The 2nd production system usually works without a problem.

Example use: Migrating from an old SAN to a new SAN, with no downtime. Including "hot" Fiber Channel cable migrations.

It sucked. But what a rush when I pulled it off!

-

If you have no idea what you're doing, hire someone else to do it instead of trying to figure it out yourself.

-
Please! If I did that, I'd just sit at my desk reading ServerFault all day! –  Matt Simmons Jun 5 '09 at 22:06

We have a policy to only edit system configuration with a script that backs up the configuration file first, before letting you edit it. It's basically a wrapper around vi, but it does the job pretty well: it's very easy to roll back even the most complex changes.

-
The other solution here is to check your config files (e.g. all of /etc) into version control (e.g. git), and then have a cronjob commit/push new changes every <timeframe> –  Dan Udey Jun 5 '09 at 19:26
There is a tool for this job called etckeeper –  cstamas Jun 5 '09 at 20:07

I'm careful to be specific about what I'm doing before I do anything. Writing a script that deletes all files in the current working directory for example can work in my test, but do something bad later on.

-
I worked on a team where the primary tools / script author had a space in the wrong place: "rm -r / *" - ran the script and wiped out an entire production cluster. –  jtimberman Jun 5 '09 at 19:36

I think it's hard to protect yourself from yourself.. if I knew I was doing it wrong in the first place, then I wouldn't have done it. That said, I are a couple of thinks I try to remember:

1. Read through instructions before attempting the task. This is sometimes hard because who really likes instructions?
2. Read ALL prompts. If there is a prompt, it was designed for a purpose.. reading these and not rushing through clicking has definitely saved me a couple of Homer DOH! moments.
3. Document difficult tasks. Most of the time when I complete something new and challenging and has not been previously documented, I'll take the time to write up some notes on the task.
4. Backups
-

We colored the bash prompts differently on dev / stage / production systems. "Oh shit, I was on production?!?!?!"

-

Some tips for linux machines:

alias rm="rm -i"
alias mv="mv -i"

• disable ctrl-alt-delete
• install molly-guard : protects remote machines from accidental shutdowns/reboots
• install metche : configuration monitor to ease collective administration
-
disabling ctl-alt-del is a good 'security' practice anyway. +1 when I have votes! –  jtimberman Jun 5 '09 at 19:38

I don't get out of bed.

Failing that I read twice and click once.

-

Document everything you do, you can use this later on as a script when you have to redo the task. Peer review. Double check and use a stage machine to test the stuff you want to do/change.Automate and keep everything configuration related under some version control system.

Most important "don't be afraid of making mistakes - you will do them". Most often this will make it easier for you to work. Mistakes will happen just be prepared to be able to clean up the mistakes nicely.

-

Since I'm too noob to comment on [http://serverfault.com/questions/20738/what-do-you-do-to-prevent-stupid-mistakes/20849#20849][1], I have to post another answer.

This is how I have colored various command prompts: $cat ~/.bashrc export FGGRAY=37 export BGRED=41 export BGYELLOW=43 export BGGREEN=42 export HIGHLIGHT=01 export NORMAL=00 export PS1="[\u@$\e[{FGGRAY};{BGRED};{HIGHLIGHT}m$\h$\e[{NORMAL}m$ \W]\\$ "


$cat ~/.cshrc setenv FGGRAY 37 setenv BGRED 41 setenv BGYELLOW 43 setenv BGGREEN 42 setenv HIGHLIGHT 01 setenv NORMAL 00 setenv ESC "^[" set prompt = "[%n@%{${ESC}[${FGGRAY};${BGRED};${HIGHLIGHT}m%}%m%{${ESC}[\${NORMAL}m%} %~]%# "


It took me surprisingly long to get those prompts working and somewhat readable. Naming the colors made it easy to change a system from production to staging and back (because our staging machine became "production" during beta testing cycles, which was part of the problem).

The astute reader will note that I'm using ANSI escape sequences which don't work everywhere. They worked fine on RedHat, but I haven't tested other OSes.

-

Don't work on sensitive things when you feel tired!

-

By far the most widespread practice in this vein is to set alias rm="rm -i" and alias mv="mv -i".

-

automated versioning on the most important configuration files and logon scripts, so everything remains tracable.

-

I suppose it really depends on your business. In my previous post as a Jr. Linux Sysadmin, anything going bad was VERY bad. We had clients who depended on things, programmers who didn't do a great job of securing/saving their code, and people in other departments messing with things they had no right to touch.

In my current position, mistakes aren't to terribly bad. The other day, my boss accidentally rm -rf *ed the wrong directory. was it a pain to rewrite the scripts? you bet. did we lose much money? nope.

All I can say is follow the mantra previous mentioned: think twice, do once. And, because we all know that doesn't always work out, have some kind of recovery plan. Personally I'm a fan of an Rsync'd directory that saves all important files nightly, but that's because it works for me. other people may need backup solutions that are far more frequent.

-

In addition to many things listed, I use Zsh as my shell.

/var/lib/mysql% rm ib_ *
zsh: sure you want to delete all the files in /var/lib/mysql [yn]? n

-

Some sensible and dangerous tasks are performed in pairs, not alone. GNU screen is used when possible, so the same terminal is shared by two admins working together.

For example, once I had a RAID disk failing when I was 300+ Km away from the server, and the on-site admin wasn't too secure of the procedure. He correctly identified and replaced the failing disk, but was afraid of dealing with the beast that is the RAID command-line management interface (called afacli). It was a tight situation for him: the array was degraded meaning that if another disk failed serious data loss would ensue.

So, we joined a shared screen session, and I watched him issuing the commands for setting the new disk as a fallback, then watching the RAID to rebuild itself in the new disk.

-