4
votes

I saw this comment:

"Been there, done that. We aliased the killall-command on the solaris-boxes after that: alias killall='echo ORLLY?' =) – Commander Keen May 28 at 12:03"

In response to an answer.

It made me wonder, what do sysadmins do to prevent stupid mistakes, either for themselves or others?

In a previous organization where every change had to go through change management (even adding a host to /etc/hosts), we did copy/paste instructions for all change records. If it required additional commands/procedures that weren't in the record, a new ticket was opened.

2
  • 5
    Perhaps this should be community wiki ...
    – tomjedrz
    Jun 5, 2009 at 19:20
  • indeed, and so it is.
    – jtimberman
    Jun 9, 2009 at 0:34

25 Answers 25

9
votes

Surprised no-one has added... go home when you're tired! Brain does not do its best work when you're 10 or 12 hours into the day, go home, grab a beer, get some shut-eye and hit the ground running in the morning!

I also find "peer review" useful... "hey bob, I'm just going to bargle the frargle - you see anything up with that?" just saying it out loud can solidify what you are doing in your mind.

Now we return you to "technical solutions for a single, tired brain" ;)

3
  • I agree with not doing something critical when you tired... better do it the next day.
    – Hapkido
    Jun 6, 2009 at 1:44
  • 1
    We have a policy at our office, that no major changes get made after 3pm, because we're all starting to look forward to hometime and start making mistakes. Jun 7, 2009 at 7:07
  • 2
    I like that - an official "3pm" policy - we can add that to "never make a big change on a friday" (As it costs more to rectify mistakes on saturday!)
    – Tom Newton
    Jun 7, 2009 at 12:43
8
votes

I don't do anything to prevent them, but rather, plan around the expectation that I'm bound to make horrific ones.

5
votes

Just don't make a mistake that was obviously noted or explained in the documentation... which is a good advice: READ THE DOCUMENTATION FIRST

2
  • 1
    Amen! Read and make sure there's no word or phrase that you don't understand. That can always throw you off.
    – Hondalex
    Jun 5, 2009 at 20:50
  • 1
    And read the entire log message, end to end, when you see an error occur.
    – jtimberman
    Jul 7, 2009 at 1:46
5
votes

I take a maxim from my carpenter freinds...

Measure twice cut once.

Before doing things that may result in me standing in a unemployment queue...

Think twice run once.

2
  • Peer review is good for this! Adding comment so I remember to +1 when I get votes back.
    – jtimberman
    Jun 5, 2009 at 19:38
  • No matter how many times I cut this board its still too short. Kinda like no matter how many times I reboot this computer it won't start. Jun 6, 2009 at 3:40
5
votes

I'm strongly against protective aliases like rm="rm -i".

Once you retrain your brain to expect rm to be safe, you become very dangerous on any machine without those protections. I'd much rather train my fingers to type "rm -i" or just use mv instead of rm, since those aren't likely to get me into trouble in a new environment.

2
  • So what do you do proactively to prevent stupid mistakes of your own, or of other admins?
    – jtimberman
    Jun 6, 2009 at 7:05
  • Get in the habit of not pressing enter quickly when doing destructive commands. Jun 6, 2009 at 12:14
4
votes

Among others, these might prove valuable:

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
alias mysql='mysql --safe-updates' (or add to your .my.cnf)
set -o noclobber

Also, if you often do a lot of browsing your database, but don't often have to make a lot of changes, create a separate user that only has SELECT privileges on tables.

1
  • commenting so I remember to +1 when I get votes back tomorrow.
    – jtimberman
    Jun 5, 2009 at 19:37
4
votes

Many commands have an option that just shows the output as if the command were run, but doesn't actually do it. (E.g. rsync --dry-run) Look for them, and use them.

4
votes

Automate whatever you can. Whenever you rely on yourself doing something manually, you allow the possibility for mistakes.

Use various techniques to write robust shell scripts.

When preparing a batch job (for loop, clusterssh job, etc.), prepend the commands that do stuff with echo to make sure that they look sane.

4
votes

Checklists and scripting

For every complex task, there is a checklist or a script that will save your butt.

If it's good enough for surgeons and airline pilots, it's good enough for us.

3
votes

When it really matters, I sit down a week before, and write the entire thing down in a Wiki page. The intent is to cut and paste the entire action without a single live edit. Basically, write a script, but with a human able to abort and restart any action.

The next day, I read it and fix it.

The next day, I read it again and fix it.

The next day, I read it again and fix it.

2-3 days before the real execution, I run it once on a machine that I can mess up. Scratch that, a machine that I will mess up. Then I fix it the wiki page.

The next day, I read it again and fix it.

On the actually execution date, I run it on the first production system. Then I fix the wiki page.

The 2nd production system usually works without a problem.

Example use: Migrating from an old SAN to a new SAN, with no downtime. Including "hot" Fiber Channel cable migrations.

It sucked. But what a rush when I pulled it off!

2
votes

If you have no idea what you're doing, hire someone else to do it instead of trying to figure it out yourself.

1
  • 1
    Please! If I did that, I'd just sit at my desk reading ServerFault all day! Jun 5, 2009 at 22:06
2
votes

We have a policy to only edit system configuration with a script that backs up the configuration file first, before letting you edit it. It's basically a wrapper around vi, but it does the job pretty well: it's very easy to roll back even the most complex changes.

2
  • 2
    The other solution here is to check your config files (e.g. all of /etc) into version control (e.g. git), and then have a cronjob commit/push new changes every <timeframe>
    – Dan Udey
    Jun 5, 2009 at 19:26
  • 1
    There is a tool for this job called etckeeper
    – cstamas
    Jun 5, 2009 at 20:07
2
votes

I'm careful to be specific about what I'm doing before I do anything. Writing a script that deletes all files in the current working directory for example can work in my test, but do something bad later on.

1
  • I worked on a team where the primary tools / script author had a space in the wrong place: "rm -r / *" - ran the script and wiped out an entire production cluster.
    – jtimberman
    Jun 5, 2009 at 19:36
2
votes

I think it's hard to protect yourself from yourself.. if I knew I was doing it wrong in the first place, then I wouldn't have done it. That said, I are a couple of thinks I try to remember:

  1. Read through instructions before attempting the task. This is sometimes hard because who really likes instructions?
  2. Read ALL prompts. If there is a prompt, it was designed for a purpose.. reading these and not rushing through clicking has definitely saved me a couple of Homer DOH! moments.
  3. Document difficult tasks. Most of the time when I complete something new and challenging and has not been previously documented, I'll take the time to write up some notes on the task.
  4. Backups
2
votes

We colored the bash prompts differently on dev / stage / production systems. "Oh shit, I was on production?!?!?!"

1
vote

Some tips for linux machines:

alias rm="rm -i"
alias mv="mv -i"
  • disable ctrl-alt-delete
  • install molly-guard : protects remote machines from accidental shutdowns/reboots
  • install metche : configuration monitor to ease collective administration
1
  • disabling ctl-alt-del is a good 'security' practice anyway. +1 when I have votes!
    – jtimberman
    Jun 5, 2009 at 19:38
1
vote

I don't get out of bed.

Failing that I read twice and click once.

1
vote

Document everything you do, you can use this later on as a script when you have to redo the task. Peer review. Double check and use a stage machine to test the stuff you want to do/change.Automate and keep everything configuration related under some version control system.

Most important "don't be afraid of making mistakes - you will do them". Most often this will make it easier for you to work. Mistakes will happen just be prepared to be able to clean up the mistakes nicely.

1
vote

Since I'm too noob to comment on [What do you do to prevent stupid mistakes?, I have to post another answer.

This is how I have colored various command prompts: $ cat ~/.bashrc

export FGGRAY=37
export BGRED=41
export BGYELLOW=43
export BGGREEN=42
export HIGHLIGHT=01
export NORMAL=00

export PS1="[\u@\[\e[${FGGRAY};${BGRED};${HIGHLIGHT}m\]\h\[\e[${NORMAL}m\] \W]\\$ "

$ cat ~/.cshrc

setenv FGGRAY 37
setenv BGRED 41
setenv BGYELLOW 43
setenv BGGREEN 42
setenv HIGHLIGHT 01
setenv NORMAL 00
setenv ESC "^["

set prompt = "[%n@%{${ESC}[${FGGRAY};${BGRED};${HIGHLIGHT}m%}%m%{${ESC}[${NORMAL}m%} %~]%# "

It took me surprisingly long to get those prompts working and somewhat readable. Naming the colors made it easy to change a system from production to staging and back (because our staging machine became "production" during beta testing cycles, which was part of the problem).

The astute reader will note that I'm using ANSI escape sequences which don't work everywhere. They worked fine on RedHat, but I haven't tested other OSes.

[1]: Trey's answer about colored prompts above

1
vote

Don't work on sensitive things when you feel tired!

0
votes

By far the most widespread practice in this vein is to set alias rm="rm -i" and alias mv="mv -i".

0
votes

automated versioning on the most important configuration files and logon scripts, so everything remains tracable.

0
votes

I suppose it really depends on your business. In my previous post as a Jr. Linux Sysadmin, anything going bad was VERY bad. We had clients who depended on things, programmers who didn't do a great job of securing/saving their code, and people in other departments messing with things they had no right to touch.

In my current position, mistakes aren't to terribly bad. The other day, my boss accidentally rm -rf *ed the wrong directory. was it a pain to rewrite the scripts? you bet. did we lose much money? nope.

All I can say is follow the mantra previous mentioned: think twice, do once. And, because we all know that doesn't always work out, have some kind of recovery plan. Personally I'm a fan of an Rsync'd directory that saves all important files nightly, but that's because it works for me. other people may need backup solutions that are far more frequent.

0
votes

In addition to many things listed, I use Zsh as my shell.

/var/lib/mysql% rm ib_ *
zsh: sure you want to delete all the files in /var/lib/mysql [yn]? n
0
votes

Some sensible and dangerous tasks are performed in pairs, not alone. GNU screen is used when possible, so the same terminal is shared by two admins working together.

For example, once I had a RAID disk failing when I was 300+ Km away from the server, and the on-site admin wasn't too secure of the procedure. He correctly identified and replaced the failing disk, but was afraid of dealing with the beast that is the RAID command-line management interface (called afacli). It was a tight situation for him: the array was degraded meaning that if another disk failed serious data loss would ensue.

So, we joined a shared screen session, and I watched him issuing the commands for setting the new disk as a fallback, then watching the RAID to rebuild itself in the new disk.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .