35
votes

What is the most damage (of whatever kind) that you have ever caused with a single mistaken/mistyped/misguided command line? I deleted a production system database by mistake a while back, for example, but I was lucky (i.e. backed-up) and there was no permanent data loss, lost money, property damage etc.

Most importantly (for votes), what do you do to make sure it will not ever happen again?

7
  • 9
    Since this is a poll, it should be CW
    – Eddie
    May 6, 2009 at 2:00
  • Then why aren't polls CW by default?
    – Luke
    May 6, 2009 at 4:19
  • 4
    How should they know it's a poll progmatically?
    – Mikeage
    May 6, 2009 at 8:14
  • 1
    If the mechanism is the same as SO's, even if you can edit a question, you can't change it to CW. Only the OP or a moderator can do that.
    – chaos
    May 6, 2009 at 14:09
  • 9
    Good idea - post your most destructive mistake publicly on the internet for all future employers to see! :) May 13, 2009 at 19:29

81 Answers 81

46
votes

In SQL server, on a production system:

update customer set password = '' <enter>

The most recent backup was like a week old.

To mitigate this, I now usually write a select statement first to make sure I've got the where clause correct, then go back and edit it to insert the set clause and change the statement to update.

6
  • 11
    Ouch. Another option would be 'BEGIN TRANSACTION'; it's nice to know you can roll back. :) May 6, 2009 at 9:35
  • 5
    Combination of the two, select count(*) to see how many I think I'll update, and then update in transaction to do the update. If the counts match commit, if not rollback and figure out why. Of course, I don't do that for "simple" updates, and those are always the ones that screw you. :-)
    – WaldenL
    May 6, 2009 at 12:47
  • Guilty, I've done that myself :( May 13, 2009 at 23:33
  • 8
    Be careful with BEGIN TRANSACTION on a live system (or a test system being used by others too) - make sure you COMMIT or ROLLBACK soon or you may end up deadlocking other processes waiting for locks on resources your transaction holds locks on. Jun 7, 2009 at 20:13
  • On some systems, you can make ; be the statement completion identifier, instead of <enter>
    – J. Polfer
    Jun 25, 2009 at 17:18
37
votes

Biggest mistake? Thinking I had set two variables when I had not. So rm -rf $VARIABLE/$VARIABLE2 became rm -rf /. FreeBSD has recently updated their rm tool so that rm -rf / is not possible anymore precisely because of this mistake!

4
  • 4
    I would like to point out, re-reading my sentence it sounds like I was the reason for changing the way rm worked, this is not the case. Apparently people were using rm in scripts and the same thing happened and they wanted to add a fail-safe.
    – X-Istence
    May 6, 2009 at 7:42
  • 3
    i withdraw my vote then :) May 19, 2009 at 20:33
  • I once saw a guy do rm -rf / usr/local/junk...I saw it, but not in time to keep him from hitting enter. Mar 17, 2011 at 14:37
  • What about "rm -rf /*" instead of "rm -rf /" ? I don't have idea how it looks on BSD's now, but should work ;) Mar 28, 2011 at 5:01
26
votes
shutdown -h now 

meant for the local workstation, but typed it while logged via ssh on production server. Since then I always have hostname in my $PS1.

7
  • At least you probably wouldn't lose any data...
    – Zifre
    May 6, 2009 at 21:10
  • True, but that was really long time ago, IPMI nor KVMoIP where not yet popular. We had to get hold of technician to actually go there and flip the switch.
    – vartec
    May 7, 2009 at 7:48
  • 4
    Check out molly-guard (packages.debian.org/search?keywords=molly-guard)
    – David Holm
    May 29, 2009 at 20:14
  • Even on local workstations, I always use "+1" just in case.
    – msanford
    Jun 7, 2009 at 19:40
  • 3
    I used to put the hostname in the prompt, but now I have patched the shutdown (and related - halt et al) commands to prompt me to enter the current machines name before it will execute the command. That gives me the extra step necessary to realise that I just shutdown the wrong server. Problem is, once I implemented that patch, I never did it again - but its saved a few of my colleagues tho :)
    – Moo
    Jun 11, 2009 at 15:33
26
votes

Omitting the -r from a shutdown command. On a remote server. On the other side of the country. With no IT staff in the remote office.

We've all done it, it's almost like a rite of passage at this stage.

0
25
votes

On a VMS system, I had been using the ASSIGN DCL command to assign logical names, and I wanted to RECALL a previous ASSIGN command line. Now, in VMS, you only typed as many characters of a command to make it unambiguous. So I intended to type

REC ASS

but I accidentally typed

REQ ASS

instead. REQ was sufficiently unambigous for the REQUEST command, which broadcasts the argument to everyone with operator privs (which was everyone in IT). So the entire department received my broadcast message which was simply "ASS".

2
  • Hilarious :-) :-) May 31, 2009 at 8:33
  • 1
    All Software Sucks, as we all know. Why should anyone in IT think it meant something else? Jun 20, 2009 at 16:59
24
votes

On a Solaris system: "killall dataLoader".

'dataLoader' was an app I was working on. On Linux, killall works like pkill. It sends a signal to processes that match a string given as the argument. On Solaris, killall trys to kill everything on the system the current user can kill. I was root.

3
  • been there, done that...
    – James
    Jan 13, 2010 at 16:01
  • 2
    I've done exactly the same two weeks ago in a Power 5 AIX 6 system... Ok, not exactly, I did a "killall -9 java" :)
    – Andor
    Jul 22, 2010 at 19:44
  • 3
    @Andor: Well, you killed java for sure. ;)
    – Bobby
    Mar 17, 2011 at 13:54
16
votes

Once, many many moons ago, I needed to find a particular executable but couldn't remember the complete name of it (but could remember a few of the letters). So I figured I'd check the /usr/bin directory with something like this

rm /usr/bin/i*g*

Strange. Nothing returned. Figuring I just had misremembered the second letter, I tried again with

rm /usr/bin/i*

Again, nothing. After doing the same with /usr/local/bin, /usr/sbin, and anything else I figured it might be in, I realized I'd been mistyping the 'ls' command.

Don't quite know where the brainfart came from, but it's definitely not a mistake I ever made again.

2
  • This is something one should be aware of when reusing paths like, say ls /etc/something, Ctrl+r, Ctrl+a, Del, Del, cat, Enter.
    – Alex
    Jun 28, 2009 at 20:01
  • 8
    Even on my server it does not return anything. Mar 17, 2011 at 14:06
16
votes

Trying to change ownership of everything in a directory, including dot files, with:

chown -R user * .*

Guess what that does?

15
  • 10
    read mail -really fast ???
    – Wiren
    May 29, 2009 at 14:16
  • 5
    Been there, this taught me the .[^.]* pattern really well May 31, 2009 at 18:52
  • 1
    @chaos: The command seems to remove everything in your current directory, and even dot-files Jun 7, 2009 at 20:20
  • 2
    @chaos: It does not remove .. and never has and never will. You are simply mistaken. Please cite a single unix implementation that does have this behavior (which I have never seen and which is documented to not be the behavior of even super-antique unixes like 7th edition unix).
    – chris
    Jun 25, 2009 at 15:44
  • 1
    Would ./.* prevent this, or would it still propagate back up the directory tree via ./../?
    – TJ L
    Jun 25, 2009 at 17:41
15
votes

Meant to destroy /dev/sdb, fortunately I had a good up-to-date backup

dd if=/dev/urandom of=/dev/sda
1
  • Did that once (except with /dev/zero), destroyed the root partition of a Linux server, recovered by copying it all back from an identical server. Sep 28, 2010 at 22:18
15
votes
setup.exe

It was Windows Vista.

14
votes
select * from <File1> join <file2>

On a production box. Note the lack of an on clause. :-) Both tables were multi-million row tables, and this was on an AS/400 in the mid 90s where once the SQL was running you couldn't kill it.

0
14
votes

On one of our data production server, one of my roots typed:

chmod -R 777 /

Because he was getting permissions error with some scripts...

Shortly after that, his private key was removed from all servers and he took care of the 1TB data restore on the data production server...

12
votes

My erm favorite was when I was at university. I was building an application (I don't recall what) and as I wasn't root I had built it with

PREFIX=~username/usr/local

So I could install it to my home directory. Unfortunately it installed into

/home/username/src/app/~username/usr/local

instead. Naturally to delete it as start again i executed

rm -rf ~username

In the source directory.

I wondered why it was taking so long....

:-)

However my worst one was when I was working with a solaris workstation and after getting it all setup we wanted to wipe the configuration ready for live config. So I executed

sys-unconfig

Agreed to the warning message and instead of the machine rebooting and going back to "factory defaults" The xterm window simply said

connection closed by foreign host.

Moral of the story Don't ever leave a root shell open on another host! EVER!!

1
  • I go for hostname in PS1, myself. At some point I used different colours for local and remote logins (as well as different colours for user/root). Sep 28, 2010 at 22:20
12
votes

Many years ago I was at home coding up some stuff in php while working on a project with a buddy of mine who was the maintainer of the project. We were IM'ing one another in a collaboration effort. We always banter back and forth in play.

I was trying to get ssh-agent to work properly on my machine while we were having a religious war of Perl vs PHP. Then I mentioned something about ssh-agent needing to be eval'ed (not sure why I said that). So then he sent me this message in an effort, so I thought, to help me with my problem (bear in mind that I was su -'ed to root):

\# eval $(echo ssh-agent | 
perl -pe 's/h-a/m -r/' | 
perl -pe 's/^ss/r/' | 
perl -pe 's/gent/f \//')

WARNING! DO NOT RUN THAT COMMAND!!!

IF you remove the eval and run the inner command by itself it is:

rm -rf /

It took me all of 4 seconds to notice what was happening but the damage was already done. I had to reinstall my OS. Fortunately, nothing of my work was wiped out except some stuff in /etc iirc. He had a HUGE laugh when I sent him a message of horror asking why he did that. We are both long-term systems engineers. He didn't think I would run it and would be more careful to check it before simply c&p'ing and I just trusted him so I didn't even consider he was playing around. Needless to say, this little story comes up all the time between us. So, I decided to immortalize it.

How have I mitigated this from happening again? I trust no-one!

Another less interesting story is a couple of years back I was doing some work on a mission critical box at work. I had a few terms open to different machine. I needed to remove some superfluous stuff in a directory. Well, I got lost in my terms and accidentally rm'ed . in my local dir but on the wrong host (wrong term)!!. I executed the command in /var/lib/mysql instead of /tmp on the application server (different term). Needless to say, I wiped out the production database. Fortunately, we had a warm standby that we flipped to while me and a co-worker rebuilt the primary from backups and the standby. That took about 18 hours to do.

Mitigation: more careful about what windows I execute commands in before executing them.

1
  • At work we all configured the .bashrc of all ours servers to display usernames in bash in unique colors. That way you allways see in an instant on what machine you are. (Ok, that may become a problem when you're managin more than maybe 20-30 machines... But still very handy.)
    – fgysin
    Sep 2, 2010 at 8:07
11
votes

the short version

#/bin/bash
$0&
$0
8
  • for what it's worth, if you ever do that not as root, run kill -9 -1
    – BCS
    May 5, 2009 at 23:24
  • 3
    what does that do? May 15, 2009 at 2:33
  • 6
    It calls it's self in the background than then without waiting for that to finish, calls it's self again: a.k.a. Fork Bomb
    – BCS
    May 18, 2009 at 16:50
  • 2
    @Masi: Not if the computer is fast enough and the script long enough. Specialy if you don't know it is going to happen.
    – voyager
    Jun 7, 2009 at 22:53
  • 3
    Drop the #!/bin/bash, and I think this would win a code golf for shortest forkbomb in bash.
    – Joey Adams
    Apr 16, 2010 at 16:42
11
votes

I once wanted to delete a bunch of files in a directory.

del *.*

The computer then said "Are you sure? [y/N]" I thought "OF COURSE I'm sure, I wouldn't have typed the darn command in otherwise! Sheesh stupid computers grumble..."

Y <enter>

C:\Windows>_

Um... WTF? Did I just erase my windows directory?....

undelete *.*

In those days of small hard disks I knew what every file in c:\windows was for and what its name was, but even after undeleting everything the system was never the same. I gained a little bit of respect for the "are you sure" prompt. Just a little.

1
  • 1
    I once did exactly the same thing. I got the system to boot and run again, but it never was quite right again. I rebuilt the system about a week later. May 13, 2009 at 23:37
11
votes

I think the most stupid thing I ever did was to remove the default route on out external facing firewall cluster - whilst vpn'ed to my desktop over a hundred miles away.

Fortunately it was in a period designated as planned downtime (just in case), but that did not save me the 200 mile round trip to go and reconfigure the firewall onsite. It also did not help that this took out our internet exposed production systems for the duration of my travel and subsequent fix.

We all know the definition of a nano-second. An ohno-second is even smaller, and is the time between hitting 'enter' and realising your mistake.

11
votes

First of two...

On a Solaris box we had a tar backup of an AIX machine..

One of the Developers typed:

tax xvf AIX_Backup.tar

Of course the paths in the tax were absolute and we ended up making a new distro of unix... Solarix... The only problem with the distro is that it did not boot :(

10
votes

Story told to me:

Another branch called because their local PBX had some issues. After some investigation, we learned that they updated their server, but not their Asterisk configuration. So, the admin decided to instruct the branch guy to redo the configuration.

Admin: "Ok, now, type rm -rf /etc/asterisk"

Guy: "Ok."

Admin: "Now, type cp /var/..."

Guy: "Wait, it is still running..."

Admin: ???... !!!

1
  • 1
    If anything can go wrong, it will go wrong.
    – Joey Adams
    Apr 16, 2010 at 16:49
9
votes
rm -rf / some/path 

instead of

rm -rf /some/path 

Luckily it didn't happened to me ;-)

4
  • happened to a classmate of mine recently. Luckily it was just with /etc /some/other/path
    – Ikke
    May 28, 2009 at 9:04
  • It must be disabled! I think it is now disabled in many OSes. Too afraid to try, on Ubuntu rm -rf /
    – lprsd
    Jun 3, 2009 at 5:11
  • Actually, it would be dangerous to the system only with sudo rm -rf /
    – voyager
    Jun 7, 2009 at 22:56
  • Been there, done that...
    – fgysin
    Sep 2, 2010 at 8:09
9
votes
source ~/.bash_history

My intention was source .bashrc, but I was too hasty with tab-complete...

1
  • +1: That one is awesome. Mar 17, 2011 at 15:02
8
votes

Something along the lines of this:

sudo dd if=/dev/zero of=/dev/sda

I meant sda4. I wiped the entire disk, not just the partition :-(

1
8
votes

XCOPY is a powerful beast - merciless in it's execution and retarded by the fact that its command line args go in reverse from Window's COPY and UNIX's cp.

A couple days ago I accidentally wrote:

xcopy src \path\to\a\new\nonexistent\directory

XCOPY was kind enough to overwrite my src directory with... nothing! And it didn't bother to put the old files in the Recycle bin either.

Oh, and it turns out that XCOPY actually overwrites the same sectors on the disk instead of allocating new ones. I've tried 3 disk recovery programs, and the best one could only recover 3 of the 10 files lost. Of course those 3 files were only vshost.exe and its pals. Swell!

8
votes

had trouble with network (on a remote machine) and just wanted to restart the interface

ifconfig eth0 down && ifconfig eth0 upp

Nowadays, I make sure someone is near the machine before trying things like this (iptables is a good candidate as well). And when nobody was there, I once typed

sleep 600; reboot

into another (screen) terminal, so that it would reboot if I could not ctrl+c the command within 10 minutes.

I learned from that mistake as well (ctrl+c the sleep will run the reboot) and now I use

sleep 600 && reboot

which will enable me to ctrl+c it.

2
  • 1
    shutdown -r +5 will reboot system in 5 minutes, unless you kill shutdown process with killall shutdown for example.
    – gelraen
    Mar 17, 2011 at 14:03
  • 1
    I can just see the admin who makes mihi's mistake under Solaris, then (learning from the mistake) tries gelraen's alternative.
    – outis
    Sep 1, 2011 at 0:06
8
votes
ifconfig eth0 down

Oops, I'm on the external side of eth0. The webserver is on the other side of the world, in a locked room. With no network access to login or reboot. Crap.

6
votes

A free cookie to anyone who can tell me why I was an idiot for trying to remove all hidden files and directories thusly:

rm -rf .*

3
  • 1
    Does your command remove subdirectories below your current directory too? Jun 7, 2009 at 20:37
  • I wonder if your pattern is matching .. hardlink Mar 16, 2010 at 20:54
  • BTW, you both win cookies :-) Mar 16, 2010 at 23:27
6
votes

While cleaning up my home folder on a production webserver, I forgot that I had symlinked the server's web root to a place in my home folder. Not thinking, I ran an rm -rf on that folder and next thing I know, people are calling that the web site is down!

OOPS!

5
votes

I was once comparing data in two folders and ran rsync with the -d option (delete files on destination that are not on the source). And then I switched around source and destination when I ran rsync. That deleted all the new files, that I wanted to backup. Now I learned running rsync with -n (dry-run).

rsync -trvd --stats --progress /destination /source

I had no backup.

4
votes

Once upon a time (probably System III, but it was a long time ago), it was possible to create a file named * by using the right shell quoting. When I found one in my home directory, I had typed rm * and had my finger on the return key when something made me hesitate and think about it...

Creating such a file for other users was a common prank.

If the file is sitting there in a directory, that is a tough one to mitigate. The reflex to just type its name exactly as ls just displayed it is pretty strong.

The other (less harmful) prank was to name files with trailing white space (or only white space) which were much harder to remove...

5
  • 2
    That's what GUIs are for :)
    – Zifre
    May 6, 2009 at 21:11
  • 1
    shell completion is a real time-saver for those trailing-space files :) May 14, 2009 at 12:58
  • 1
    This would have been ca. 1985, and accessed via dialup or via a RS-232 cable from a clone of a VT-100 terminal. Not a terminal emulator, but an actual terminal. I don't recall csh having name completion at the time... ;-) Today, of course, GUIs and name completion make both of those less risky, and lots of OSs make it a lot harder to create such files without resorting to disk sector editing....
    – RBerteig
    May 15, 2009 at 20:18
  • touch ./--help then neither rm --help nor rm * will work...
    – mihi
    Jun 7, 2009 at 18:34
  • You can still create a file named , at least on Linux: " touch '' ". And " rm '*' " also works...
    – sleske
    Sep 15, 2009 at 1:02
4
votes

Due to some bad experience with general JBoss flakiness after a restart, I like to clear JBoss' working files before a restart. I would normally do:

# cd /var/cache/jboss
# rm -rf tmp/* work/*

In order to protect myself from typing any of the many possible disastrous mistakes, like:

  • /tmp/*
  • tmp /*
  • you get the idea

I make the last command:

# sudo -u jboss rm -rf tmp/* work/*

Since the JBoss user would find it difficult to remove any critical files that don't belong to it.

I never actually made that mistake, but I'm safe in the case that I do.

Not the answer you're looking for? Browse other questions tagged .