Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What is the most damage (of whatever kind) that you have ever caused with a single mistaken/mistyped/misguided command line? I deleted a production system database by mistake a while back, for example, but I was lucky (i.e. backed-up) and there was no permanent data loss, lost money, property damage etc.

Most importantly (for votes), what do you do to make sure it will not ever happen again?

share

locked by Michael Hampton Oct 23 '13 at 15:06

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by Mark Henderson Mar 28 '11 at 4:31

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

9  
Since this is a poll, it should be CW –  Eddie May 6 '09 at 2:00
4  
How should they know it's a poll progmatically? –  Mikeage May 6 '09 at 8:14
1  
If the mechanism is the same as SO's, even if you can edit a question, you can't change it to CW. Only the OP or a moderator can do that. –  chaos May 6 '09 at 14:09
9  
Good idea - post your most destructive mistake publicly on the internet for all future employers to see! :) –  Sam Schutte May 13 '09 at 19:29
show 3 more comments

81 Answers 81

up vote 46 down vote accepted

In SQL server, on a production system:

update customer set password = '' <enter>

The most recent backup was like a week old.

To mitigate this, I now usually write a select statement first to make sure I've got the where clause correct, then go back and edit it to insert the set clause and change the statement to update.

share
11  
Ouch. Another option would be 'BEGIN TRANSACTION'; it's nice to know you can roll back. :) –  Murali Suriar May 6 '09 at 9:35
5  
Combination of the two, select count(*) to see how many I think I'll update, and then update in transaction to do the update. If the counts match commit, if not rollback and figure out why. Of course, I don't do that for "simple" updates, and those are always the ones that screw you. :-) –  WaldenL May 6 '09 at 12:47
8  
Be careful with BEGIN TRANSACTION on a live system (or a test system being used by others too) - make sure you COMMIT or ROLLBACK soon or you may end up deadlocking other processes waiting for locks on resources your transaction holds locks on. –  David Spillett Jun 7 '09 at 20:13
show 3 more comments

Biggest mistake? Thinking I had set two variables when I had not. So rm -rf $VARIABLE/$VARIABLE2 became rm -rf /. FreeBSD has recently updated their rm tool so that rm -rf / is not possible anymore precisely because of this mistake!

share
4  
I would like to point out, re-reading my sentence it sounds like I was the reason for changing the way rm worked, this is not the case. Apparently people were using rm in scripts and the same thing happened and they wanted to add a fail-safe. –  X-Istence May 6 '09 at 7:42
3  
i withdraw my vote then :) –  Colin Pickard May 19 '09 at 20:33
show 2 more comments
shutdown -h now

meant for the local workstation, but typed it while logged via ssh on production server. Since then I always have hostname in my $PS1.

share
4  
Check out molly-guard (packages.debian.org/search?keywords=molly-guard) –  David Holm May 29 '09 at 20:14
3  
I used to put the hostname in the prompt, but now I have patched the shutdown (and related - halt et al) commands to prompt me to enter the current machines name before it will execute the command. That gives me the extra step necessary to realise that I just shutdown the wrong server. Problem is, once I implemented that patch, I never did it again - but its saved a few of my colleagues tho :) –  Moo Jun 11 '09 at 15:33
show 5 more comments

Omitting the -r from a shutdown command. On a remote server. On the other side of the country. With no IT staff in the remote office.

We've all done it, it's almost like a rite of passage at this stage.

share
2  
Thank god for Remote Power Control and KVMs! –  Dean Perry May 14 '11 at 20:40
add comment

On a VMS system, I had been using the ASSIGN DCL command to assign logical names, and I wanted to RECALL a previous ASSIGN command line. Now, in VMS, you only typed as many characters of a command to make it unambiguous. So I intended to type

REC ASS

but I accidentally typed

REQ ASS

instead. REQ was sufficiently unambigous for the REQUEST command, which broadcasts the argument to everyone with operator privs (which was everyone in IT). So the entire department received my broadcast message which was simply "ASS".

share
1  
All Software Sucks, as we all know. Why should anyone in IT think it meant something else? –  Adriano Varoli Piazza Jun 20 '09 at 16:59
show 1 more comment

On a Solaris system: "killall dataLoader".

'dataLoader' was an app I was working on. On Linux, killall works like pkill. It sends a signal to processes that match a string given as the argument. On Solaris, killall trys to kill everything on the system the current user can kill. I was root.

share
2  
I've done exactly the same two weeks ago in a Power 5 AIX 6 system... Ok, not exactly, I did a "killall -9 java" :) –  Andor Jul 22 '10 at 19:44
3  
@Andor: Well, you killed java for sure. ;) –  Bobby Mar 17 '11 at 13:54
show 1 more comment

Once, many many moons ago, I needed to find a particular executable but couldn't remember the complete name of it (but could remember a few of the letters). So I figured I'd check the /usr/bin directory with something like this

rm /usr/bin/i*g*

Strange. Nothing returned. Figuring I just had misremembered the second letter, I tried again with

rm /usr/bin/i*

Again, nothing. After doing the same with /usr/local/bin, /usr/sbin, and anything else I figured it might be in, I realized I'd been mistyping the 'ls' command.

Don't quite know where the brainfart came from, but it's definitely not a mistake I ever made again.

share
8  
Even on my server it does not return anything. –  Mircea Vutcovici Mar 17 '11 at 14:06
show 1 more comment
setup.exe

It was Windows Vista.

share
add comment

Trying to change ownership of everything in a directory, including dot files, with:

chown -R user * .*

Guess what that does?

share
10  
read mail -really fast ??? –  superwiren May 29 '09 at 14:16
5  
Been there, this taught me the .[^.]* pattern really well –  Maciej Pasternacki May 31 '09 at 18:52
1  
@chaos: The command seems to remove everything in your current directory, and even dot-files –  Masi Jun 7 '09 at 20:20
2  
@chaos: It does not remove .. and never has and never will. You are simply mistaken. Please cite a single unix implementation that does have this behavior (which I have never seen and which is documented to not be the behavior of even super-antique unixes like 7th edition unix). –  chris Jun 25 '09 at 15:44
1  
Would ./.* prevent this, or would it still propagate back up the directory tree via ./../? –  tj111 Jun 25 '09 at 17:41
show 10 more comments

Meant to destroy /dev/sdb, fortunately I had a good up-to-date backup

dd if=/dev/urandom of=/dev/sda
share
show 1 more comment
select * from <File1> join <file2>

On a production box. Note the lack of an on clause. :-) Both tables were multi-million row tables, and this was on an AS/400 in the mid 90s where once the SQL was running you couldn't kill it.

share
add comment

On one of our data production server, one of my roots typed:

chmod -R 777 /

Because he was getting permissions error with some scripts...

Shortly after that, his private key was removed from all servers and he took care of the 1TB data restore on the data production server...

share
add comment

My erm favorite was when I was at university. I was building an application (I don't recall what) and as I wasn't root I had built it with

PREFIX=~username/usr/local

So I could install it to my home directory. Unfortunately it installed into

/home/username/src/app/~username/usr/local

instead. Naturally to delete it as start again i executed

rm -rf ~username

In the source directory.

I wondered why it was taking so long....

:-)

However my worst one was when I was working with a solaris workstation and after getting it all setup we wanted to wipe the configuration ready for live config. So I executed

sys-unconfig

Agreed to the warning message and instead of the machine rebooting and going back to "factory defaults" The xterm window simply said

connection closed by foreign host.

Moral of the story Don't ever leave a root shell open on another host! EVER!!

share
show 1 more comment

Many years ago I was at home coding up some stuff in php while working on a project with a buddy of mine who was the maintainer of the project. We were IM'ing one another in a collaboration effort. We always banter back and forth in play.

I was trying to get ssh-agent to work properly on my machine while we were having a religious war of Perl vs PHP. Then I mentioned something about ssh-agent needing to be eval'ed (not sure why I said that). So then he sent me this message in an effort, so I thought, to help me with my problem (bear in mind that I was su -'ed to root):

\# eval $(echo ssh-agent | 
perl -pe 's/h-a/m -r/' | 
perl -pe 's/^ss/r/' | 
perl -pe 's/gent/f \//')

WARNING! DO NOT RUN THAT COMMAND!!!

IF you remove the eval and run the inner command by itself it is:

rm -rf /

It took me all of 4 seconds to notice what was happening but the damage was already done. I had to reinstall my OS. Fortunately, nothing of my work was wiped out except some stuff in /etc iirc. He had a HUGE laugh when I sent him a message of horror asking why he did that. We are both long-term systems engineers. He didn't think I would run it and would be more careful to check it before simply c&p'ing and I just trusted him so I didn't even consider he was playing around. Needless to say, this little story comes up all the time between us. So, I decided to immortalize it.

How have I mitigated this from happening again? I trust no-one!

Another less interesting story is a couple of years back I was doing some work on a mission critical box at work. I had a few terms open to different machine. I needed to remove some superfluous stuff in a directory. Well, I got lost in my terms and accidentally rm'ed . in my local dir but on the wrong host (wrong term)!!. I executed the command in /var/lib/mysql instead of /tmp on the application server (different term). Needless to say, I wiped out the production database. Fortunately, we had a warm standby that we flipped to while me and a co-worker rebuilt the primary from backups and the standby. That took about 18 hours to do.

Mitigation: more careful about what windows I execute commands in before executing them.

share
show 2 more comments

the short version

#/bin/bash
$0&
$0
share
3  
what does that do? –  Nathan DeWitt May 15 '09 at 2:33
6  
It calls it's self in the background than then without waiting for that to finish, calls it's self again: a.k.a. Fork Bomb –  BCS May 18 '09 at 16:50
2  
@Masi: Not if the computer is fast enough and the script long enough. Specialy if you don't know it is going to happen. –  voyager Jun 7 '09 at 22:53
3  
Drop the #!/bin/bash, and I think this would win a code golf for shortest forkbomb in bash. –  Joey Adams Apr 16 '10 at 16:42
show 4 more comments

I once wanted to delete a bunch of files in a directory.

del *.*

The computer then said "Are you sure? [y/N]" I thought "OF COURSE I'm sure, I wouldn't have typed the darn command in otherwise! Sheesh stupid computers grumble..."

Y <enter>

C:\Windows>_

Um... WTF? Did I just erase my windows directory?....

undelete *.*

In those days of small hard disks I knew what every file in c:\windows was for and what its name was, but even after undeleting everything the system was never the same. I gained a little bit of respect for the "are you sure" prompt. Just a little.

share
1  
I once did exactly the same thing. I got the system to boot and run again, but it never was quite right again. I rebuilt the system about a week later. –  Jim OHalloran May 13 '09 at 23:37
add comment

I think the most stupid thing I ever did was to remove the default route on out external facing firewall cluster - whilst vpn'ed to my desktop over a hundred miles away.

Fortunately it was in a period designated as planned downtime (just in case), but that did not save me the 200 mile round trip to go and reconfigure the firewall onsite. It also did not help that this took out our internet exposed production systems for the duration of my travel and subsequent fix.

We all know the definition of a nano-second. An ohno-second is even smaller, and is the time between hitting 'enter' and realising your mistake.

share
add comment

First of two...

On a Solaris box we had a tar backup of an AIX machine..

One of the Developers typed:

tax xvf AIX_Backup.tar

Of course the paths in the tax were absolute and we ended up making a new distro of unix... Solarix... The only problem with the distro is that it did not boot :(

share
add comment

Story told to me:

Another branch called because their local PBX had some issues. After some investigation, we learned that they updated their server, but not their Asterisk configuration. So, the admin decided to instruct the branch guy to redo the configuration.

Admin: "Ok, now, type rm -rf /etc/asterisk"

Guy: "Ok."

Admin: "Now, type cp /var/..."

Guy: "Wait, it is still running..."

Admin: ???... !!!

share
1  
If anything can go wrong, it will go wrong. –  Joey Adams Apr 16 '10 at 16:49
add comment
rm -rf / some/path

instead of

rm -rf /some/path

Luckily it didn't happened to me ;-)

share
show 4 more comments
source ~/.bash_history

My intention was source .bashrc, but I was too hasty with tab-complete...

share
show 1 more comment

Something along the lines of this:

sudo dd if=/dev/zero of=/dev/sda

I meant sda4. I wiped the entire disk, not just the partition :-(

share
show 1 more comment

XCOPY is a powerful beast - merciless in it's execution and retarded by the fact that its command line args go in reverse from Window's COPY and UNIX's cp.

A couple days ago I accidentally wrote:

xcopy src \path\to\a\new\nonexistent\directory

XCOPY was kind enough to overwrite my src directory with... nothing! And it didn't bother to put the old files in the Recycle bin either.

Oh, and it turns out that XCOPY actually overwrites the same sectors on the disk instead of allocating new ones. I've tried 3 disk recovery programs, and the best one could only recover 3 of the 10 files lost. Of course those 3 files were only vshost.exe and its pals. Swell!

share
add comment

had trouble with network (on a remote machine) and just wanted to restart the interface

ifconfig eth0 down && ifconfig eth0 upp

Nowadays, I make sure someone is near the machine before trying things like this (iptables is a good candidate as well). And when nobody was there, I once typed

sleep 600; reboot

into another (screen) terminal, so that it would reboot if I could not ctrl+c the command within 10 minutes.

I learned from that mistake as well (ctrl+c the sleep will run the reboot) and now I use

sleep 600 && reboot

which will enable me to ctrl+c it.

share
1  
shutdown -r +5 will reboot system in 5 minutes, unless you kill shutdown process with killall shutdown for example. –  gelraen Mar 17 '11 at 14:03
1  
I can just see the admin who makes mihi's mistake under Solaris, then (learning from the mistake) tries gelraen's alternative. –  outis Sep 1 '11 at 0:06
add comment
ifconfig eth0 down

Oops, I'm on the external side of eth0. The webserver is on the other side of the world, in a locked room. With no network access to login or reboot. Crap.

share
add comment

A free cookie to anyone who can tell me why I was an idiot for trying to remove all hidden files and directories thusly:

rm -rf .*

share
1  
Does your command remove subdirectories below your current directory too? –  Masi Jun 7 '09 at 20:37
show 2 more comments

While cleaning up my home folder on a production webserver, I forgot that I had symlinked the server's web root to a place in my home folder. Not thinking, I ran an rm -rf on that folder and next thing I know, people are calling that the web site is down!

OOPS!

share
add comment

I was once comparing data in two folders and ran rsync with the -d option (delete files on destination that are not on the source). And then I switched around source and destination when I ran rsync. That deleted all the new files, that I wanted to backup. Now I learned running rsync with -n (dry-run).

rsync -trvd --stats --progress /destination /source

I had no backup.

share
add comment

Once upon a time (probably System III, but it was a long time ago), it was possible to create a file named * by using the right shell quoting. When I found one in my home directory, I had typed rm * and had my finger on the return key when something made me hesitate and think about it...

Creating such a file for other users was a common prank.

If the file is sitting there in a directory, that is a tough one to mitigate. The reflex to just type its name exactly as ls just displayed it is pretty strong.

The other (less harmful) prank was to name files with trailing white space (or only white space) which were much harder to remove...

share
2  
That's what GUIs are for :) –  Zifre May 6 '09 at 21:11
1  
shell completion is a real time-saver for those trailing-space files :) –  Mr. Shiny and New 安宇 May 14 '09 at 12:58
1  
This would have been ca. 1985, and accessed via dialup or via a RS-232 cable from a clone of a VT-100 terminal. Not a terminal emulator, but an actual terminal. I don't recall csh having name completion at the time... ;-) Today, of course, GUIs and name completion make both of those less risky, and lots of OSs make it a lot harder to create such files without resorting to disk sector editing.... –  RBerteig May 15 '09 at 20:18
show 2 more comments

Due to some bad experience with general JBoss flakiness after a restart, I like to clear JBoss' working files before a restart. I would normally do:

# cd /var/cache/jboss
# rm -rf tmp/* work/*

In order to protect myself from typing any of the many possible disastrous mistakes, like:

  • /tmp/*
  • tmp /*
  • you get the idea

I make the last command:

# sudo -u jboss rm -rf tmp/* work/*

Since the JBoss user would find it difficult to remove any critical files that don't belong to it.

I never actually made that mistake, but I'm safe in the case that I do.

share
add comment

protected by Chris S Mar 17 '11 at 12:23

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.