This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here.

More info: http://serverfault.com/faq


What is the most damage (of whatever kind) that you have ever caused with a single mistaken/mistyped/misguided command line? I deleted a production system database by mistake a while back, for example, but I was lucky (i.e. backed-up) and there was no permanent data loss, lost money, property damage etc.

Most importantly (for votes), what do you do to make sure it will not ever happen again?

link|improve this question
9  
Since this is a poll, it should be CW – Eddie May 6 '09 at 2:00
4  
How should they know it's a poll progmatically? – Mikeage May 6 '09 at 8:14
1  
If the mechanism is the same as SO's, even if you can edit a question, you can't change it to CW. Only the OP or a moderator can do that. – chaos May 6 '09 at 14:09
9  
Good idea - post your most destructive mistake publicly on the internet for all future employers to see! :) – Sam Schutte May 13 '09 at 19:29
show 3 more comments
feedback

closed as off topic by Mark Henderson Mar 28 '11 at 4:31

Questions on Server Fault are expected to generally relate to servers, networking, or desktop infrastructure, within the scope defined in the faq.

81 Answers

1 2 3
up vote 44 down vote accepted

In SQL server, on a production system:

update customer set password = '' <enter>

The most recent backup was like a week old.

To mitigate this, I now usually write a select statement first to make sure I've got the where clause correct, then go back and edit it to insert the set clause and change the statement to update.

link|improve this answer
10  
Ouch. Another option would be 'BEGIN TRANSACTION'; it's nice to know you can roll back. :) – Murali Suriar May 6 '09 at 9:35
5  
Combination of the two, select count(*) to see how many I think I'll update, and then update in transaction to do the update. If the counts match commit, if not rollback and figure out why. Of course, I don't do that for "simple" updates, and those are always the ones that screw you. :-) – WaldenL May 6 '09 at 12:47
7  
Be careful with BEGIN TRANSACTION on a live system (or a test system being used by others too) - make sure you COMMIT or ROLLBACK soon or you may end up deadlocking other processes waiting for locks on resources your transaction holds locks on. – David Spillett Jun 7 '09 at 20:13
show 3 more comments
feedback

Biggest mistake? Thinking I had set two variables when I had not. So rm -rf $VARIABLE/$VARIABLE2 became rm -rf /. FreeBSD has recently updated their rm tool so that rm -rf / is not possible anymore precisely because of this mistake!

link|improve this answer
4  
I would like to point out, re-reading my sentence it sounds like I was the reason for changing the way rm worked, this is not the case. Apparently people were using rm in scripts and the same thing happened and they wanted to add a fail-safe. – X-Istence May 6 '09 at 7:42
3  
i withdraw my vote then :) – Colin Pickard May 19 '09 at 20:33
show 2 more comments
feedback
shutdown -h now

meant for the local workstation, but typed it while logged via ssh on production server. Since then I always have hostname in my $PS1.

link|improve this answer
4  
Check out molly-guard (packages.debian.org/search?keywords=molly-guard) – David Holm May 29 '09 at 20:14
2  
I used to put the hostname in the prompt, but now I have patched the shutdown (and related - halt et al) commands to prompt me to enter the current machines name before it will execute the command. That gives me the extra step necessary to realise that I just shutdown the wrong server. Problem is, once I implemented that patch, I never did it again - but its saved a few of my colleagues tho :) – Moo Jun 11 '09 at 15:33
show 5 more comments
feedback

On a VMS system, I had been using the ASSIGN DCL command to assign logical names, and I wanted to RECALL a previous ASSIGN command line. Now, in VMS, you only typed as many characters of a command to make it unambiguous. So I intended to type

REC ASS

but I accidentally typed

REQ ASS

instead. REQ was sufficiently unambigous for the REQUEST command, which broadcasts the argument to everyone with operator privs (which was everyone in IT). So the entire department received my broadcast message which was simply "ASS".

link|improve this answer
show 2 more comments
feedback

On a Solaris system: "killall dataLoader".

'dataLoader' was an app I was working on. On Linux, killall works like pkill. It sends a signal to processes that match a string given as the argument. On Solaris, killall trys to kill everything on the system the current user can kill. I was root.

link|improve this answer
1  
I've done exactly the same two weeks ago in a Power 5 AIX 6 system... Ok, not exactly, I did a "killall -9 java" :) – Andor Jul 22 '10 at 19:44
2  
@Andor: Well, you killed java for sure. ;) – Bobby Mar 17 '11 at 13:54
show 1 more comment
feedback

Omitting the -r from a shutdown command. On a remote server. On the other side of the country. With no IT staff in the remote office.

We've all done it, it's almost like a rite of passage at this stage.

link|improve this answer
show 1 more comment
feedback

Trying to change ownership of everything in a directory, including dot files, with:

chown -R user * .*

Guess what that does?

link|improve this answer
9  
read mail -really fast ??? – superwiren May 29 '09 at 14:16
5  
Been there, this taught me the .[^.]* pattern really well – Maciej Pasternacki May 31 '09 at 18:52
1  
@chaos: The command seems to remove everything in your current directory, and even dot-files – Masi Jun 7 '09 at 20:20
2  
@chaos: It does not remove .. and never has and never will. You are simply mistaken. Please cite a single unix implementation that does have this behavior (which I have never seen and which is documented to not be the behavior of even super-antique unixes like 7th edition unix). – chris Jun 25 '09 at 15:44
1  
Would ./.* prevent this, or would it still propagate back up the directory tree via ./../? – tj111 Jun 25 '09 at 17:41
show 10 more comments
feedback

Once, many many moons ago, I needed to find a particular executable but couldn't remember the complete name of it (but could remember a few of the letters). So I figured I'd check the /usr/bin directory with something like this

rm /usr/bin/i*g*

Strange. Nothing returned. Figuring I just had misremembered the second letter, I tried again with

rm /usr/bin/i*

Again, nothing. After doing the same with /usr/local/bin, /usr/sbin, and anything else I figured it might be in, I realized I'd been mistyping the 'ls' command.

Don't quite know where the brainfart came from, but it's definitely not a mistake I ever made again.

link|improve this answer
4  
Even on my server it does not return anything. – Mircea Vutcovici Mar 17 '11 at 14:06
show 1 more comment
feedback
setup.exe

It was Windows Vista.

link|improve this answer
feedback

Meant to destroy /dev/sdb, fortunately I had a good up-to-date backup

dd if=/dev/urandom of=/dev/sda
link|improve this answer
show 1 more comment
feedback
select * from <File1> join <file2>

On a production box. Note the lack of an on clause. :-) Both tables were multi-million row tables, and this was on an AS/400 in the mid 90s where once the SQL was running you couldn't kill it.

link|improve this answer
feedback

On one of our data production server, one of my roots typed:

chmod -R 777 /

Because he was getting permissions error with some scripts...

Shortly after that, his private key was removed from all servers and he took care of the 1TB data restore on the data production server...

link|improve this answer
feedback

My erm favorite was when I was at university. I was building an application (I don't recall what) and as I wasn't root I had built it with

PREFIX=~username/usr/local

So I could install it to my home directory. Unfortunately it installed into

/home/username/src/app/~username/usr/local

instead. Naturally to delete it as start again i executed

rm -rf ~username

In the source directory.

I wondered why it was taking so long....

:-)

However my worst one was when I was working with a solaris workstation and after getting it all setup we wanted to wipe the configuration ready for live config. So I executed

sys-unconfig

Agreed to the warning message and instead of the machine rebooting and going back to "factory defaults" The xterm window simply said

connection closed by foreign host.

Moral of the story Don't ever leave a root shell open on another host! EVER!!

link|improve this answer
show 1 more comment
feedback

I once wanted to delete a bunch of files in a directory.

del *.*

The computer then said "Are you sure? [y/N]" I thought "OF COURSE I'm sure, I wouldn't have typed the darn command in otherwise! Sheesh stupid computers grumble..."

Y <enter>

C:\Windows>_

Um... WTF? Did I just erase my windows directory?....

undelete *.*

In those days of small hard disks I knew what every file in c:\windows was for and what its name was, but even after undeleting everything the system was never the same. I gained a little bit of respect for the "are you sure" prompt. Just a little.

link|improve this answer
show 1 more comment
feedback

Many years ago I was at home coding up some stuff in php while working on a project with a buddy of mine who was the maintainer of the project. We were IM'ing one another in a collaboration effort. We always banter back and forth in play.

I was trying to get ssh-agent to work properly on my machine while we were having a religious war of Perl vs PHP. Then I mentioned something about ssh-agent needing to be eval'ed (not sure why I said that). So then he sent me this message in an effort, so I thought, to help me with my problem (bear in mind that I was su -'ed to root):

\# eval $(echo ssh-agent | 
perl -pe 's/h-a/m -r/' | 
perl -pe 's/^ss/r/' | 
perl -pe 's/gent/f \//')

WARNING! DO NOT RUN THAT COMMAND!!!

IF you remove the eval and run the inner command by itself it is:

rm -rf /

It took me all of 4 seconds to notice what was happening but the damage was already done. I had to reinstall my OS. Fortunately, nothing of my work was wiped out except some stuff in /etc iirc. He had a HUGE laugh when I sent him a message of horror asking why he did that. We are both long-term systems engineers. He didn't think I would run it and would be more careful to check it before simply c&p'ing and I just trusted him so I didn't even consider he was playing around. Needless to say, this little story comes up all the time between us. So, I decided to immortalize it.

How have I mitigated this from happening again? I trust no-one!

Another less interesting story is a couple of years back I was doing some work on a mission critical box at work. I had a few terms open to different machine. I needed to remove some superfluous stuff in a directory. Well, I got lost in my terms and accidentally rm'ed . in my local dir but on the wrong host (wrong term)!!. I executed the command in /var/lib/mysql instead of /tmp on the application server (different term). Needless to say, I wiped out the production database. Fortunately, we had a warm standby that we flipped to while me and a co-worker rebuilt the primary from backups and the standby. That took about 18 hours to do.

Mitigation: more careful about what windows I execute commands in before executing them.

link|improve this answer
show 1 more comment
feedback

First of two...

On a Solaris box we had a tar backup of an AIX machine..

One of the Developers typed:

tax xvf AIX_Backup.tar

Of course the paths in the tax were absolute and we ended up making a new distro of unix... Solarix... The only problem with the distro is that it did not boot :(

link|improve this answer
feedback

the short version

#/bin/bash
$0&
$0
link|improve this answer
3  
what does that do? – Nathan DeWitt May 15 '09 at 2:33
5  
It calls it's self in the background than then without waiting for that to finish, calls it's self again: a.k.a. Fork Bomb – BCS May 18 '09 at 16:50
2  
@Masi: Not if the computer is fast enough and the script long enough. Specialy if you don't know it is going to happen. – voyager Jun 7 '09 at 22:53
2  
Drop the #!/bin/bash, and I think this would win a code golf for shortest forkbomb in bash. – Joey Adams Apr 16 '10 at 16:42
show 4 more comments
feedback

I think the most stupid thing I ever did was to remove the default route on out external facing firewall cluster - whilst vpn'ed to my desktop over a hundred miles away.

Fortunately it was in a period designated as planned downtime (just in case), but that did not save me the 200 mile round trip to go and reconfigure the firewall onsite. It also did not help that this took out our internet exposed production systems for the duration of my travel and subsequent fix.

We all know the definition of a nano-second. An ohno-second is even smaller, and is the time between hitting 'enter' and realising your mistake.

link|improve this answer
feedback

Story told to me:

Another branch called because their local PBX had some issues. After some investigation, we learned that they updated their server, but not their Asterisk configuration. So, the admin decided to instruct the branch guy to redo the configuration.

Admin: "Ok, now, type rm -rf /etc/asterisk"

Guy: "Ok."

Admin: "Now, type cp /var/..."

Guy: "Wait, it is still running..."

Admin: ???... !!!

link|improve this answer
show 1 more comment
feedback
rm -rf / some/path

instead of

rm -rf /some/path

Luckily it didn't happened to me ;-)

link|improve this answer
show 4 more comments
feedback

XCOPY is a powerful beast - merciless in it's execution and retarded by the fact that its command line args go in reverse from Window's COPY and UNIX's cp.

A couple days ago I accidentally wrote:

xcopy src \path\to\a\new\nonexistent\directory

XCOPY was kind enough to overwrite my src directory with... nothing! And it didn't bother to put the old files in the Recycle bin either.

Oh, and it turns out that XCOPY actually overwrites the same sectors on the disk instead of allocating new ones. I've tried 3 disk recovery programs, and the best one could only recover 3 of the 10 files lost. Of course those 3 files were only vshost.exe and its pals. Swell!

link|improve this answer
feedback

had trouble with network (on a remote machine) and just wanted to restart the interface

ifconfig eth0 down && ifconfig eth0 upp

Nowadays, I make sure someone is near the machine before trying things like this (iptables is a good candidate as well). And when nobody was there, I once typed

sleep 600; reboot

into another (screen) terminal, so that it would reboot if I could not ctrl+c the command within 10 minutes.

I learned from that mistake as well (ctrl+c the sleep will run the reboot) and now I use

sleep 600 && reboot

which will enable me to ctrl+c it.

link|improve this answer
1  
shutdown -r +5 will reboot system in 5 minutes, unless you kill shutdown process with killall shutdown for example. – gelraen Mar 17 '11 at 14:03
show 1 more comment
feedback
source ~/.bash_history

My intention was source .bashrc, but I was too hasty with tab-complete...

link|improve this answer
show 1 more comment
feedback

Something along the lines of this:

sudo dd if=/dev/zero of=/dev/sda

I meant sda4. I wiped the entire disk, not just the partition :-(

link|improve this answer
show 1 more comment
feedback
ifconfig eth0 down

Oops, I'm on the external side of eth0. The webserver is on the other side of the world, in a locked room. With no network access to login or reboot. Crap.

link|improve this answer
feedback

A free cookie to anyone who can tell me why I was an idiot for trying to remove all hidden files and directories thusly:

rm -rf .*

link|improve this answer
1  
Does your command remove subdirectories below your current directory too? – Masi Jun 7 '09 at 20:37
show 2 more comments
feedback

While cleaning up my home folder on a production webserver, I forgot that I had symlinked the server's web root to a place in my home folder. Not thinking, I ran an rm -rf on that folder and next thing I know, people are calling that the web site is down!

OOPS!

link|improve this answer
feedback

I was once comparing data in two folders and ran rsync with the -d option (delete files on destination that are not on the source). And then I switched around source and destination when I ran rsync. That deleted all the new files, that I wanted to backup. Now I learned running rsync with -n (dry-run).

rsync -trvd --stats --progress /destination /source

I had no backup.

link|improve this answer
feedback

Once upon a time (probably System III, but it was a long time ago), it was possible to create a file named * by using the right shell quoting. When I found one in my home directory, I had typed rm * and had my finger on the return key when something made me hesitate and think about it...

Creating such a file for other users was a common prank.

If the file is sitting there in a directory, that is a tough one to mitigate. The reflex to just type its name exactly as ls just displayed it is pretty strong.

The other (less harmful) prank was to name files with trailing white space (or only white space) which were much harder to remove...

link|improve this answer
1  
That's what GUIs are for :) – Zifre May 6 '09 at 21:11
1  
shell completion is a real time-saver for those trailing-space files :) – Mr. Shiny and New 安宇 May 14 '09 at 12:58
1  
This would have been ca. 1985, and accessed via dialup or via a RS-232 cable from a clone of a VT-100 terminal. Not a terminal emulator, but an actual terminal. I don't recall csh having name completion at the time... ;-) Today, of course, GUIs and name completion make both of those less risky, and lots of OSs make it a lot harder to create such files without resorting to disk sector editing.... – RBerteig May 15 '09 at 20:18
show 2 more comments
feedback

Due to some bad experience with general JBoss flakiness after a restart, I like to clear JBoss' working files before a restart. I would normally do:

# cd /var/cache/jboss
# rm -rf tmp/* work/*

In order to protect myself from typing any of the many possible disastrous mistakes, like:

  • /tmp/*
  • tmp /*
  • you get the idea

I make the last command:

# sudo -u jboss rm -rf tmp/* work/*

Since the JBoss user would find it difficult to remove any critical files that don't belong to it.

I never actually made that mistake, but I'm safe in the case that I do.

link|improve this answer
feedback
1 2 3

protected by Chris S Mar 17 '11 at 12:23

This question is protected to prevent "thanks!", "me too!", or spam answers by new users. To answer it, you must have earned at least 10 reputation on this site.

Not the answer you're looking for? Browse other questions tagged or ask your own question.