Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What is the most damage (of whatever kind) that you have ever caused with a single mistaken/mistyped/misguided command line? I deleted a production system database by mistake a while back, for example, but I was lucky (i.e. backed-up) and there was no permanent data loss, lost money, property damage etc.

Most importantly (for votes), what do you do to make sure it will not ever happen again?

share

locked by Michael Hampton Oct 23 '13 at 15:06

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by Mark Henderson Mar 28 '11 at 4:31

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

9  
Since this is a poll, it should be CW –  Eddie May 6 '09 at 2:00
4  
How should they know it's a poll progmatically? –  Mikeage May 6 '09 at 8:14
1  
If the mechanism is the same as SO's, even if you can edit a question, you can't change it to CW. Only the OP or a moderator can do that. –  chaos May 6 '09 at 14:09
9  
Good idea - post your most destructive mistake publicly on the internet for all future employers to see! :) –  Sam Schutte May 13 '09 at 19:29

81 Answers 81

up vote 46 down vote accepted

In SQL server, on a production system:

update customer set password = '' <enter>

The most recent backup was like a week old.

To mitigate this, I now usually write a select statement first to make sure I've got the where clause correct, then go back and edit it to insert the set clause and change the statement to update.

share
11  
Ouch. Another option would be 'BEGIN TRANSACTION'; it's nice to know you can roll back. :) –  Murali Suriar May 6 '09 at 9:35
5  
Combination of the two, select count(*) to see how many I think I'll update, and then update in transaction to do the update. If the counts match commit, if not rollback and figure out why. Of course, I don't do that for "simple" updates, and those are always the ones that screw you. :-) –  WaldenL May 6 '09 at 12:47
8  
Be careful with BEGIN TRANSACTION on a live system (or a test system being used by others too) - make sure you COMMIT or ROLLBACK soon or you may end up deadlocking other processes waiting for locks on resources your transaction holds locks on. –  David Spillett Jun 7 '09 at 20:13

Back in dark ages of DOS there was a utility called, as far as I remember, killtree, that deleted a directory tree without any confirmation.

I then had a keyboard that had the \ under the Return.

Under Norton Commander I wanted to delete a directory tree in c:\ I had the selection over, so I type:

killtree Right Ctrl + Return

Ctrl Return combination was used to insert file/dir name under the selection into the command line.

The problem is, instead of Right Ctrl, my finger went to the \...

share
1  
"With great power comes a really large hole in your foot." –  MadHatter Oct 23 '13 at 12:36

I was doing a log parse for a mail server using grep, so I typed:

grep -r event_string /var/log > output

Unfortunately, I was in /var/log when I typed it, so every time it matched the string it wrote the string to a file that it was in the directory it was checking...When it got to that file, the file became a space devouring beast that could not be fed. Filled up the whole machine, and, since it was a mail server, caused a pretty pressing problem.

share

The worst I've seen:

In /var/www// Instead of removing the folder with [rm -r var]

He removed the complete /var folder with [rm -r /var] One slash can do a lot :/

So yeh, that guy had fun trying to get backups... what only made it worse because shutting down the server and doing a recovery would just get everything back upto the last little update, the backups were only done once a week

share

Removed my whole home directory when trying to clean a floppy disk with the command:

rm -rf * /mnt/floppy
share

Intending to clean up the build directory, I removed all the source code for a project with:

rm * .o

Since then, first thing in my projects is a clean target in the Makefile

share

I've done a rm -rf * .o but the best I've seen is:

userdel -r root

It was done by a novice though and he was learning.

share

When trying to clean my home folder of temp/backup files:

Run this:

rm -rf *~

# OOPS! DON'T RUN THESE:
rm -rf *  # OOPS! Deleted all files in $PWD
rm -rf ~  # OOPS! Deleted all files in $HOME
rm -rf ~* # OOPS! Deleted all users' home folders!!!
share

Ubuntu 10.10, I had only 3gb left on my partition so clearing up files.

I thought, ah well, might as well remove old kernels (GRUB screen was becoming too big too).

sudo apt-get remove --purge 2.6.28*

The asterisk was a little bit... greedy (though I still don't know how it managed to match 2.6.29 and .30).

Epic freakout, didn't dare to ctrl+c (what's worse then deinstalled kernels? that's right, semi-deinstalled kernels). Let it run and reinstalled latest kernel. I even opened an edited document in gedit so it would block any restart (I know, I know...).

share

We had a FTP script that would update directories which housed our Client's reports. The script looked like this...

ftp www.ourcompany.com
cd <ClientsDirectory>/
put *.html ( including index.html ) 

So just what happens when <ClientsDirectory> does not exist?

You get the client's index page as your companys home page... Opps

share

Easy easy... hitting ENTER when my screen has been scrolled up in order to return it to the command line. I've ran the wrong command a few times doing this. I've gotten in the habit of hitting CTRL-C instead.

share

rm -f /bin

instead of

rm -f ./bin

as root (of course) on an old FreeBSD system. Luckily all the commands in /usr/local/bin and /usr/bin sufficed to copy binaries from the install CD /bin and unhose the system.

share
source ~/.bash_history

My intention was source .bashrc, but I was too hasty with tab-complete...

share

Good stuff here. I've done a LOT Of them. To avoid the missing WHERE clause in SQL, I now use: alias mysql='mysql --i-am-a-dummy'

An interesting one was while updating a remote server: mv /home /home_old && mv /home_new /home mv /var /var_old && mv /var_new /var mv /lib /lib_old && mv /lib_new /lib

It seems that Linux stops running (or booting) when /usr/lib dissappears. Remote servers now have either an IMPI card, or a serial console lnk to another server, plus a boot CD in the drive.

share

Most annoying mistake I remember doing was a some years back when hardware where expensive and we often used software routers/firewalls rather than hardware ones. One night we got hit by some pretty bad DDoS. Trying to debug this/figure out what I could do, I logged onto the main router and started doing tcpdumps, however the system were so insanely unresponsive that I figured Id better just drop the firewall rules so that I can hopefully get some sorely needed cputime. So I typed;

ipchains -F

Problem is, default policy of all chains were drop. Had to call up my boss in the middle of the night, take a cab over and fetch a keycard then get onsite and manually reboot the router.

share

I had my home directory mounted into a chroot in loop mode to do some work. The chroot was taking too much space, so I had to remove it, and I just did

rm -rf /path/to/the/chroot

It too a bit longer than I thought before I realized it was removing the documents from my loop-mounted homedir altogether...

share

I'll start with the standard disclaimer: It was many years ago, I had just started, I was a noob, honest!

I was trying to debug something on a Cisco router (don't even remember what it was), but I just couldn't figure out what exactly I needed. If you've ever tried the debug command in IOS, you'll know what I mean. Every command has a ton of subcommands, arguments, options, etc.

Finally, I came up with a brilliant idea! I'll just "debug all" and then strip out the useless stuff. (I can hear the groans from the other people who've tried this).

Long story short: it killed the router. It just happened to be the core router for the ISP I was working for. 30,000 people suddenly had no internet access.

Yes, it was dumb. Yes, the ISP should not have been letting newbies run around on core routers unsupervised. Yes, we should have had hot standby. Etc, etc. :)

share

I wanted to remove all "~" backup files (passwd~, group~, resolv.conf~) and on a Swedish keyboard you need to press the ~ button and a space to write a "~" and what I did was:

$ rm  * ~

In /etc

share

Trying to change ownership of everything in a directory, including dot files, with:

chown -R user * .*

Guess what that does?

share
10  
read mail -really fast ??? –  superwiren May 29 '09 at 14:16
5  
Been there, this taught me the .[^.]* pattern really well –  Maciej Pasternacki May 31 '09 at 18:52
1  
@chaos: The command seems to remove everything in your current directory, and even dot-files –  Masi Jun 7 '09 at 20:20
2  
@chaos: It does not remove .. and never has and never will. You are simply mistaken. Please cite a single unix implementation that does have this behavior (which I have never seen and which is documented to not be the behavior of even super-antique unixes like 7th edition unix). –  chris Jun 25 '09 at 15:44
1  
Would ./.* prevent this, or would it still propagate back up the directory tree via ./../? –  tj111 Jun 25 '09 at 17:41

A lot of people have mentioned rm -rf. I'd like to point out that it isn't the trickiest command. The one you really have to worry about is sudo rm -rf. After deleting my entire Python site-packages directory, I now refer to that command as the "nuclear option".

share
net send /domain "Hello World"

Oh sh!+, this isn't the lab domain.....

share
1  
Hah. Once upon a time, in a poorly configured college network, a friend of mine found out that, if you logged into a netware machine using the machine login that was bound to a mac, it didn't set your permissions correctly, and you could do university wide broadcasts...He set up a program that would do one every hour or so, but screwed up his count zero, so it did one after an hour, and then another 30 minutes later, and then 15, then 7...I wandered in to work to find the student techs trying to find out how a mac that they'd ripped out of the wall was sending constant broadcasts. Good times. –  Satanicpuppy Mar 17 '11 at 14:57

I used to be a linux lab admin, and one of the students was trying to get her Joomla install to work. She needed to add execute permissions to one of the files. Not knowing much, she was going to do:

chmod -R 777 /path/to/joomla/

Except what she did was:

chmod -R 777 /path/to/joomla /

Turns out that her complete filesystem(/etc/passwd, /etc/shadow...) was 777.

When you don't specify all the modes, it presumes you want to remove the permissions that would go with them. So everything was now mode 0777(i.e. no sticky, set user/groupID). There's a lot of important stuff that's set-uid, like su.

share

I once copy/pasted some "bash magic" script which I didn't understand from a slashdot comment, which had no warning and no comment other than that one liner. That's just stupid, I know.

It turned out that evil lines spawned bash processes in a loop which rendered the server completely unresponsive. Luckily it was based on XEN and I was able to reboot it using xm reboot (calling data center to do manual reboot is always messy). On the other hand, I didn't have access to dom0 instantly.

Lesson learned - don't copy/paste stuff you don't know :)

share

First of two...

On a Solaris box we had a tar backup of an AIX machine..

One of the Developers typed:

tax xvf AIX_Backup.tar

Of course the paths in the tax were absolute and we ended up making a new distro of unix... Solarix... The only problem with the distro is that it did not boot :(

share

Needed to extract a file from a tar file. No problem:

tar cvf filename.tar somefile

Just reinstalled SunOS4, restored local files from backup, and was cleaning up, removing the restored files from their temporary location when I typed rm -rf /usr and wondered why it was taking so long... I got to do it all over again.

share
ifconfig eth0 down

Oops, I'm on the external side of eth0. The webserver is on the other side of the world, in a locked room. With no network access to login or reboot. Crap.

share

as root:

find / -type d -exec chmod 755 {} \;
find / -type f -exec chmod 644 {} \;

I ended up rebuilding the box.

... true story.

share

A friend of mine works on a classifieds site running on Windows / IIS. The site had lots of image files stored on their web server's disks that were no longer referenced from their database so he decided to write a quick cleanup script. He wrote a small program to generate the script, something like this:

List<string> imagesInFileSystem = Directory.GetFiles(imageDirectory);

foreach (string file in databaseImages) {
    if (!imagesInFileSystem.Contains(file)) {
        Console.Out.WriteLine("del " + file);
    }
}

And it generate the following output:

del c:\images\image1.jpg
del c:\images\image2.jpg
...
etc

Which looked pretty good. However when he ran the program most of the images for the site were deleted! He forgot one thing: the server's filesystem is case insensitive but his check against the database wasn't.

The sales guys had to spend the next couple of days calling up people to get them to upload the files again.

share

A colleague decided to be funny and did:

net send  /DOMAIN *my_username* Hey How You Going!

Unfortunately, instead of ME getting the message, the whole floor gets a popup window saying:

*my_username* Hey How You Going

Luckily messenger stuff is blocked at the floor router, although a manager did run out to abuse me for "wasting company resources!"

share

I was working for an Internet marketing company as the systems / database administrator. I had given a fellow employee (now good friend) read/write access to a web advertising system we were building - a sort of DoubleClick.net platform style thing but just for us, so we could manage all our ads across various networks. We were writing a new reporting module to gather data on something or other and he was playing around and did the classic SQL newbie mistake:

update advertisements set url='something'

Without the where clause. So that afternoon, when the boss came in and asked "why are all of our banner ads pointing at the Tragically Hip??" I really had no idea... the Hip being a popular Canadian band, my colleague had basically sent tens of thousands of dollars worth of ad space that we had purchased on behalf of our clients to the website of his favorite band on accident. We were both able to keep our jobs, though groveling was involved.

I got really good with Netsaint (the predecessor of Nagios) after that, and learned not to give SQL newbies access to production.

share

protected by Chris S Mar 17 '11 at 12:23

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.