Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here.

More info: http://serverfault.com/faq


I once accidentally dropped the indexes on a production sql server system all because the test system needed a domain admin to login. Took all monday morning to rebuild them.

share

locked by HopelessN00b Dec 5 at 11:44

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by John Gardeniers, Mark Henderson Mar 28 '11 at 4:29

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

1  
Shouldn't this be a community wiki? –  notandy May 27 '09 at 15:47
    
Yes it should. Reputation-generating polls are quite popular. –  Mihai Limbăşan May 27 '09 at 16:12
    
I've changed it into a community wiki. –  mrdenny May 27 '09 at 16:32
    
Thank you, mrdenny. Upvoted post. –  Mihai Limbăşan May 27 '09 at 17:50

21 Answers 21

chown randomuser /* -R

instead of

chown randomuser ./* -R
share
    
Ouch. Could you recover from that? –  Josh Feb 26 '10 at 14:58
    
Didn't. Couldn't. Had to reinstall :( –  Stephen Melrose Mar 4 '10 at 15:50
    
At least you probably didn't lose any data. –  Demetri Oct 28 '13 at 0:33
kill 1

instead of

kill %1

as root. init and all its children die...

This was a linux box that controlled a 150 or so metre long sorting machine that sorted 200kg wool bales.

I recovered by running out and hitting the big red emergency stop on the operators console before the machine ripped itself to bits.

Sometimes there's only so far you can recover. ;-)

share
    
On my machine kill 1 does nothing, even as root. kill -6 1 on the other hand... –  Demetri Oct 28 '13 at 0:34
    
It actually happened in the late 90s. There's two relevant implications: 1) A lot has happened since then, such as a complete re-implementation of the unix startup, e.g. upstart - are you running init on your system? 2) I may not have all the detail in the above post. I'll cheerfully concede there may have been another option such as a SIGKILL. –  Jason Tan Nov 14 '13 at 9:32
    
init (and only init) can (and on my machine, does) ignore SIGKILL. It does die (causing a kernel panic) on receiving SIGABRT. –  Demetri Jun 16 at 2:17

I accidentally deleted the "root" user, while logged in as root. The Red Hat box ran for another six months, doing qmail quietly in the corner, until the PSU died. I still blush when I think about it.

share

2 Things:

Running a full screen SSH session on one virtual terminal, doing other tasks on another virtual terminal. Didn't realise which VT I was on, ran FDISK, wiped all the partitions on HDB, then rebooted. Blew away all the userdata on that server.. (Managed to get it all back, as the exact geometry of the partitions was still displayed on the virtual terminal when the server in question came back up. Recreated the partitions without formatting, all data was stil there)

Trying to clean up a failed attempt at compiling a kernel, blew away all the module binaries for the currently running kernel. Only realised after I rebooted the server, which happened to be in a datacenter in California somewhere. Only problem is I'm in Canberra, Australia..

share
    
wow that is seriously lucky. –  Preet Sangha May 27 '09 at 13:00

rm rf / instead of rm -rf * on a mail server. I recovered by getting a new job and learning to sudo /bin/bash only when necessary.

share

Not necessarily logged in but...

In the early 90's, I worked for a highly respected university in Michigan and we used walkie-talkies for communications within the IT department. One day I walked into the server room, keyed the walkie-talkie, and boom, all 20 of our servers simultaneously rebooted.

Needless to say, the radios were banned from entering the server room again.

share

Setup a new user account and accidentally set the root password instead of the user's password. User ended up never needing the account, password was lost, no one logged into the box as root for months. Got to do a password recovery on it months later when I needed to get in and make a change requiring root access. Was painful since it was a production box, so taking it down for even just the few minutes to recover the root password was bad.

share

Me, I'm just a technical author, so I struggle to know what you lot are talking about sometimes and am never allowed close enough to any real kit to do anything remotely dangerous. However, in a previous job (very previous - read on) I recall a certain programmer here on the Cambridge Science Park called Verka S (who's catchphrase was 'good code does not need annotation') once calling over the sysadmin to ask if he could figure out why her win 3.11 box was no longer working (told you it was very previous). After a brief examination of the dead object he asked 'what did you do?' and she replied 'I was trying to free up some hard disk space, so I deleted the largest file i could find'. 'Can you remember what it was called' asked the sysadmin: 'yes' she goes: 'win.exe'. You'd be right to guess that her code most certainly needed fullsome annotation.

share

I ran some Perl someone in an IRC channel gave me, which did something tricky to exec "rm -rf /". Whups.

share
    
Oh, those helpful folks in IRC! Where would we ever be without them? ;-) –  Josh Feb 26 '10 at 14:59

after a long night, followed by a lack of caffeine that morning, i accidentally ran

rm -R web/*

instead of

rm -R _web/*

and deleted an entire clients site rather than a backup folder. It was at this point I realised the massive flaws in our backup strategies. The offsite copy was out of date, the backups on the server were stored in an 8GB file we couldn't download to access and because of the spaces in file names, we couldn't restore a whole directory and instead had to specify each filename in double quotes manually. took me a couple of hours to write the script just to automate the process of restoring files one by one.

share

Deleted a disk image file of a vmware guest VM, I didn't have the backup of.

Luckily the guest was still running so I've managed to back it up and restore in the new VM. It's still running, two years later.

But the worst thing happened not when I was logged in. I was messing around, moving things in the rack and suddenly 500GB external disk containg the archives for last year for 3 customers fell on the floor. All gone. And it was double-disk, RAID1. I have learned the hard way RAID1 wasn't a catch-all solution for disk failure ;)

share
    
how did you handle the data loss? –  Demetri Jun 16 at 2:19

I accidentally shut down the wrong computer. It was a ssh connection to the proxy server at work, I wanted to shut down my laptop but instead I put 50 people without internet. I recovered by running to the serverroom and booting the server again.

share

Someone I worked with removed the password file then logged out of a critical production sun server. I had to repair it as we could not just reinstall it.

Another admin create an account with the same uid and home as the main production database and then removed the copy account. It deleted the databases user and home - this was on a non stop Tandem, there was some egg on the admin face as he made it stop.

I have removed the resource forks from Apple files on a Sun server as I did not know why the temp files where taking up so much space. The Mac users where not impressed ;)

share

Ran a script that deletes all files on a file server recursively.

We had a partial backup of files, so we never fully recovered..

share
    
You do realize when people say that rm -rf / fixes all problems on your server, they're joking, right? ;-) –  Josh Feb 26 '10 at 14:58

Went to init 1 to restart some services I had patched, forgetting that I was SSH'ed in and the server was a 3-hour drive away. Luckily we had a service call to that location the next day, and the server in question was a backup, not in production.

share

Mistakenly restarted all (20) of the production servers while rebooting our workstations after patches. The server names were in the text file my script was feeding from.

It killed uptime, which wasn't that big a deal in that organization.. but Exchange didn't come back up on its own -- which was how it was noticed the next morning. (Not technically as 'root', but Domain Admin counts, right? :) )

share

sudo rm -Rf /*

share

Converted a hard drive to a dynamic disk in Windows 2003 Server. It was the right thing to do, except on the wrong hard drive. Took some time to get the data back, also because I had no clue what to do at first.

share

On my home server, I chained too many important commands together with && when resizing a partition. I believe it was (excluding arguments) fsck && resize2fs, followed by an lvreduce on the next line. fsck didn't return the exit code I assumed it had, so resize2fs never ran, and the lvreduce totally hosed a bunch of my data when it truncated the volume.

I am much more careful with && these days.

share
    
Sounds like not enough '&&', instead of too much. Would '&& lvreduce' have worked? –  romandas Jul 9 '09 at 20:30

I've accidentally deleted data that I shouldn't have deleted. I've also accidentally shut down the server. :-\

share

Not the answer you're looking for? Browse other questions tagged or ask your own question.