3
votes

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here.

More info: https://serverfault.com/faq


I once accidentally dropped the indexes on a production sql server system all because the test system needed a domain admin to login. Took all monday morning to rebuild them.

3
  • 1
    Shouldn't this be a community wiki?
    – notandy
    May 27, 2009 at 15:47
  • Yes it should. Reputation-generating polls are quite popular. May 27, 2009 at 16:12
  • I've changed it into a community wiki.
    – mrdenny
    May 27, 2009 at 16:32

21 Answers 21

6
votes
chown randomuser /* -R

instead of

chown randomuser ./* -R
3
  • Ouch. Could you recover from that?
    – Josh
    Feb 26, 2010 at 14:58
  • Didn't. Couldn't. Had to reinstall :( Mar 4, 2010 at 15:50
  • At least you probably didn't lose any data.
    – Demi
    Oct 28, 2013 at 0:33
5
votes
kill 1

instead of

kill %1

as root. init and all its children die...

This was a linux box that controlled a 150 or so metre long sorting machine that sorted 200kg wool bales.

I recovered by running out and hitting the big red emergency stop on the operators console before the machine ripped itself to bits.

Sometimes there's only so far you can recover. ;-)

3
  • On my machine kill 1 does nothing, even as root. kill -6 1 on the other hand...
    – Demi
    Oct 28, 2013 at 0:34
  • It actually happened in the late 90s. There's two relevant implications: 1) A lot has happened since then, such as a complete re-implementation of the unix startup, e.g. upstart - are you running init on your system? 2) I may not have all the detail in the above post. I'll cheerfully concede there may have been another option such as a SIGKILL.
    – Jason Tan
    Nov 14, 2013 at 9:32
  • init (and only init) can (and on my machine, does) ignore SIGKILL. It does die (causing a kernel panic) on receiving SIGABRT.
    – Demi
    Jun 16, 2014 at 2:17
4
votes

I accidentally deleted the "root" user, while logged in as root. The Red Hat box ran for another six months, doing qmail quietly in the corner, until the PSU died. I still blush when I think about it.

2
votes

2 Things:

Running a full screen SSH session on one virtual terminal, doing other tasks on another virtual terminal. Didn't realise which VT I was on, ran FDISK, wiped all the partitions on HDB, then rebooted. Blew away all the userdata on that server.. (Managed to get it all back, as the exact geometry of the partitions was still displayed on the virtual terminal when the server in question came back up. Recreated the partitions without formatting, all data was stil there)

Trying to clean up a failed attempt at compiling a kernel, blew away all the module binaries for the currently running kernel. Only realised after I rebooted the server, which happened to be in a datacenter in California somewhere. Only problem is I'm in Canberra, Australia..

1
  • wow that is seriously lucky. May 27, 2009 at 13:00
2
votes

rm rf / instead of rm -rf * on a mail server. I recovered by getting a new job and learning to sudo /bin/bash only when necessary.

2
votes

Not necessarily logged in but...

In the early 90's, I worked for a highly respected university in Michigan and we used walkie-talkies for communications within the IT department. One day I walked into the server room, keyed the walkie-talkie, and boom, all 20 of our servers simultaneously rebooted.

Needless to say, the radios were banned from entering the server room again.

1
vote

Setup a new user account and accidentally set the root password instead of the user's password. User ended up never needing the account, password was lost, no one logged into the box as root for months. Got to do a password recovery on it months later when I needed to get in and make a change requiring root access. Was painful since it was a production box, so taking it down for even just the few minutes to recover the root password was bad.

1
vote

Me, I'm just a technical author, so I struggle to know what you lot are talking about sometimes and am never allowed close enough to any real kit to do anything remotely dangerous. However, in a previous job (very previous - read on) I recall a certain programmer here on the Cambridge Science Park called Verka S (who's catchphrase was 'good code does not need annotation') once calling over the sysadmin to ask if he could figure out why her win 3.11 box was no longer working (told you it was very previous). After a brief examination of the dead object he asked 'what did you do?' and she replied 'I was trying to free up some hard disk space, so I deleted the largest file i could find'. 'Can you remember what it was called' asked the sysadmin: 'yes' she goes: 'win.exe'. You'd be right to guess that her code most certainly needed fullsome annotation.

1
vote

I ran some Perl someone in an IRC channel gave me, which did something tricky to exec "rm -rf /". Whups.

1
  • Oh, those helpful folks in IRC! Where would we ever be without them? ;-)
    – Josh
    Feb 26, 2010 at 14:59
1
vote

after a long night, followed by a lack of caffeine that morning, i accidentally ran

rm -R web/*

instead of

rm -R _web/*

and deleted an entire clients site rather than a backup folder. It was at this point I realised the massive flaws in our backup strategies. The offsite copy was out of date, the backups on the server were stored in an 8GB file we couldn't download to access and because of the spaces in file names, we couldn't restore a whole directory and instead had to specify each filename in double quotes manually. took me a couple of hours to write the script just to automate the process of restoring files one by one.

1
vote

Deleted a disk image file of a vmware guest VM, I didn't have the backup of.

Luckily the guest was still running so I've managed to back it up and restore in the new VM. It's still running, two years later.

But the worst thing happened not when I was logged in. I was messing around, moving things in the rack and suddenly 500GB external disk containg the archives for last year for 3 customers fell on the floor. All gone. And it was double-disk, RAID1. I have learned the hard way RAID1 wasn't a catch-all solution for disk failure ;)

1
  • how did you handle the data loss?
    – Demi
    Jun 16, 2014 at 2:19
0
votes

I accidentally shut down the wrong computer. It was a ssh connection to the proxy server at work, I wanted to shut down my laptop but instead I put 50 people without internet. I recovered by running to the serverroom and booting the server again.

0
votes

Someone I worked with removed the password file then logged out of a critical production sun server. I had to repair it as we could not just reinstall it.

Another admin create an account with the same uid and home as the main production database and then removed the copy account. It deleted the databases user and home - this was on a non stop Tandem, there was some egg on the admin face as he made it stop.

I have removed the resource forks from Apple files on a Sun server as I did not know why the temp files where taking up so much space. The Mac users where not impressed ;)

0
votes

Ran a script that deletes all files on a file server recursively.

We had a partial backup of files, so we never fully recovered..

1
  • You do realize when people say that rm -rf / fixes all problems on your server, they're joking, right? ;-)
    – Josh
    Feb 26, 2010 at 14:58
0
votes

Went to init 1 to restart some services I had patched, forgetting that I was SSH'ed in and the server was a 3-hour drive away. Luckily we had a service call to that location the next day, and the server in question was a backup, not in production.

0
votes

Mistakenly restarted all (20) of the production servers while rebooting our workstations after patches. The server names were in the text file my script was feeding from.

It killed uptime, which wasn't that big a deal in that organization.. but Exchange didn't come back up on its own -- which was how it was noticed the next morning. (Not technically as 'root', but Domain Admin counts, right? :) )

0
votes

sudo rm -Rf /*

0
votes

Converted a hard drive to a dynamic disk in Windows 2003 Server. It was the right thing to do, except on the wrong hard drive. Took some time to get the data back, also because I had no clue what to do at first.

0
votes

On my home server, I chained too many important commands together with && when resizing a partition. I believe it was (excluding arguments) fsck && resize2fs, followed by an lvreduce on the next line. fsck didn't return the exit code I assumed it had, so resize2fs never ran, and the lvreduce totally hosed a bunch of my data when it truncated the volume.

I am much more careful with && these days.

1
  • Sounds like not enough '&&', instead of too much. Would '&& lvreduce' have worked?
    – romandas
    Jul 9, 2009 at 20:30
-1
votes

I've accidentally deleted data that I shouldn't have deleted. I've also accidentally shut down the server. :-\

Not the answer you're looking for? Browse other questions tagged .