Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.

I'll add my own story as an answer.

share

locked by sysadmin1138 Jan 6 '12 at 3:44

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as not constructive by Mark Henderson Jan 4 '12 at 6:29

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

3  
See also serverfault.com/questions/5066 –  Zoredache May 13 '09 at 18:26
14  
This really is more of a poll then a question. This should probably be set to community wiki. –  Zoredache May 13 '09 at 18:31
7  
Yup, this should definitely be a community wiki. In the intent of the question, though, my favorite story is the 500 mile email one - ibiblio.org/harris/500milemail.html - although, obviously, that wasn't me. –  Mihai Limbăşan May 13 '09 at 20:26

86 Answers 86

I would often use the "sys-unconfig" command on Solaris machines to reset the machine Name service, I.P. address, and root password. I was on a users system and I logged into the building install server and looked something up (as root), then forgetting that i had logged into another machine (non descriptive "#" prompt) I ran the "sys-unconfig" command.

# sys-unconfig     
		WARNING

This program will unconfigure your system.  It will cause it
to revert to a "blank" system - it will not have a name or know
about other systems or networks.

This program will also halt the system.

Do you want to continue (y/n) ? y

Connection closed

#

That "connection closed" message slowly turned to panic... what machine was I logged into when I ran that command.

The worst part of this was not the hard time my co-workers gave me, it was that I did the same thing a month later.

share
24  
Oh, the classic, "what machine am I on??" moment of panic. I've been there. I feel your pain. –  sysadmin1138 May 19 '09 at 17:35
2  
Me too. Anyone who has experienced that knows exactly what the phrase "ice cold blood" means. –  Matt Simmons May 28 '09 at 18:23
21  
There is a reason my shell prompt always includes the username and hostname... –  derobert Jun 5 '09 at 17:35
2  
I once set up aliases for logging in to the different machines to get different colored backgrounds to be able to even more clearly recognize which machine I was on. –  Zitrax Jun 11 '09 at 22:10

Everyone 'rm -rf /'s at some point accidentally. Mine was trying to delete some of the extra files in my home directory 2 days before my last data structures assignment was due.

Professionally I've been capable enough to not have any catastrophic screw ups so far.

share
3  
I always put the -rf at the end, so when I'm typing "rm -rf /tmp/blah*" I can't accidentally run "rm -rf /" by prematurely hitting enter. It'll be "rm /" which will fail with an error and do no harm. –  J. Pablo Fernández May 28 '09 at 11:27
3  
I always start my rm -fr prompt with a # so I don't accidentally press enter at / :) –  rkthkr Jun 5 '09 at 19:35
4  
I've yet to delete the root accidentally, but I have done it on purpose once, on an ancient HP-UX box that had caused me much pain over the years and was being decommissioned. It gave me great joy to stand and watch as it happily nuked its own filesystem. –  RainyRat Jun 23 '09 at 14:44

Ahhh,mine was about 10 years ago, when I was still getting my feet wet. I had the joy of installing battery backups on all the programmers computers. They also wanted the software loaded to warn of power outage and shut down properly.

So I set it up on my computer to test everything first of course and make sure it all worked. So I disconnect the power cord and the message comes up on my screen. "external power lost, beginning system shutdown".

So I thought, Hey cool, it worked. But for some weird reason, I don't even remember, it sent that message out as a network message so all 200+ computers in the company got that message, where 100+ users where programmers.

Yeah, talk about mass freak out!!

I kept my head low in that place for awhile!

share
3  
Haha, sounds like what happened to a friend of mine here with "net send" and about 1300 recipients :) –  squillman May 13 '09 at 20:17
10  
Ugh. I did the "net send" thing in college, too. I thought "they had to have disabled this" ba-ding! ba-ding! all over the lab, computers had the message up. I decided to go apologize to the net admin, and on my way, every computer I passed had the message up. /sigh –  Matt Simmons May 28 '09 at 18:19
3  
Yup, walk of shame right there! My friend didn't get out of his chair. Didn't bother answering his phone, either. –  squillman Jun 12 '09 at 15:29
10  
Haha a friend of mine net send'd everyone at the school "HELP! I'm trapped in Room 114" and of course 60 seconds later a small army of people turned up to find out who had sent the message –  Mark Henderson Jun 21 '09 at 5:12
2  
We actually used that feature to punk the sys admin at my junior college. Cleared all the computer labs with fake AV messages and system shutdown messages that looked like they came from him. He strongly suspected we were the culprits but me and my buddy each had an alibi (I sent the first one when he was in class and he sent the second one while I was in class) and the computers had generic lab logons –  Shial Aug 18 '09 at 13:08

At some point in my career a legal investigation at the company I was working for placed a requirement on us that all email be kept from "this day" forward, until told otherwise. After about a year of storing daily full backups of our exchange environment (1TB nightly) we started to run out of space.

The exchange admins suggested that we only keep every 8th copy of the email. To do this, we had them restore a days worth of the exchange databases, extract the email they needed (specific people flagged for investigation) and re-archive it. They did this for every 8th day of email for all of our backups. The 8th day was chose because exchange had a parameter set where "deleted items" are kept in the database for 8 days.

After they would finish each archive, I would go back through and delete any backups which were older than what they had archived.

TSM does not have an easy way to do this, so you have to manually delete objects from the backup database.

I wrote a script which would delete all backups older than some date, by way of a date calculation using the difference between today, and the date in question. On some day I had to delete about a months worth of backups, except when I made the date calculation I made a typo and entered the date as 7/10/2007 instead of 6/10/2007, and ran the script. I deleted an entire extra month worth of data, accidentally which was part of a very important lawsuit.

After that, I added some steps to the script to confirm that you wanted to delete the data, and show you what it was going to delete...

Luckily, they never even used any of the data we worked so hard to preserve, and I still have my job.

share
5  
1 TB a DAY of Exchange data? Wow - you guys had 365 terabytes dedicated JUST to extra Exchange backups? That's some serious storage you've got going there. –  Carl Campos Jun 20 '09 at 17:17
28  
You would have only lost 1 day if you had been in Europe :-) –  PowerApp101 Aug 5 '09 at 6:04

Back in the day, when I was very green, I needed to install AV software on my users PC's, as no-one seemed to have it. So I spent a bit of time figuring out how to do a remote install, rather than poking around 40 or 50 desktops. The remote installation ran perfectly and everything seemed fine, until various managers dropped by my office to complain that they couldn't log in.

It turned out that a few individuals had Symantec AV installed on their machines, and this did not coexist at all well with the McAfee software I was using and would lock up the machines after a login attempt.

Fortunately, it was possible to remotely disable the service if you got to the machine before they tried to log in, so I managed to get points for fixing it instead of having to rebuild all of senior managements PC's...

share

This accident didn't happen... but it's worth mentioning:

I was sent to a heavily-used data center to conduct bandwidth tests on a new circuit. I got to the demarc room/IDF, found a spot on one of the racks for my test router, made my connections, and started the tests. Unfortunately, I completely failed to notice the in-production border router not only being exactly on the next rack (almost at the same level), but that it was also the same make and model as my testing router.

When the test was done, I began pressing the power switch to the off position (...imagine it in slow motion...) and, I swear, just as I was applying pressure it dawned on me that the router I was about to turn off was the one in production. My heart stopped and I almost... well, use your imagination.

I left the data center's MDF looking spooked and pale, but at the same time glad I still had a job!

share
18  
Just imagine it could have been one of those landmine situations. You realize as your finger is depressing the button what you've done, and you can't remove your finger or the server will go down. So all you can do it stand there and yell for help. –  Tom Ritter Jun 16 '09 at 17:21
20  
LOL.. hadn't thought about that! It would have been a WAY better story. I can imagine the data center network engineers huddling around, pitching different solutions to the problem. Maybe one of them would have posted (If SF was available then) a question like: "How can we disable the power switch on a router once it has been pushed by idiot techie?" (+500 bounty points) –  l0c0b0x Jun 16 '09 at 18:07
21  
If it's similar to where I've worked, the solution would involve quickly removing the offending finger and replacing it with a large quantity of duct tape. (And then intending to schedule downtime of the machine to remove the tape, but putting it off for 18 months until until the tape finally fails). –  iAn Jul 14 '09 at 7:40

In a previous job, we had a great homegrown system that logged and archived every single piece of mail that entered, left or stayed within the company.

Blew away your entire mailbox? No problem! Looking for a piece of mail that somebody sent you a week/month/year ago but you can't remember who sent it or what the subject was? No problem! We'll just redeliver everything from February for you to a special folder.

At some point, the need came for the CEO of the company to monitor mail going between a competitor and an internal salesperson under suspicion. So we setup a script than ran every night and delivered relevant mail from the previous day to the CEO. No problem!

Around a month later word of a double-plus urgent problem came down from on high. Seems that as the CEO was reading through the list of mails sent to $OTHERCOMPANY, he came across this one:

To: somebody@$OTHERCOMPANY
From: CEO
Subject: CEO has read your message (subject line here)

Naturally, the CEO being an important person and all, he was too busy to click on all those "Send Read Receipt" dialogs in Outlook and had configured his client to just send them all. One of the messages caught by the monitoring filter had a read-receipt request set. Guess what Outlook did? Certainly buggered up the 'clandestine' monitoring.

Our next task: adding rules to the mail filter to block outgoing read receipts from the CEO to that company. Yes, it was the easiest way. :)

share
3  
That's not legal in my country though. At all. –  mafu Jun 1 '10 at 10:32
1  
Ah, well that's your country. :) In Canada, it's just fine. –  MikeyB Jun 9 '10 at 18:17
7  
It's not legal to monitor email coming into or leaving your own company's servers? What country do you live in? –  Andrew Oct 29 '10 at 0:10
1  
+1 for the newspeak –  fahadsadah Dec 19 '10 at 13:18

I was in charge of our corporate web proxy which at the time was Netscape's product. While playing around in the admin forms (it was a web based interface) there was a big (and I swear it was red) button that said Delete User Database. No problem, I thought. Let's see what the options it gives me are when I hit that. Surely there will be a confirmation prompt if there are no options.

Yeah, no confirmation. No options. No more users.

So, went over to Mr. Solaris Sysadmin and said that I was in desperate need of a restore from tape to which he replied, "I don't back that box up."

"Uh, come again," I retorted.

"I don't back that box up. It's on my list of things to add to the backup rotation but I haven't gotten around to it yet."

"This server's been in production for nearly 8 months!" I screamed.

shrug, he replied. "Sorry."

share
50  
This is definitely a story for all those people who complain about those pesky 'Are you sure?' dialogs ;) –  MikeyB May 13 '09 at 20:24
45  
I'm not really sure you should be the one to be screaming at him... –  Mikeage May 14 '09 at 11:10
14  
This isn't really an accident - you pressed the Delete User Database button on purpose, come on dude... –  Wayne Koorts Jun 20 '09 at 23:07
5  
Yeah, you're right. Pushing the button was not an accident. Actually deleting the database was the accident part. Whatever... –  squillman Jun 21 '09 at 3:30
7  
That button should have been followed by two or three confirmation prompts. How is it a useful feature to delete the user database? I totally blame the programmer that put the button in there. And the fact that the db wasn't backed up. "Genius may have its limitations, but stupidity is not thus handicapped." (attr to Elbert Hubbard). Building with the assumption that humans aren't curious, careless apes is just building a disaster. –  Jared Updike Aug 26 '09 at 17:44

I had fun discovering the difference between the linux "killall" command (kills all processes matching the specified name, useful for stopping zombies) and the solaris "killall" command (kills all processes and halts the system, useful for stopping the production server in the middle of peak hours and getting all your co-workers to laugh at you for a week).

share
74  
Been there, done that. We aliased the killall-command on the solaris-boxes after that: alias killall='echo ORLLY?' =) –  Commander Keen May 28 '09 at 12:03
29  
There's also an important difference on Solaris between 'ifconfig -a4' (show all interfaces' IPv4 information) and 'ifconfig -a 4' (set all interfaces to 0.0.0.4). –  Zanchey Jul 19 '09 at 7:28
3  
+1 "oops, this wind is logged into solaris?" –  Mark Harrison Jul 21 '09 at 0:30
4  
@Commander, I was going to upvote your comment, but there are exactly 42 upvotes in this very moment... I just couldn't. –  Massimo Jul 29 '10 at 18:42
5  
Also, hostname -f on Linux prints the fully-qualified domain name on Linux. On Solaris, it sets the hostname to -f. –  200_success Oct 14 '10 at 19:11

Many years ago the company I worked for had a client which ran a nightly backup of their NT 4.0 Server to a Jaz drive (like a high capacity zip disk).

We set up a batch file, which ran as a scheduled job overnight. Every morning they'd collect last nights disk from the drive, and before they left in the evening they'd insert the next disk in the sequence.

Anyway, the batch file looked something like this (the Jaz drive was drive F:)...

@echo off
F:
deltree /y *.*
xcopy <important files> F:

Anyway, one night they forgot to put the disk in. The change to drive F: failed (no disk in drive), and the batch file continued to run. The default working directory for the batch file? C:. First time I've ever seen a backup routine destroy the server it was backing up.

I learned a little something about sysadminning (and exception handling) that day.

Jim.

PS: The fix? "deltree /y F:\*.*".

share
57  
... we have to explain what Jaz drives are? Am I really that old?? –  Spencer Ruport May 14 '09 at 0:54
3  
It was an IOMEGA thing (remember Zip Drives?) Was the big brother of the Zip Drives, and it was like a hard disk without heads, just the platters, that went inside of a plastic case with a little window, and when inserted on the drive, the drive inserted the headers on it. Where like 1 or 2GB, expensive, and tend to overheat, so wasn't recomended to leave the cartridges inside the reader (this said by an old IOMEGA boss) –  Andor May 14 '09 at 11:29
3  
Hehe, I like how jaz drive is explained by comparison to the just-as-dead-and-irrelevant zip drive. –  Luke Jun 9 '09 at 19:51
2  
If it makes you feel older, there are sysadmins that have never seen deltree before. –  Joseph Jun 20 '09 at 16:17
5  
I've had to explain to a lot of programmers that when you attempt an operation that changes a state, and yet might fail and leave you in the previous state, you have to check whether it succeeded if you're going to do something dangerous should you be in the previous state. Why do I have to explain this? –  carlito Jun 29 '09 at 22:49

I once had a fight with the APC UPS monitoring software. Being a small company, we had a couple of small-ish UPSes and various servers were setup to monitor them. Most of the servers were Linux, but a few were running Windows and so they were the ones used because the APC software is Windows only.

However, the APC software at the time was hard-coded to assume the UPS it is talking to is also powering the PC its running on! This was not the case for this server, but I discovered that too late to tell it to halt. Also unfortunately, the lead programmer was demonstrating the company product to a partner - it was a web-based app, running on the same server I didn't want the APC software to shut down...

share
1  
thankfully we have the apcupsd for linux –  Hubert Kario Oct 5 '10 at 20:56

Early on when I was a young one, I was trying to be 'helpful' and tried to copy 250 MB of data over a 128 kbit/s line to 86 different sites at the same time... during business hours. While I was doing this, I overheard people asking why everything was taking so long.

Needless to say, I killed the transfers, and (luckily) no one knew it was me!

share

Another of my favorites:

When setting up a computer and a local laser printer on a system, I had the bright idea to plug them both into the computer's UPS. Ever try to print to a local laser printer when it's plugged into a desktop UPS? Well, if you don't know, it tends to pull all the amps... Which restarts the computer... And the print job never finishes...!

Ever get the call: 'Whenever I print, it restarts my computer and doesn't print!!!'?

Ooops!

JFV

share
1  
Oooh, I am just waiting to see that happen at my church now :) –  squillman May 31 '09 at 4:34
2  
Most UPS manufacturers say "don't plug in laser printers" because they will overload them. –  Andrew Jul 23 '10 at 7:01

Another story that didn't happen (phew):

We were doing incremental backups religiously every day to a tape drive.

We happened to write a tape containing data to ship to someone else. They said 'we can't read your tape'. In fact, neither could we. Or any tape in fact.

We bought another tape drive and held our breath until we installed it.

Moral of the story. Always make sure you test your backups.

share

I love the way everyone qualifies their story with "when I was young/green" as if they would never do it again. Accidents can happen to even the most seasoned pros.

My own worst moment is so bad I still get palpitations thinking about it...

We had a SAN with production data on it. Critical to the company. My "mentor" decided to extend a partition to free up some disk space. Can you see where this is heading? He said that the SAN software could do this live, in production hours and no-one would notice. Alarm bells should have started ringing, but were conspicuously silent. He said he'd done it "loads of times before" with no problems. But here's the thing - he got ME to click the button that said "are you sure?"! As I was new to the company I assumed this guy knew what he was talking about. Big mistake. The good news was that the LUN got extended. The bad news was...well I knew there was bad news when I started seeing disk write errors on the Windows box.

I'm glad I was wearing brown pants.

We had to explain why 1TB of data had disappeared at lunchtime. That was a really, really bad day.

It's a good principle actually - before you do something that you have doubts about, imagine having to explain to management if something goes wrong. If you can't think of a good answer to explain your actions then don't do it.

share
17  
+1 for the last paragraph - the "sit on your hands" technique, one vital minute for reflection –  Andy Jun 6 '09 at 21:02
12  
There's a certain skill you get after working with live systems a while: a spider sense for what's dangerous and what's not. Like pausing an extra second before pressing return at a root prompt, or making sure the SQL update statement has a proper where clause (that's already been run in a select count(*)). –  jplindstrom Sep 9 '09 at 16:47
18  
I want a popup that says something like "Would you like to print your resume before continuing?" .. and have only one option: "Yes" –  warren Jun 14 '10 at 17:57
3  
+1, ditto. I remember hearing from a sysadmin friend that there had been issues in his annual performance appraisal about his poor typing speed, which he contemptuously (and correctly) dismissed with "I'm not paid to type fast. I'm paid to hit enter very slowly and thoughtfully.". –  MadHatter Sep 9 '11 at 8:23

It's kind of a sysadmin accident.. in so far as sysadmins occasionally have to physically haul large numbers of machines from point A to point B (where A and B are seemingly always separated by several flights of stairs in a building with no lift). On the n'th trip of the day, I stopped for a breather three flights up from the basement loading level to chat with someone coming down, propped the full-size tower w/station I was schlepping on the inside handrail of the open stairwell and... well, you guessed... slightly lost my grip on it. It plunged unerringly straight down the well and when it reached the bottom, er... not so much with the functionality for that one! Total salvageable parts: two sticks of RAM, one floppy drive and one ISDN card (God bless the Hermstedt engineering folks!). Everything else either cracked, rattling or smashed into tiny pieces.

By the grace of God, nobody was walking underneath, which, thankfully for me, was my boss' first though, so I got to keep my job. Felt very sick for an hour or so though.

Moral: gravity always wins!

share
2  
You didn't take a picture of the wreckage, did you? –  J. Pablo Fernández May 28 '09 at 11:13
5  
@J.Pablo - No, I'm afraid this was over ten years ago, when cameras were big and had chemicals in them and stuff. If I'd done it recently, I'd have had the aftermath on YouTube within ten minutes! –  avstrallen May 28 '09 at 11:29

I was giving a new sysadmin a tour of a Service Manager app. I said "if you ever needed to stop this service you would click this button, but you should never do it during the day." You would never believe how sensitive her mouse button was!

Two minutes later the service had started up again, and no-one seemed to notice.

share

I work for a wireless provider in North America, and had done some training for a person in my group to run through work orders. I had stayed up the first couple of nights (we do everything during the maintenance window), but he was doing fine and said he's got to learn it on his own, so I let him and left my cell phone and pager on. I logged in and checked the configuration when I got up at 8 a.m. the following morning.

The change was that we were adding a new pool of IP addresses for BlackBerrys, the pool we were adding was about 10000 addresses. To do this, we add routes on the router that point to the processor address on a blade that does all the call processing (essentially it works like a proxy). Also, we log into the processor and configure the IP pool, and link the IP pool to be used for our wireless users. However for testing, we normally configure this on one processor (actually boot up a phone and test all the features), and then just move the configuration to the actual processor we want it on.

Fast forward two weeks, and I get a call from our control center that there's been a lot of call in's about some intermittent BlackBerry problems, and the few BlackBerrys they've looked at seem to be cycling through a common pool, but weren't really sure what was happening. It only took me about 5 minutes to realize that this was the new pool my colleage had just added two weeks before. It also didn't take long to see that the router had two routes in it, one going to the test processor, and one going to the proper call processor. This being what it was, he forgot to delete the route to the test processor, and it superceded the proper route.

Essentially a BlackBerry would connect to the network, connect to the proxy to get its IP address, the proxy would give it an address from the pool with the incorrect route, and the BlackBerry would try and talk to the RIM relay, and the response would be routed to the test proxy and never make it back to the user, essentially meaning no connectivity.

We got lucky though since BlackBerrys have a behaviour that if they can't contact the relay, they will disconnect / reconnect to the network, but nonetheless some RIM devices were without service for up to several hours until they were able to cycle onto a working pool. I thought back, and when I double checked the work, I had only check the proxy configuration which was new to this guy, I never checked the routing configuration since this guy was previously with the backbone team and routing was his thing. Oops!

I fixed it and called him up that afternoon, his day was going well, but I started with I'm sorry, but I'm about to ruin you're entire week. A year later the story still comes up around beers.

share

My aunt asked me to fix their computer. They said it wouldn't boot up and its been like that for 2 weeks. I suspected it was either the BIOS or the OS.

I sat down in front of their computer. I crouched down to push the power button. I look up.

The BIOS passed. That's good.

The OS booted. That's good.

I moved the mouse around thinking maybe there's a problem with the input devices. There was no problem with the input devices.

I opened up her word processor. It ran.

I print test the printer. It printed.

By this point, I stood up and told my aunt (who was watching me) that there is nothing wrong with the computer. She claimed that it wasn't like that before I sat down.

I can now claim to my family that I am so good, that I can fix any computer just by sitting in front of it.

share
1  
P.s. I'm a programmer, not a system administrator. But my non-technical family members don't know the difference. Meh. –  MrValdez May 15 '09 at 5:25
19  
I regularly scare computers into working just by walking into the room. –  sysadmin1138 May 19 '09 at 17:48
8  
@sysadmin1138 Being able to do that is a prerequisite for the job. –  MikeyB May 31 '09 at 4:17
2  
My parents will call me when they have problems, usually just having me on the phone is usually enough to fix the issue. –  Bryan Rehbein Jun 12 '09 at 17:13
14  
A well-known AI koan: A novice was trying to fix a broken Lisp machine by turning the power off and on. Knight, seeing what the student was doing spoke sternly: "You can not fix a machine by just power-cycling it with no understanding of what is going wrong." Knight turned the machine off and on. The machine worked. –  Dennis Williamson Jun 21 '09 at 13:42

A long time ago, I decided to change the mount point of my data partition. So I created a new directory, changed the mount point in /etc/fstab, and deleted the directory it was previously mounted on.

The thing is I only realized that the partitions was still mounted on the old directory when nautilus showed me a progress bar (for what should be a 4Kb deletion). Thankfully I was able to cancel it before a great damage was done, but I did lose some files.

share

Accidentally installed a tar.gz file on my Gentoo Linux box in the wrong place and it left files all over the place. This must've been around 1999, 19 at the time (thanks for the comments below)

Being the geek that I am, I decided to try to script myself out of the work of going manually through each file.

So I tried:

tar --list evilevilpackage.tar.gz | xargs rm -rf

It didn't take me very long to notice that tar also listed all the directories the program was using, those included were ''/usr, /var, /etc'' and a few others that I didn't really want gone.

CTRL-C! CTRL-C! CTRL-C! Too late! Everything gone, reinstall time. Fortunately the box didn't contain anything important.

share
10  
I hate to nag the details, but I'm pretty sure Gentoo started around 2002 –  Matt Simmons May 28 '09 at 18:28

This didn't happen to me, but…

I was working at a company that made software that ran on Linux machines provided by the client. We would essentially 'take over' the machines, completely configure them to our specs, and do all of the management and monitoring. Essentially, we were a team of 10-15 sysadmins, managing thousands of servers for hundreds of customers. Mistakes were bound to happen.

One of our team found some issues on a server (a backup, I believe), and decided that he should run fsck on it. He stopped all relevant services, made sure that the system had had backups taken recently, and then ran the fsck, but it complained that the filesystem was mounted. Since we were remote and had no remote access (DRAC, ILO, etc.), he couldn't do the fsck, but he was pretty sure that it was safe to do it with the filesystem mounted, if you were careful.

He decided to try it himself by running fsck on his root partition, with predictable results – he corrupted his root partition and couldn't boot anymore.

Confused, he went over and talked to our team lead. The lead said he was pretty sure that you couldn't do that, and the team member said 'Sure you can!', took the lead's keyboard, and showed him that you could – by running fsck on the lead's root partition. Which completely corrupted HIS root partition.

End result? No customer data lost, thanks to the team member's testing. Two days of employee productivity were lost, but that was worth far, far less than the data on the customer's machine. And for the record? You can run fsck on a mounted drive, but only to verify data. Not to repair it. That was the team member's mistake.

--

To add my own story, I was working at the same company, and was trying to reset a user password. Our system refused to let me set it to the password he needed, because it tracked old password hashes and refused to let you duplicate the password. The mechanism was simple: it validated your password against the most recent hash in the database.

(And for the record, it needed to be the old password because it was a shared account, and making sure everyone knew the new password was impractical)

I decided to just go into the users database and delete the new records so that it would use the older one. It's all just SQL (running an ancient version of Sybase), so it's easy. First, I had to find the records:

SELECT * FROM users_passwords WHERE username='someuser';

I found the old record he wanted to keep; there were two more in front of it. I decided to be clever and just delete anything newer than the old record. Looking at the result set, I saw that the old password was ID #28 in the database, and the new ones were ID #several thousand (very busy system). That's simple, all the old rows were > 28, so:

DELETE FROM users_passwords WHERE id > 28;

There's nothing worse than doing some simple row pruning and seeing '212,500 rows affected'. Fortunately, we had two master database servers (with the user ID), but Sybase (at least, our version) didn't support automatic replication, so it didn't automatically wipe out the old records. It was a trivial matter to get a dump of the users_passwords table and re-import it. Still, a pretty big 'oh f**k!' moment.

share
20  
More to the point, that's why you always issue a "DELETE FROM xxx" as a "SELECT * FROM xxx" first. Then double-check what it shows, and then just replace SELECT * with DELETE. –  sleske May 29 '09 at 1:11
1  
FYI, I'm not sure even a fsck w/o changes is safe on mounted ext3, it does a journal replay after all... –  derobert Jun 5 '09 at 17:41
14  
No, that is why you always issue a begin tran first. Then when you see many more rows affected than intended you can issue a rollback. –  pipTheGeek Jun 5 '09 at 18:13
1  
@Dan That is almost the same as any other RDBMS, you just make sure you run your statement, have your checks already planned, like just checking rows affected or having a select ready to check the result of an update, then commit or rollback. It saved me once when I forgot to highlight the WHERE clause in a T-SQL statement in MS-SQL 2005. –  pipTheGeek Jun 8 '09 at 18:51

root@dbhost# find / -name core -exec rm -f {} \;

Me: "You can't get in? OK. What's the DB name?"

Cu: "Core."

Me: "Oh."

share
3  
and from then on, use of the 'file' command was added to the cleanup crontab... :) –  MikeyB May 31 '09 at 4:14
3  
Oh dear God.... –  squillman May 31 '09 at 4:33
4  
Oh my god... wow. I'm going to make a mental note of this one, so I never do it. –  Glenn Willen Jul 16 '09 at 22:23

As a smallish part of my former life I administered the company's file server, a netware 4:11 box. It hardly EVER needed any input at all, but if it did, you opened up a remote console window.

Used to using DOS all the time, when I was finished, I naturally would type "Exit". For Netware, "exit" is the command to shut down the OS. Luckily, it won't let you shut down unless you first "Down" the server.(Make it unavailable to the network/clients) So when you type "Exit" in the console, it helpfully says, "You must first type "Down" before you can exit"

Ask me how many times I 1: typed "exit" in the console session and 2: Obediently typed "Down" and then "Exit" so I could "finish what I was trying to do"

And then the phone starts ringing.....

LOL

share
8  
more than once?? –  BradC Jun 4 '09 at 19:51
2  
That's why I use ^D to logout. It just ends a shell session. –  Hubert Kario Oct 5 '10 at 20:55

I had an employee complain that his laptop was slow, so I checked the hard drive fragmentation and it was (and is to this day) the worst I had ever seen. Attempts to defragment the drive were fruitless because there was not enough free space. I tried cleaning up temporary files (not sure why I didn't just move stuff to the server temporarily) and stupidly deleted his entire outlook.pst thinking that it was a backup of his e-mail and not his actual e-mail. He forgave me, but never let me forget it.

(This happened many years ago shortly after I graduated university. I'm much more competent now.)

share

DELETE statement without a WHERE clause, on the customers' live patron database.

share
9  
"BEGIN TRANSACTION" FTW! –  spoulson Sep 1 '09 at 17:03
6  
Where were you before i wiped out their live database! –  Ian Boyd Sep 2 '09 at 14:53
1  
That's a kind of rite of passage; every fledgling DBA has to do that at least once. Hopefully on something that's not too important... –  RainyRat Nov 5 '09 at 12:22

I got called to investigate an alert coming from a Windows machine that was indicating that the monitoring system had no license file. I opened up the command prompt and started to investigate the problem and found that the basic windows commands were not even there.

A sysadmin who had run a script remotely had written a script which used the del command to delete a folder specified by a root and subfolder with the folders specified in Environment Variables. If the Environment Variables were not set, it silently deleted the whole partition.

When told, the sysadmin was so surprised that they confirmed the action by running the said script on their own notebook, thus trashing it too.

The amazing thing was that Windows was running fine, until we rebooted the server. Only the stingy monitoring software complained.

It was the secondary Active Directory server for a political party. Oops.

share

On my first installation task (many years ago, in DOS age) I accidentally delete almost all system files and half application files on computer which belongs to director of public institution. But it wasn't my fault. I try to delete non important files in C:/TEMP folder to free some space. Delete begins...after a few moments I see some familiar names from root and DOS folder scrolling up on screen...Hitting hard Ctrl+Break...but too late...

That was the harder way to learn what cross linked files problem on FAT file system is.

share

When I was first hired as sysadmin by the lead admin...within the first week we received a brand new Dell server...Windows Server 2003...it was his little baby until I was secretly called to the server room at midnight one Saturday night to clean numerous instances of malware from it because he was SURFING THE WEB with it before deployment WITHOUT ANTIVIRUS!!!

Malware cleaning is something that I have had much experience with, but since this was a server I did a format and reinstall to be extra safe.

I never said a word to him about it. He knew he had messed up royally.

share

Not the answer you're looking for? Browse other questions tagged or ask your own question.