87
votes

I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.

I'll add my own story as an answer.

5
  • 3
    See also serverfault.com/questions/5066
    – Zoredache
    May 13, 2009 at 18:26
  • 14
    This really is more of a poll then a question. This should probably be set to community wiki.
    – Zoredache
    May 13, 2009 at 18:31
  • 7
    Yup, this should definitely be a community wiki. In the intent of the question, though, my favorite story is the 500 mile email one - ibiblio.org/harris/500milemail.html - although, obviously, that wasn't me. May 13, 2009 at 20:26
  • the 500miles is just wonderful Oct 5, 2010 at 20:13
  • This should be called "worse system administrator accident."
    – Rilindo
    Sep 2, 2011 at 3:20

86 Answers 86

133
votes

I had fun discovering the difference between the linux "killall" command (kills all processes matching the specified name, useful for stopping zombies) and the solaris "killall" command (kills all processes and halts the system, useful for stopping the production server in the middle of peak hours and getting all your co-workers to laugh at you for a week).

6
  • 74
    Been there, done that. We aliased the killall-command on the solaris-boxes after that: alias killall='echo ORLLY?' =) May 28, 2009 at 12:03
  • 29
    There's also an important difference on Solaris between 'ifconfig -a4' (show all interfaces' IPv4 information) and 'ifconfig -a 4' (set all interfaces to 0.0.0.4).
    – Zanchey
    Jul 19, 2009 at 7:28
  • 3
    +1 "oops, this wind is logged into solaris?" Jul 21, 2009 at 0:30
  • 4
    @Commander, I was going to upvote your comment, but there are exactly 42 upvotes in this very moment... I just couldn't.
    – Massimo
    Jul 29, 2010 at 18:42
  • 5
    Also, hostname -f on Linux prints the fully-qualified domain name on Linux. On Solaris, it sets the hostname to -f. Oct 14, 2010 at 19:11
73
votes

I was in charge of our corporate web proxy which at the time was Netscape's product. While playing around in the admin forms (it was a web based interface) there was a big (and I swear it was red) button that said Delete User Database. No problem, I thought. Let's see what the options it gives me are when I hit that. Surely there will be a confirmation prompt if there are no options.

Yeah, no confirmation. No options. No more users.

So, went over to Mr. Solaris Sysadmin and said that I was in desperate need of a restore from tape to which he replied, "I don't back that box up."

"Uh, come again," I retorted.

"I don't back that box up. It's on my list of things to add to the backup rotation but I haven't gotten around to it yet."

"This server's been in production for nearly 8 months!" I screamed.

shrug, he replied. "Sorry."

12
  • 50
    This is definitely a story for all those people who complain about those pesky 'Are you sure?' dialogs ;)
    – MikeyB
    May 13, 2009 at 20:24
  • 45
    I'm not really sure you should be the one to be screaming at him...
    – Mikeage
    May 14, 2009 at 11:10
  • 14
    This isn't really an accident - you pressed the Delete User Database button on purpose, come on dude... Jun 20, 2009 at 23:07
  • 5
    Yeah, you're right. Pushing the button was not an accident. Actually deleting the database was the accident part. Whatever...
    – squillman
    Jun 21, 2009 at 3:30
  • 7
    That button should have been followed by two or three confirmation prompts. How is it a useful feature to delete the user database? I totally blame the programmer that put the button in there. And the fact that the db wasn't backed up. "Genius may have its limitations, but stupidity is not thus handicapped." (attr to Elbert Hubbard). Building with the assumption that humans aren't curious, careless apes is just building a disaster. Aug 26, 2009 at 17:44
66
votes

Many years ago the company I worked for had a client which ran a nightly backup of their NT 4.0 Server to a Jaz drive (like a high capacity zip disk).

We set up a batch file, which ran as a scheduled job overnight. Every morning they'd collect last nights disk from the drive, and before they left in the evening they'd insert the next disk in the sequence.

Anyway, the batch file looked something like this (the Jaz drive was drive F:)...

@echo off
F:
deltree /y *.*
xcopy <important files> F:

Anyway, one night they forgot to put the disk in. The change to drive F: failed (no disk in drive), and the batch file continued to run. The default working directory for the batch file? C:. First time I've ever seen a backup routine destroy the server it was backing up.

I learned a little something about sysadminning (and exception handling) that day.

Jim.

PS: The fix? "deltree /y F:\*.*".

13
  • 57
    ... we have to explain what Jaz drives are? Am I really that old?? May 14, 2009 at 0:54
  • 3
    It was an IOMEGA thing (remember Zip Drives?) Was the big brother of the Zip Drives, and it was like a hard disk without heads, just the platters, that went inside of a plastic case with a little window, and when inserted on the drive, the drive inserted the headers on it. Where like 1 or 2GB, expensive, and tend to overheat, so wasn't recomended to leave the cartridges inside the reader (this said by an old IOMEGA boss)
    – Andor
    May 14, 2009 at 11:29
  • 3
    Hehe, I like how jaz drive is explained by comparison to the just-as-dead-and-irrelevant zip drive.
    – Luke
    Jun 9, 2009 at 19:51
  • 2
    If it makes you feel older, there are sysadmins that have never seen deltree before.
    – Joseph
    Jun 20, 2009 at 16:17
  • 5
    I've had to explain to a lot of programmers that when you attempt an operation that changes a state, and yet might fail and leave you in the previous state, you have to check whether it succeeded if you're going to do something dangerous should you be in the previous state. Why do I have to explain this?
    – carlito
    Jun 29, 2009 at 22:49
61
votes

root@dbhost# find / -name core -exec rm -f {} \;

Me: "You can't get in? OK. What's the DB name?"

Cu: "Core."

Me: "Oh."

3
  • 3
    and from then on, use of the 'file' command was added to the cleanup crontab... :)
    – MikeyB
    May 31, 2009 at 4:14
  • 3
    Oh dear God....
    – squillman
    May 31, 2009 at 4:33
  • 4
    Oh my god... wow. I'm going to make a mental note of this one, so I never do it. Jul 16, 2009 at 22:23
60
votes

I love the way everyone qualifies their story with "when I was young/green" as if they would never do it again. Accidents can happen to even the most seasoned pros.

My own worst moment is so bad I still get palpitations thinking about it...

We had a SAN with production data on it. Critical to the company. My "mentor" decided to extend a partition to free up some disk space. Can you see where this is heading? He said that the SAN software could do this live, in production hours and no-one would notice. Alarm bells should have started ringing, but were conspicuously silent. He said he'd done it "loads of times before" with no problems. But here's the thing - he got ME to click the button that said "are you sure?"! As I was new to the company I assumed this guy knew what he was talking about. Big mistake. The good news was that the LUN got extended. The bad news was...well I knew there was bad news when I started seeing disk write errors on the Windows box.

I'm glad I was wearing brown pants.

We had to explain why 1TB of data had disappeared at lunchtime. That was a really, really bad day.

It's a good principle actually - before you do something that you have doubts about, imagine having to explain to management if something goes wrong. If you can't think of a good answer to explain your actions then don't do it.

5
  • 17
    +1 for the last paragraph - the "sit on your hands" technique, one vital minute for reflection
    – Andy
    Jun 6, 2009 at 21:02
  • 12
    There's a certain skill you get after working with live systems a while: a spider sense for what's dangerous and what's not. Like pausing an extra second before pressing return at a root prompt, or making sure the SQL update statement has a proper where clause (that's already been run in a select count(*)). Sep 9, 2009 at 16:47
  • 18
    I want a popup that says something like "Would you like to print your resume before continuing?" .. and have only one option: "Yes"
    – warren
    Jun 14, 2010 at 17:57
  • +1 for the advice in the last paragraph Jun 25, 2010 at 11:40
  • 3
    +1, ditto. I remember hearing from a sysadmin friend that there had been issues in his annual performance appraisal about his poor typing speed, which he contemptuously (and correctly) dismissed with "I'm not paid to type fast. I'm paid to hit enter very slowly and thoughtfully.".
    – MadHatter
    Sep 9, 2011 at 8:23
53
votes

Nagios pinged us one morning when business hours started to say that it couldn't connect to a non-critical server. Ok, hike to the server room. It's an old server, a Dell 1650 purchased in '02, and we knew that the 1650s have been having hardware problems. The PFY stabs the power button. Nothing. Hit it again, and hold it for five seconds to 'force power on' ... which overrides the BMC's error protection, since without a DRAC there's no way to examine the BMC logs without having the power on to the chassis.

The machine starts POST, and then dies again. I'm standing above it and go, "I smell smoke." We pull the server out on it's rails, and one of the power supplies feels warm, so the PFY pulls it and is about to close the box back up. I say, "No, that's not power supply smoke, that's motherboard smoke."

We open the case again and look for the source of the burning smell. Turns out an inductor coil and a capacitor something blew off the voltage regulator on the motherboard, and sprayed molten copper and capacitor goop across everything, shorting a bunch of stuff and basically making a big mess.

The worst part for me was recognizing that I'd smoked enough hardware to recognize the difference between the smell of a burnt motherboard and a burnt power supply.

2
  • 17
    This job's selection criteria include: good sense of smell. Brilliant.
    – mlp
    Jul 7, 2009 at 22:43
  • 13
    Buddy of mine screwed a set of drive rails into a disk so far that they went into the circuit board and shorted it out. Pretty pink smoke. Very distinguishable.
    – squillman
    Aug 20, 2009 at 2:54
47
votes

Three days ago (seriously) I was remotely logged in to a school server, installing Service Pack 2 on a Windows Server 2008 file server.

I decided to schedule the needed reboot for late at night, when teachers wouldn't be logged on finishing their end-of-year report cards. I typed something like:

 at 23:59 "shutdown -r -t 0" 

...which might have worked fine.

But then I second guessed myself. Was my 'shutdown' syntax correct? I tried to view the usage help by typing

 shutdown /h 

...and instantly lost my RDP connection. Panicking, I hit up Google for the syntax. A quick search revealed that the Server 2008 version of shutdown includes a /h switch, which (as you may have guessed) hibernates the machine.

Teachers started calling me within minutes to report that they could no longer open or save the report cards they had been working on. Since I was offsite and the server room was locked, I had to call the school principal directly and walk her through the process of powering the machine back on.

Today I brought homemade cookies to everyone as a form of apology.

5
  • 32
    A perfect example of flawed user interface design at the command line: whatever happened to the "Principle of Least Surprise"?
    – Mei
    Jun 17, 2009 at 23:06
  • 9
    on DOS/Windows isn't it /? Aug 26, 2009 at 17:52
  • 3
    It's usually /? on Win, but there are lots of utilities that are ported from UNIX or written by UNIX people (including quite a few such at MS) which are -h or /h Oct 9, 2009 at 14:17
  • 6
    ..but you still ALWAYS try the /? first!
    – warren
    Jun 14, 2010 at 18:01
  • 14
    That's why I like Linux. man shutdown. I know I'm not going to cause problems with man!
    – Josh
    Oct 13, 2010 at 23:01
37
votes

In a previous job, we had a great homegrown system that logged and archived every single piece of mail that entered, left or stayed within the company.

Blew away your entire mailbox? No problem! Looking for a piece of mail that somebody sent you a week/month/year ago but you can't remember who sent it or what the subject was? No problem! We'll just redeliver everything from February for you to a special folder.

At some point, the need came for the CEO of the company to monitor mail going between a competitor and an internal salesperson under suspicion. So we setup a script than ran every night and delivered relevant mail from the previous day to the CEO. No problem!

Around a month later word of a double-plus urgent problem came down from on high. Seems that as the CEO was reading through the list of mails sent to $OTHERCOMPANY, he came across this one:

To: somebody@$OTHERCOMPANY
From: CEO
Subject: CEO has read your message (subject line here)

Naturally, the CEO being an important person and all, he was too busy to click on all those "Send Read Receipt" dialogs in Outlook and had configured his client to just send them all. One of the messages caught by the monitoring filter had a read-receipt request set. Guess what Outlook did? Certainly buggered up the 'clandestine' monitoring.

Our next task: adding rules to the mail filter to block outgoing read receipts from the CEO to that company. Yes, it was the easiest way. :)

3
  • 3
    That's not legal in my country though. At all.
    – mafu
    Jun 1, 2010 at 10:32
  • 1
    Ah, well that's your country. :) In Canada, it's just fine.
    – MikeyB
    Jun 9, 2010 at 18:17
  • 7
    It's not legal to monitor email coming into or leaving your own company's servers? What country do you live in? Oct 29, 2010 at 0:10
36
votes

Ahhh,mine was about 10 years ago, when I was still getting my feet wet. I had the joy of installing battery backups on all the programmers computers. They also wanted the software loaded to warn of power outage and shut down properly.

So I set it up on my computer to test everything first of course and make sure it all worked. So I disconnect the power cord and the message comes up on my screen. "external power lost, beginning system shutdown".

So I thought, Hey cool, it worked. But for some weird reason, I don't even remember, it sent that message out as a network message so all 200+ computers in the company got that message, where 100+ users where programmers.

Yeah, talk about mass freak out!!

I kept my head low in that place for awhile!

5
  • 3
    Haha, sounds like what happened to a friend of mine here with "net send" and about 1300 recipients :)
    – squillman
    May 13, 2009 at 20:17
  • 10
    Ugh. I did the "net send" thing in college, too. I thought "they had to have disabled this" ba-ding! ba-ding! all over the lab, computers had the message up. I decided to go apologize to the net admin, and on my way, every computer I passed had the message up. /sigh May 28, 2009 at 18:19
  • 3
    Yup, walk of shame right there! My friend didn't get out of his chair. Didn't bother answering his phone, either.
    – squillman
    Jun 12, 2009 at 15:29
  • 10
    Haha a friend of mine net send'd everyone at the school "HELP! I'm trapped in Room 114" and of course 60 seconds later a small army of people turned up to find out who had sent the message Jun 21, 2009 at 5:12
  • 2
    We actually used that feature to punk the sys admin at my junior college. Cleared all the computer labs with fake AV messages and system shutdown messages that looked like they came from him. He strongly suspected we were the culprits but me and my buddy each had an alibi (I sent the first one when he was in class and he sent the second one while I was in class) and the computers had generic lab logons
    – Shial
    Aug 18, 2009 at 13:08
35
votes

I would often use the "sys-unconfig" command on Solaris machines to reset the machine Name service, I.P. address, and root password. I was on a users system and I logged into the building install server and looked something up (as root), then forgetting that i had logged into another machine (non descriptive "#" prompt) I ran the "sys-unconfig" command.

# sys-unconfig     
        WARNING

This program will unconfigure your system.  It will cause it
to revert to a "blank" system - it will not have a name or know
about other systems or networks.

This program will also halt the system.

Do you want to continue (y/n) ? y

Connection closed

#

That "connection closed" message slowly turned to panic... what machine was I logged into when I ran that command.

The worst part of this was not the hard time my co-workers gave me, it was that I did the same thing a month later.

5
  • 24
    Oh, the classic, "what machine am I on??" moment of panic. I've been there. I feel your pain.
    – sysadmin1138
    May 19, 2009 at 17:35
  • 2
    Me too. Anyone who has experienced that knows exactly what the phrase "ice cold blood" means. May 28, 2009 at 18:23
  • 21
    There is a reason my shell prompt always includes the username and hostname...
    – derobert
    Jun 5, 2009 at 17:35
  • 2
    I once set up aliases for logging in to the different machines to get different colored backgrounds to be able to even more clearly recognize which machine I was on.
    – Zitrax
    Jun 11, 2009 at 22:10
  • As I use Linux on both servers and my local machine, I often have root prompts opened on servers and my PC, the servers have red prompts for distinction Oct 5, 2010 at 20:28
27
votes

I've got a pretty good one. Admittedly, it was prior to my time as a sysadmin, but still tech-related so I figured I'd add it.

Back in the day, I was working as a satcom/wideband tech for the USAF. Having recently graduated technical school, I found myself stationed in South Korea. Shortly after arriving on-station an opportunity arose to travel down south with the "big guys" who'd been there for a while and actually work on some real-world,(i.e. `production') equipment.

I went down with the crew and as an eager, young tech, was chomping at the bit, quite excited at the prospect of getting my hands on an actual piece of equipment that was passing LIVE military voice and data traffic.

To start me off slowly, they handed me a manual, turned to the preventative maintenance section and pointed me in the direction of four racks filled with several large digital multiplexers. The equipment was easy enough, we'd covered the same equipment in tech school.

First page of the manual read; "Apply power to the ditigal multiplexer. Turn both rear switches to the ON position and wait for the equipment to power-up, then begin tests." I looked up, and there was already power APPLIED!

I was in a quandary for sure. Not knowing how to proceed, I shot my best, `Ummmm.. Kinda lost here' look at the senior.

He looked over at me and laughed, "No, no, it's ok. You can ignore that part of the checklist." Then, as he noticed the look on my face, (since we were taught in school to NEVER, EVER ignore any part of a checklist, and it was certain death and destruction if one was to do so) he put a serious look on his face and said, "Ignore ONLY that part! Follow the rest of it, to the letter!"

Dutifully, I ran through the multi-step PM instructions, happy as a clam and proud that they were letting such a low-ranking, (albeit smart) tech do this important work.

Somewhere between the fifth and sixth preventative maintenance checklist on these huge multiplexers I started noticing an increased level of activity around me. Phones were ringing, people were moving quickly. Quizzical looks were being exchanged.

Finally, a group of folks ran up to me, headed by one of the senior techs who had brought me down.

"Hey! We're seeing HUGE outages in data traffic, and we've isolated/traced the path back to the racks that you're working on! Are you seeing any weird.."

(At that point he was cut off by another one of the troubleshooters who'd made her way around to the first group of multiplexers that I had been performing the PMs on.)

"HOLY NUTS! THEY'RE TURNED OFF! HE'S BEEN TURNING THEM OFF!!!!"

In short order, I watched as they hurriedly ran through the first step in the manual, "Turn both rear switches to the ON position..." When the senior tech was done, he came over to me and incredulously asked what I was thinking of, by turning the critical pieces of equipment off.

Scared out of my wits, I handed him the checklist that I'd been following, swearing that I hadn't deviated at ALL. That I had followed it, `to the letter' as he'd instructed.

After a while he laughed and pointed out where the problem lay.

In the manual, the FINAL step in the preventative maintenance checklist was:

"Record final probe reading, wipe down front panel, removing all dust and particulate, then turn both rear power switches to the OFF position."

:)

3
  • I don't now what any of this means Oct 16, 2010 at 5:24
  • Didn't the last step saying to power off the machines seem a little strange? Just wondering. Oct 29, 2010 at 0:19
  • 2
    It probably did seem strange, but remember, he was in the military. Imagine trying to command an army where every soldier says, "Wait, are you sure? That order sounds funny to me."
    – Kyralessa
    Dec 11, 2010 at 2:54
26
votes

It's kind of a sysadmin accident.. in so far as sysadmins occasionally have to physically haul large numbers of machines from point A to point B (where A and B are seemingly always separated by several flights of stairs in a building with no lift). On the n'th trip of the day, I stopped for a breather three flights up from the basement loading level to chat with someone coming down, propped the full-size tower w/station I was schlepping on the inside handrail of the open stairwell and... well, you guessed... slightly lost my grip on it. It plunged unerringly straight down the well and when it reached the bottom, er... not so much with the functionality for that one! Total salvageable parts: two sticks of RAM, one floppy drive and one ISDN card (God bless the Hermstedt engineering folks!). Everything else either cracked, rattling or smashed into tiny pieces.

By the grace of God, nobody was walking underneath, which, thankfully for me, was my boss' first though, so I got to keep my job. Felt very sick for an hour or so though.

Moral: gravity always wins!

3
  • 2
    You didn't take a picture of the wreckage, did you? May 28, 2009 at 11:13
  • 5
    @J.Pablo - No, I'm afraid this was over ten years ago, when cameras were big and had chemicals in them and stuff. If I'd done it recently, I'd have had the aftermath on YouTube within ten minutes!
    – avstrallen
    May 28, 2009 at 11:29
  • True story: this happened with me 9 years ago, but with an Air Conditioner and from the window of the 9th floor. It wasn't my fault (I was just helping someone), and no one was injured, but I felt sick for days.
    – imgx64
    Jan 1, 2012 at 5:31
26
votes

I was reloading a system for someone, and during the manual backup process I asked him the question "Do you have any other programs you use?" and "Is there anything else important you do on the computer?"

He said "no" SEVERAL times.

I was convinced and formatted the drive.

About 30 minutes later he said "oh my god" and put both hands on his head.

Turns out he had been working on a book script for over 10 YEARS in a specialized program. This was back when programs used to save user data in its program files directory and I missed it.

Whhhhooooops.

He wasn't mad at me, but it was a sobering feeling.

6
  • 7
    You've taught him a valuable lesson: Backups are important
    – MikeyB
    Sep 10, 2009 at 5:00
  • 5
    Just reading this really hurts. So painful. Sure a lesson, but...
    – mafu
    Jun 1, 2010 at 10:36
  • Still a good experience for the poor guy. He could have been working for 30 YEARS and have a heart attack in front of you.
    – halp
    Aug 8, 2010 at 11:56
  • Man I remember doing something similar. Aparently my uncle had a ton of like clientelle stuff for this custom windows accouting app his job gave him. It was in program files /sigh. Luckily I remembered it when my I fixed my dads work pc which had some custom insurance app on it.
    – PHGamer
    Oct 16, 2010 at 3:50
  • @phgamer, can I ask you to watch your language please - this isn't that kind of place sorry.
    – Chopper3
    Oct 29, 2010 at 13:06
23
votes

My personal favourite isn't actually mine, and I'm VERY glad of it. Take a look here.

1
23
votes

This didn't happen to me, but…

I was working at a company that made software that ran on Linux machines provided by the client. We would essentially 'take over' the machines, completely configure them to our specs, and do all of the management and monitoring. Essentially, we were a team of 10-15 sysadmins, managing thousands of servers for hundreds of customers. Mistakes were bound to happen.

One of our team found some issues on a server (a backup, I believe), and decided that he should run fsck on it. He stopped all relevant services, made sure that the system had had backups taken recently, and then ran the fsck, but it complained that the filesystem was mounted. Since we were remote and had no remote access (DRAC, ILO, etc.), he couldn't do the fsck, but he was pretty sure that it was safe to do it with the filesystem mounted, if you were careful.

He decided to try it himself by running fsck on his root partition, with predictable results – he corrupted his root partition and couldn't boot anymore.

Confused, he went over and talked to our team lead. The lead said he was pretty sure that you couldn't do that, and the team member said 'Sure you can!', took the lead's keyboard, and showed him that you could – by running fsck on the lead's root partition. Which completely corrupted HIS root partition.

End result? No customer data lost, thanks to the team member's testing. Two days of employee productivity were lost, but that was worth far, far less than the data on the customer's machine. And for the record? You can run fsck on a mounted drive, but only to verify data. Not to repair it. That was the team member's mistake.

--

To add my own story, I was working at the same company, and was trying to reset a user password. Our system refused to let me set it to the password he needed, because it tracked old password hashes and refused to let you duplicate the password. The mechanism was simple: it validated your password against the most recent hash in the database.

(And for the record, it needed to be the old password because it was a shared account, and making sure everyone knew the new password was impractical)

I decided to just go into the users database and delete the new records so that it would use the older one. It's all just SQL (running an ancient version of Sybase), so it's easy. First, I had to find the records:

SELECT * FROM users_passwords WHERE username='someuser';

I found the old record he wanted to keep; there were two more in front of it. I decided to be clever and just delete anything newer than the old record. Looking at the result set, I saw that the old password was ID #28 in the database, and the new ones were ID #several thousand (very busy system). That's simple, all the old rows were > 28, so:

DELETE FROM users_passwords WHERE id > 28;

There's nothing worse than doing some simple row pruning and seeing '212,500 rows affected'. Fortunately, we had two master database servers (with the user ID), but Sybase (at least, our version) didn't support automatic replication, so it didn't automatically wipe out the old records. It was a trivial matter to get a dump of the users_passwords table and re-import it. Still, a pretty big 'oh f**k!' moment.

9
  • That's why there is a "LIMIT" statement in sql ;)
    – hayalci
    May 28, 2009 at 23:56
  • 20
    More to the point, that's why you always issue a "DELETE FROM xxx" as a "SELECT * FROM xxx" first. Then double-check what it shows, and then just replace SELECT * with DELETE.
    – sleske
    May 29, 2009 at 1:11
  • 1
    FYI, I'm not sure even a fsck w/o changes is safe on mounted ext3, it does a journal replay after all...
    – derobert
    Jun 5, 2009 at 17:41
  • 14
    No, that is why you always issue a begin tran first. Then when you see many more rows affected than intended you can issue a rollback.
    – pipTheGeek
    Jun 5, 2009 at 18:13
  • 1
    @Dan That is almost the same as any other RDBMS, you just make sure you run your statement, have your checks already planned, like just checking rows affected or having a select ready to check the result of an update, then commit or rollback. It saved me once when I forgot to highlight the WHERE clause in a T-SQL statement in MS-SQL 2005.
    – pipTheGeek
    Jun 8, 2009 at 18:51
22
votes

Another of my favorites:

When setting up a computer and a local laser printer on a system, I had the bright idea to plug them both into the computer's UPS. Ever try to print to a local laser printer when it's plugged into a desktop UPS? Well, if you don't know, it tends to pull all the amps... Which restarts the computer... And the print job never finishes...!

Ever get the call: 'Whenever I print, it restarts my computer and doesn't print!!!'?

Ooops!

JFV

2
  • 1
    Oooh, I am just waiting to see that happen at my church now :)
    – squillman
    May 31, 2009 at 4:34
  • 2
    Most UPS manufacturers say "don't plug in laser printers" because they will overload them.
    – Andrew
    Jul 23, 2010 at 7:01
22
votes

DELETE statement without a WHERE clause, on the customers' live patron database.

3
  • 9
    "BEGIN TRANSACTION" FTW!
    – spoulson
    Sep 1, 2009 at 17:03
  • 6
    Where were you before i wiped out their live database!
    – Ian Boyd
    Sep 2, 2009 at 14:53
  • 1
    That's a kind of rite of passage; every fledgling DBA has to do that at least once. Hopefully on something that's not too important...
    – RainyRat
    Nov 5, 2009 at 12:22
22
votes

Typed kill 1 as root. init and all of her children died. And all of their children. etc, etc. Oops.

What I meant to type was kill %1

After I realised what I did I ran to the control panel of a BIG wool bale sorting machine and hit the emergency stop button. This stopped the machine ripping itself to bits, as I had just killed the software which controlled it.

4
  • 1
    That one made me laugh! Having worked with large machinery in the past I can appericate control systems. Jun 21, 2009 at 0:12
  • 1
    On BSD, kill -1 1 caused init to reread inittab or /etc/ttys. Leaving off the "minus" had consequences...
    – kmarsh
    Jul 15, 2009 at 17:53
  • 2
    Thank goodness for that emergency stop button! :-)
    – staticsan
    Aug 24, 2009 at 23:23
  • +1 for not being able to say "You can't cause any hardware damage by mucking about with software."
    – outis
    Aug 31, 2011 at 13:02
20
votes

At some point in my career a legal investigation at the company I was working for placed a requirement on us that all email be kept from "this day" forward, until told otherwise. After about a year of storing daily full backups of our exchange environment (1TB nightly) we started to run out of space.

The exchange admins suggested that we only keep every 8th copy of the email. To do this, we had them restore a days worth of the exchange databases, extract the email they needed (specific people flagged for investigation) and re-archive it. They did this for every 8th day of email for all of our backups. The 8th day was chose because exchange had a parameter set where "deleted items" are kept in the database for 8 days.

After they would finish each archive, I would go back through and delete any backups which were older than what they had archived.

TSM does not have an easy way to do this, so you have to manually delete objects from the backup database.

I wrote a script which would delete all backups older than some date, by way of a date calculation using the difference between today, and the date in question. On some day I had to delete about a months worth of backups, except when I made the date calculation I made a typo and entered the date as 7/10/2007 instead of 6/10/2007, and ran the script. I deleted an entire extra month worth of data, accidentally which was part of a very important lawsuit.

After that, I added some steps to the script to confirm that you wanted to delete the data, and show you what it was going to delete...

Luckily, they never even used any of the data we worked so hard to preserve, and I still have my job.

2
  • 5
    1 TB a DAY of Exchange data? Wow - you guys had 365 terabytes dedicated JUST to extra Exchange backups? That's some serious storage you've got going there.
    – Carl C
    Jun 20, 2009 at 17:17
  • 28
    You would have only lost 1 day if you had been in Europe :-) Aug 5, 2009 at 6:04
20
votes

After a long day or performance tracing and tuning a huge mainframe (you know the beasts that take a couple of hours before all standby backup-sites have agreed that it is indeed booted up again and fully synced) I stretched my fingers, typed satisfied shutdown -p now in my laptop prompt, closed the lid, yanked the serial cable out of the mainframe, with the anticipation of a nice cold glass of lager.

Suddenly I hear the deafening sound of spinning down mainframe while my laptop was still happily displaying X.

While waiting for the machine to come fully online again I decided that I got time to get my ACPI working on my laptop so I never ever are tempted to cli shutdown my laptop.

1
  • If spinning DOWN mainframes makes a heck of a noise, I don't want to hear it spinning UP, or worse, at full-speed. Jul 6, 2010 at 23:04
20
votes

We were in the middle of a power outage and saw that the UPS was running at 112% of it's configured load. This wasn't much of an issue as we were running on the generator at the time.

So we went around pulling backup power cables to reduce the power usage on that UPS (we had two, one much larger than the other). We got to the network switch which ran the server room (this was the server room with all the internal servers for the company, with the customer facing servers in another server room). The switch was a large enterprise class switch with three power supplies in it. The supplies were N+1 so we only needed two in order to run the switch.

We picked a cable and pulled it out. Unfortunately for us the other two were plugged into a single power strip, which promptly blew as the load went up on the two power supplies which were plugged into it. The sysadmin then panicked and plugged the third cable in. The switch tried to fire up, putting the entire load of the switch unto the single power supply. Instead of the power supply shutting down, it exploded in a shower of sparks not 12 inches from my face sending me jumping back into the rack of servers.

Out of instinct I tried to jump to the side, but unfortunately on my left was a wall, and two my right was a very large 6'4" facilities guy. I some how managed to jump over him, or possibly through him bouncing off of the Compaq racks (the ones with the thin mesh fronts) without putting a whole in the rack, and without touching the facilities guy.

1
  • 1
    Thank you. Denis the facilities guy was impressed by the fact that I didn't knock him over.
    – mrdenny
    Jun 24, 2009 at 4:27
16
votes

This accident didn't happen... but it's worth mentioning:

I was sent to a heavily-used data center to conduct bandwidth tests on a new circuit. I got to the demarc room/IDF, found a spot on one of the racks for my test router, made my connections, and started the tests. Unfortunately, I completely failed to notice the in-production border router not only being exactly on the next rack (almost at the same level), but that it was also the same make and model as my testing router.

When the test was done, I began pressing the power switch to the off position (...imagine it in slow motion...) and, I swear, just as I was applying pressure it dawned on me that the router I was about to turn off was the one in production. My heart stopped and I almost... well, use your imagination.

I left the data center's MDF looking spooked and pale, but at the same time glad I still had a job!

3
  • 18
    Just imagine it could have been one of those landmine situations. You realize as your finger is depressing the button what you've done, and you can't remove your finger or the server will go down. So all you can do it stand there and yell for help.
    – Tom Ritter
    Jun 16, 2009 at 17:21
  • 20
    LOL.. hadn't thought about that! It would have been a WAY better story. I can imagine the data center network engineers huddling around, pitching different solutions to the problem. Maybe one of them would have posted (If SF was available then) a question like: "How can we disable the power switch on a router once it has been pushed by idiot techie?" (+500 bounty points)
    – l0c0b0x
    Jun 16, 2009 at 18:07
  • 21
    If it's similar to where I've worked, the solution would involve quickly removing the offending finger and replacing it with a large quantity of duct tape. (And then intending to schedule downtime of the machine to remove the tape, but putting it off for 18 months until until the tape finally fails).
    – Ian
    Jul 14, 2009 at 7:40
16
votes

I deleted someone's account by mistake, got the names mixed up with the one I was suspose to delete. Opps

The cool part is they never knew what happened. Got the call they couldn't log in, the penny dropped about the account I deleted.

While on the phone with them, I quickly re-created their account, re-attached their old mailbox to it (thankfully Exchange doesn't delete mailboxes right away) and pointed it back to their old user files.

Then I blamed them for forgetting their password which I had just reset for them :)

1
  • 9
    I'm torn between +1'ing the balls you had to use that excuse, and -1'ing for lying to the user....
    – Babu
    Jul 8, 2009 at 2:28
16
votes

Accidentally installed a tar.gz file on my Gentoo Linux box in the wrong place and it left files all over the place. This must've been around 1999, 19 at the time (thanks for the comments below)

Being the geek that I am, I decided to try to script myself out of the work of going manually through each file.

So I tried:

tar --list evilevilpackage.tar.gz | xargs rm -rf

It didn't take me very long to notice that tar also listed all the directories the program was using, those included were ''/usr, /var, /etc'' and a few others that I didn't really want gone.

CTRL-C! CTRL-C! CTRL-C! Too late! Everything gone, reinstall time. Fortunately the box didn't contain anything important.

5
  • 10
    I hate to nag the details, but I'm pretty sure Gentoo started around 2002 May 28, 2009 at 18:28
  • Yeah, 1.0 was released in 2002. RedHat or Slackware would be a more likely choice back in '98 Jun 2, 2009 at 23:59
  • Unless he was really hardcore, Enoch Linux release 0.75 in Dec 1999 and eventually became Gentoo with version 1.0 in 2002
    – Shial
    Aug 18, 2009 at 13:20
  • Was living in my mothers basement at the time, so it's around that time, at least before 2000. Not sure if it's hard core or not; it was the next logical step after Slackware.
    – Andrioid
    Aug 20, 2009 at 15:00
  • Slackware had .tgz packages. Sep 28, 2010 at 23:34
12
votes

As a smallish part of my former life I administered the company's file server, a netware 4:11 box. It hardly EVER needed any input at all, but if it did, you opened up a remote console window.

Used to using DOS all the time, when I was finished, I naturally would type "Exit". For Netware, "exit" is the command to shut down the OS. Luckily, it won't let you shut down unless you first "Down" the server.(Make it unavailable to the network/clients) So when you type "Exit" in the console, it helpfully says, "You must first type "Down" before you can exit"

Ask me how many times I 1: typed "exit" in the console session and 2: Obediently typed "Down" and then "Exit" so I could "finish what I was trying to do"

And then the phone starts ringing.....

LOL

2
  • 8
    more than once??
    – BradC
    Jun 4, 2009 at 19:51
  • 2
    That's why I use ^D to logout. It just ends a shell session. Oct 5, 2010 at 20:55
11
votes

Another story that didn't happen (phew):

We were doing incremental backups religiously every day to a tape drive.

We happened to write a tape containing data to ship to someone else. They said 'we can't read your tape'. In fact, neither could we. Or any tape in fact.

We bought another tape drive and held our breath until we installed it.

Moral of the story. Always make sure you test your backups.

11
votes

The last place I worked, my co-worker had his kids with him in the server room (why? I have NO IDEA!).

He made sure that they were far away from the servers and explained to his 5-year-old that he shouldn't touch ANY of the servers and ESPECIALLY none of the power switches.

In fact, he had them right near the door... (can you see where this is going...?)

The boy didn't touch any of the server power buttons... No, that would be entirely too easy to explain. Instead he hit the BIG RED BUTTON that was near the door... The button that shuts down power to the ENTIRE SERVER ROOM!!!

Phone lines immediately started to light up wondering why Exchange, File Servers, etc. weren't available... Imagine trying to explain THAT to the CEO!

-JFV

4
  • 4
    we had a courier do the same thing. Not quite a sysadmin accident though, although a strange metal box suddenly appeared over the Big Red Button after that.
    – marty
    Jul 14, 2009 at 4:34
  • see also "Molly Guard": catb.org/~esr/jargon/html/M/molly-guard.html
    – jay_dubya
    Jul 21, 2009 at 7:37
  • My brother did the same thing when he was 3 at a communications link site my father used to work at. Half the link network went down when he toggled the main power switch that was conveniently 18 inches off the floor. Fortunately, the network was fully duplicated, so no traffic was lost, but still...
    – staticsan
    Aug 24, 2009 at 23:18
  • @marty - the proper quote is: "Why? I don't know. He's on third. and I don't give a darn." "What?" "I don't give a darn." "Oh! That's our shortstop!"
    – warren
    Sep 29, 2010 at 1:30
10
votes

I once had a fight with the APC UPS monitoring software. Being a small company, we had a couple of small-ish UPSes and various servers were setup to monitor them. Most of the servers were Linux, but a few were running Windows and so they were the ones used because the APC software is Windows only.

However, the APC software at the time was hard-coded to assume the UPS it is talking to is also powering the PC its running on! This was not the case for this server, but I discovered that too late to tell it to halt. Also unfortunately, the lead programmer was demonstrating the company product to a partner - it was a web-based app, running on the same server I didn't want the APC software to shut down...

2
  • 1
    thankfully we have the apcupsd for linux Oct 5, 2010 at 20:56
  • Well, we do now. This incident was 10 years ago. :-)
    – staticsan
    Oct 5, 2010 at 22:04
8
votes

I was giving a new sysadmin a tour of a Service Manager app. I said "if you ever needed to stop this service you would click this button, but you should never do it during the day." You would never believe how sensitive her mouse button was!

Two minutes later the service had started up again, and no-one seemed to notice.

8
votes

Tripping over a tower server that was wedged behind a rack and hitting my head on the back of the main Cisco router on my way down. Thus revealing how loosely the power cords were actually seated in the power supplies on the front of the Catalyst 6500.

Yeah. We've got a hardhat on a hook in the server room now. With my name on it.

Not the answer you're looking for? Browse other questions tagged .