Best system administrator accident [closed]

Question

I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.

I'll add my own story as an answer.

Answer 1

I had fun discovering the difference between the linux "killall" command (kills all processes matching the specified name, useful for stopping zombies) and the solaris "killall" command (kills all processes and halts the system, useful for stopping the production server in the middle of peak hours and getting all your co-workers to laugh at you for a week).

Answer 2

I was in charge of our corporate web proxy which at the time was Netscape's product. While playing around in the admin forms (it was a web based interface) there was a big (and I swear it was red) button that said Delete User Database. No problem, I thought. Let's see what the options it gives me are when I hit that. Surely there will be a confirmation prompt if there are no options.

Yeah, no confirmation. No options. No more users.

So, went over to Mr. Solaris Sysadmin and said that I was in desperate need of a restore from tape to which he replied, "I don't back that box up."

"Uh, come again," I retorted.

"I don't back that box up. It's on my list of things to add to the backup rotation but I haven't gotten around to it yet."

"This server's been in production for nearly 8 months!" I screamed.

shrug, he replied. "Sorry."

Answer 3

Many years ago the company I worked for had a client which ran a nightly backup of their NT 4.0 Server to a Jaz drive (like a high capacity zip disk).

We set up a batch file, which ran as a scheduled job overnight. Every morning they'd collect last nights disk from the drive, and before they left in the evening they'd insert the next disk in the sequence.

Anyway, the batch file looked something like this (the Jaz drive was drive F:)...

@echo off
F:
deltree /y *.*
xcopy <important files> F:

Anyway, one night they forgot to put the disk in. The change to drive F: failed (no disk in drive), and the batch file continued to run. The default working directory for the batch file? C:. First time I've ever seen a backup routine destroy the server it was backing up.

I learned a little something about sysadminning (and exception handling) that day.

Jim.

PS: The fix? "deltree /y F:\*.*".

Answer 4

root@dbhost# find / -name core -exec rm -f {} \;

Me: "You can't get in? OK. What's the DB name?"

Cu: "Core."

Me: "Oh."

Answer 5

I love the way everyone qualifies their story with "when I was young/green" as if they would never do it again. Accidents can happen to even the most seasoned pros.

My own worst moment is so bad I still get palpitations thinking about it...

We had a SAN with production data on it. Critical to the company. My "mentor" decided to extend a partition to free up some disk space. Can you see where this is heading? He said that the SAN software could do this live, in production hours and no-one would notice. Alarm bells should have started ringing, but were conspicuously silent. He said he'd done it "loads of times before" with no problems. But here's the thing - he got ME to click the button that said "are you sure?"! As I was new to the company I assumed this guy knew what he was talking about. Big mistake. The good news was that the LUN got extended. The bad news was...well I knew there was bad news when I started seeing disk write errors on the Windows box.

I'm glad I was wearing brown pants.

We had to explain why 1TB of data had disappeared at lunchtime. That was a really, really bad day.

It's a good principle actually - before you do something that you have doubts about, imagine having to explain to management if something goes wrong. If you can't think of a good answer to explain your actions then don't do it.

Answer 6

Nagios pinged us one morning when business hours started to say that it couldn't connect to a non-critical server. Ok, hike to the server room. It's an old server, a Dell 1650 purchased in '02, and we knew that the 1650s have been having hardware problems. The PFY stabs the power button. Nothing. Hit it again, and hold it for five seconds to 'force power on' ... which overrides the BMC's error protection, since without a DRAC there's no way to examine the BMC logs without having the power on to the chassis.

The machine starts POST, and then dies again. I'm standing above it and go, "I smell smoke." We pull the server out on it's rails, and one of the power supplies feels warm, so the PFY pulls it and is about to close the box back up. I say, "No, that's not power supply smoke, that's motherboard smoke."

We open the case again and look for the source of the burning smell. Turns out an inductor coil and a capacitor something blew off the voltage regulator on the motherboard, and sprayed molten copper and capacitor goop across everything, shorting a bunch of stuff and basically making a big mess.

The worst part for me was recognizing that I'd smoked enough hardware to recognize the difference between the smell of a burnt motherboard and a burnt power supply.

Answer 7

Three days ago (seriously) I was remotely logged in to a school server, installing Service Pack 2 on a Windows Server 2008 file server.

I decided to schedule the needed reboot for late at night, when teachers wouldn't be logged on finishing their end-of-year report cards. I typed something like:

 at 23:59 "shutdown -r -t 0" 

...which might have worked fine.

But then I second guessed myself. Was my 'shutdown' syntax correct? I tried to view the usage help by typing

 shutdown /h 

...and instantly lost my RDP connection. Panicking, I hit up Google for the syntax. A quick search revealed that the Server 2008 version of shutdown includes a /h switch, which (as you may have guessed) hibernates the machine.

Teachers started calling me within minutes to report that they could no longer open or save the report cards they had been working on. Since I was offsite and the server room was locked, I had to call the school principal directly and walk her through the process of powering the machine back on.

Today I brought homemade cookies to everyone as a form of apology.

Answer 8

In a previous job, we had a great homegrown system that logged and archived every single piece of mail that entered, left or stayed within the company.

Blew away your entire mailbox? No problem! Looking for a piece of mail that somebody sent you a week/month/year ago but you can't remember who sent it or what the subject was? No problem! We'll just redeliver everything from February for you to a special folder.

At some point, the need came for the CEO of the company to monitor mail going between a competitor and an internal salesperson under suspicion. So we setup a script than ran every night and delivered relevant mail from the previous day to the CEO. No problem!

Around a month later word of a double-plus urgent problem came down from on high. Seems that as the CEO was reading through the list of mails sent to $OTHERCOMPANY, he came across this one:

To: somebody@$OTHERCOMPANY
From: CEO
Subject: CEO has read your message (subject line here)

Naturally, the CEO being an important person and all, he was too busy to click on all those "Send Read Receipt" dialogs in Outlook and had configured his client to just send them all. One of the messages caught by the monitoring filter had a read-receipt request set. Guess what Outlook did? Certainly buggered up the 'clandestine' monitoring.

Our next task: adding rules to the mail filter to block outgoing read receipts from the CEO to that company. Yes, it was the easiest way. :)

Answer 9

Ahhh,mine was about 10 years ago, when I was still getting my feet wet. I had the joy of installing battery backups on all the programmers computers. They also wanted the software loaded to warn of power outage and shut down properly.

So I set it up on my computer to test everything first of course and make sure it all worked. So I disconnect the power cord and the message comes up on my screen. "external power lost, beginning system shutdown".

So I thought, Hey cool, it worked. But for some weird reason, I don't even remember, it sent that message out as a network message so all 200+ computers in the company got that message, where 100+ users where programmers.

Yeah, talk about mass freak out!!

I kept my head low in that place for awhile!

Answer 10

I would often use the "sys-unconfig" command on Solaris machines to reset the machine Name service, I.P. address, and root password. I was on a users system and I logged into the building install server and looked something up (as root), then forgetting that i had logged into another machine (non descriptive "#" prompt) I ran the "sys-unconfig" command.

# sys-unconfig     
        WARNING

This program will unconfigure your system.  It will cause it
to revert to a "blank" system - it will not have a name or know
about other systems or networks.

This program will also halt the system.

Do you want to continue (y/n) ? y

Connection closed

#

That "connection closed" message slowly turned to panic... what machine was I logged into when I ran that command.

The worst part of this was not the hard time my co-workers gave me, it was that I did the same thing a month later.

Answer 11

I've got a pretty good one. Admittedly, it was prior to my time as a sysadmin, but still tech-related so I figured I'd add it.

Back in the day, I was working as a satcom/wideband tech for the USAF. Having recently graduated technical school, I found myself stationed in South Korea. Shortly after arriving on-station an opportunity arose to travel down south with the "big guys" who'd been there for a while and actually work on some real-world,(i.e. `production') equipment.

I went down with the crew and as an eager, young tech, was chomping at the bit, quite excited at the prospect of getting my hands on an actual piece of equipment that was passing LIVE military voice and data traffic.

To start me off slowly, they handed me a manual, turned to the preventative maintenance section and pointed me in the direction of four racks filled with several large digital multiplexers. The equipment was easy enough, we'd covered the same equipment in tech school.

First page of the manual read; "Apply power to the ditigal multiplexer. Turn both rear switches to the ON position and wait for the equipment to power-up, then begin tests." I looked up, and there was already power APPLIED!

I was in a quandary for sure. Not knowing how to proceed, I shot my best, `Ummmm.. Kinda lost here' look at the senior.

He looked over at me and laughed, "No, no, it's ok. You can ignore that part of the checklist." Then, as he noticed the look on my face, (since we were taught in school to NEVER, EVER ignore any part of a checklist, and it was certain death and destruction if one was to do so) he put a serious look on his face and said, "Ignore ONLY that part! Follow the rest of it, to the letter!"

Dutifully, I ran through the multi-step PM instructions, happy as a clam and proud that they were letting such a low-ranking, (albeit smart) tech do this important work.

Somewhere between the fifth and sixth preventative maintenance checklist on these huge multiplexers I started noticing an increased level of activity around me. Phones were ringing, people were moving quickly. Quizzical looks were being exchanged.

Finally, a group of folks ran up to me, headed by one of the senior techs who had brought me down.

"Hey! We're seeing HUGE outages in data traffic, and we've isolated/traced the path back to the racks that you're working on! Are you seeing any weird.."

(At that point he was cut off by another one of the troubleshooters who'd made her way around to the first group of multiplexers that I had been performing the PMs on.)

"HOLY NUTS! THEY'RE TURNED OFF! HE'S BEEN TURNING THEM OFF!!!!"

In short order, I watched as they hurriedly ran through the first step in the manual, "Turn both rear switches to the ON position..." When the senior tech was done, he came over to me and incredulously asked what I was thinking of, by turning the critical pieces of equipment off.

Scared out of my wits, I handed him the checklist that I'd been following, swearing that I hadn't deviated at ALL. That I had followed it, `to the letter' as he'd instructed.

After a while he laughed and pointed out where the problem lay.

In the manual, the FINAL step in the preventative maintenance checklist was:

"Record final probe reading, wipe down front panel, removing all dust and particulate, then turn both rear power switches to the OFF position."

:)

Answer 12

It's kind of a sysadmin accident.. in so far as sysadmins occasionally have to physically haul large numbers of machines from point A to point B (where A and B are seemingly always separated by several flights of stairs in a building with no lift). On the n'th trip of the day, I stopped for a breather three flights up from the basement loading level to chat with someone coming down, propped the full-size tower w/station I was schlepping on the inside handrail of the open stairwell and... well, you guessed... slightly lost my grip on it. It plunged unerringly straight down the well and when it reached the bottom, er... not so much with the functionality for that one! Total salvageable parts: two sticks of RAM, one floppy drive and one ISDN card (God bless the Hermstedt engineering folks!). Everything else either cracked, rattling or smashed into tiny pieces.

By the grace of God, nobody was walking underneath, which, thankfully for me, was my boss' first though, so I got to keep my job. Felt very sick for an hour or so though.

Moral: gravity always wins!

Answer 13

I was reloading a system for someone, and during the manual backup process I asked him the question "Do you have any other programs you use?" and "Is there anything else important you do on the computer?"

He said "no" SEVERAL times.

I was convinced and formatted the drive.

About 30 minutes later he said "oh my god" and put both hands on his head.

Turns out he had been working on a book script for over 10 YEARS in a specialized program. This was back when programs used to save user data in its program files directory and I missed it.

Whhhhooooops.

He wasn't mad at me, but it was a sobering feeling.

Answer 14

My personal favourite isn't actually mine, and I'm VERY glad of it. Take a look here.

Answer 15

This didn't happen to me, but…

I was working at a company that made software that ran on Linux machines provided by the client. We would essentially 'take over' the machines, completely configure them to our specs, and do all of the management and monitoring. Essentially, we were a team of 10-15 sysadmins, managing thousands of servers for hundreds of customers. Mistakes were bound to happen.

One of our team found some issues on a server (a backup, I believe), and decided that he should run fsck on it. He stopped all relevant services, made sure that the system had had backups taken recently, and then ran the fsck, but it complained that the filesystem was mounted. Since we were remote and had no remote access (DRAC, ILO, etc.), he couldn't do the fsck, but he was pretty sure that it was safe to do it with the filesystem mounted, if you were careful.

He decided to try it himself by running fsck on his root partition, with predictable results – he corrupted his root partition and couldn't boot anymore.

Confused, he went over and talked to our team lead. The lead said he was pretty sure that you couldn't do that, and the team member said 'Sure you can!', took the lead's keyboard, and showed him that you could – by running fsck on the lead's root partition. Which completely corrupted HIS root partition.

End result? No customer data lost, thanks to the team member's testing. Two days of employee productivity were lost, but that was worth far, far less than the data on the customer's machine. And for the record? You can run fsck on a mounted drive, but only to verify data. Not to repair it. That was the team member's mistake.

--

To add my own story, I was working at the same company, and was trying to reset a user password. Our system refused to let me set it to the password he needed, because it tracked old password hashes and refused to let you duplicate the password. The mechanism was simple: it validated your password against the most recent hash in the database.

(And for the record, it needed to be the old password because it was a shared account, and making sure everyone knew the new password was impractical)

I decided to just go into the users database and delete the new records so that it would use the older one. It's all just SQL (running an ancient version of Sybase), so it's easy. First, I had to find the records:

SELECT * FROM users_passwords WHERE username='someuser';

I found the old record he wanted to keep; there were two more in front of it. I decided to be clever and just delete anything newer than the old record. Looking at the result set, I saw that the old password was ID #28 in the database, and the new ones were ID #several thousand (very busy system). That's simple, all the old rows were > 28, so:

DELETE FROM users_passwords WHERE id > 28;

There's nothing worse than doing some simple row pruning and seeing '212,500 rows affected'. Fortunately, we had two master database servers (with the user ID), but Sybase (at least, our version) didn't support automatic replication, so it didn't automatically wipe out the old records. It was a trivial matter to get a dump of the users_passwords table and re-import it. Still, a pretty big 'oh f**k!' moment.

Answer 16

Another of my favorites:

When setting up a computer and a local laser printer on a system, I had the bright idea to plug them both into the computer's UPS. Ever try to print to a local laser printer when it's plugged into a desktop UPS? Well, if you don't know, it tends to pull all the amps... Which restarts the computer... And the print job never finishes...!

Ever get the call: 'Whenever I print, it restarts my computer and doesn't print!!!'?

Ooops!

JFV

Answer 17

DELETE statement without a WHERE clause, on the customers' live patron database.

Answer 18

Typed kill 1 as root. init and all of her children died. And all of their children. etc, etc. Oops.

What I meant to type was kill %1

After I realised what I did I ran to the control panel of a BIG wool bale sorting machine and hit the emergency stop button. This stopped the machine ripping itself to bits, as I had just killed the software which controlled it.

Answer 19

At some point in my career a legal investigation at the company I was working for placed a requirement on us that all email be kept from "this day" forward, until told otherwise. After about a year of storing daily full backups of our exchange environment (1TB nightly) we started to run out of space.

The exchange admins suggested that we only keep every 8th copy of the email. To do this, we had them restore a days worth of the exchange databases, extract the email they needed (specific people flagged for investigation) and re-archive it. They did this for every 8th day of email for all of our backups. The 8th day was chose because exchange had a parameter set where "deleted items" are kept in the database for 8 days.

After they would finish each archive, I would go back through and delete any backups which were older than what they had archived.

TSM does not have an easy way to do this, so you have to manually delete objects from the backup database.

I wrote a script which would delete all backups older than some date, by way of a date calculation using the difference between today, and the date in question. On some day I had to delete about a months worth of backups, except when I made the date calculation I made a typo and entered the date as 7/10/2007 instead of 6/10/2007, and ran the script. I deleted an entire extra month worth of data, accidentally which was part of a very important lawsuit.

After that, I added some steps to the script to confirm that you wanted to delete the data, and show you what it was going to delete...

Luckily, they never even used any of the data we worked so hard to preserve, and I still have my job.

Answer 20

After a long day or performance tracing and tuning a huge mainframe (you know the beasts that take a couple of hours before all standby backup-sites have agreed that it is indeed booted up again and fully synced) I stretched my fingers, typed satisfied shutdown -p now in my laptop prompt, closed the lid, yanked the serial cable out of the mainframe, with the anticipation of a nice cold glass of lager.

Suddenly I hear the deafening sound of spinning down mainframe while my laptop was still happily displaying X.

While waiting for the machine to come fully online again I decided that I got time to get my ACPI working on my laptop so I never ever are tempted to cli shutdown my laptop.

Answer 21

We were in the middle of a power outage and saw that the UPS was running at 112% of it's configured load. This wasn't much of an issue as we were running on the generator at the time.

So we went around pulling backup power cables to reduce the power usage on that UPS (we had two, one much larger than the other). We got to the network switch which ran the server room (this was the server room with all the internal servers for the company, with the customer facing servers in another server room). The switch was a large enterprise class switch with three power supplies in it. The supplies were N+1 so we only needed two in order to run the switch.

We picked a cable and pulled it out. Unfortunately for us the other two were plugged into a single power strip, which promptly blew as the load went up on the two power supplies which were plugged into it. The sysadmin then panicked and plugged the third cable in. The switch tried to fire up, putting the entire load of the switch unto the single power supply. Instead of the power supply shutting down, it exploded in a shower of sparks not 12 inches from my face sending me jumping back into the rack of servers.

Out of instinct I tried to jump to the side, but unfortunately on my left was a wall, and two my right was a very large 6'4" facilities guy. I some how managed to jump over him, or possibly through him bouncing off of the Compaq racks (the ones with the thin mesh fronts) without putting a whole in the rack, and without touching the facilities guy.

Answer 22

This accident didn't happen... but it's worth mentioning:

I was sent to a heavily-used data center to conduct bandwidth tests on a new circuit. I got to the demarc room/IDF, found a spot on one of the racks for my test router, made my connections, and started the tests. Unfortunately, I completely failed to notice the in-production border router not only being exactly on the next rack (almost at the same level), but that it was also the same make and model as my testing router.

When the test was done, I began pressing the power switch to the off position (...imagine it in slow motion...) and, I swear, just as I was applying pressure it dawned on me that the router I was about to turn off was the one in production. My heart stopped and I almost... well, use your imagination.

I left the data center's MDF looking spooked and pale, but at the same time glad I still had a job!

Answer 23

I deleted someone's account by mistake, got the names mixed up with the one I was suspose to delete. Opps

The cool part is they never knew what happened. Got the call they couldn't log in, the penny dropped about the account I deleted.

While on the phone with them, I quickly re-created their account, re-attached their old mailbox to it (thankfully Exchange doesn't delete mailboxes right away) and pointed it back to their old user files.

Then I blamed them for forgetting their password which I had just reset for them :)

Answer 24

Accidentally installed a tar.gz file on my Gentoo Linux box in the wrong place and it left files all over the place. This must've been around 1999, 19 at the time (thanks for the comments below)

Being the geek that I am, I decided to try to script myself out of the work of going manually through each file.

So I tried:

tar --list evilevilpackage.tar.gz | xargs rm -rf

It didn't take me very long to notice that tar also listed all the directories the program was using, those included were ''/usr, /var, /etc'' and a few others that I didn't really want gone.

CTRL-C! CTRL-C! CTRL-C! Too late! Everything gone, reinstall time. Fortunately the box didn't contain anything important.

Answer 25

As a smallish part of my former life I administered the company's file server, a netware 4:11 box. It hardly EVER needed any input at all, but if it did, you opened up a remote console window.

Used to using DOS all the time, when I was finished, I naturally would type "Exit". For Netware, "exit" is the command to shut down the OS. Luckily, it won't let you shut down unless you first "Down" the server.(Make it unavailable to the network/clients) So when you type "Exit" in the console, it helpfully says, "You must first type "Down" before you can exit"

Ask me how many times I 1: typed "exit" in the console session and 2: Obediently typed "Down" and then "Exit" so I could "finish what I was trying to do"

And then the phone starts ringing.....

LOL

Answer 26

Another story that didn't happen (phew):

We were doing incremental backups religiously every day to a tape drive.

We happened to write a tape containing data to ship to someone else. They said 'we can't read your tape'. In fact, neither could we. Or any tape in fact.

We bought another tape drive and held our breath until we installed it.

Moral of the story. Always make sure you test your backups.

Answer 27

The last place I worked, my co-worker had his kids with him in the server room (why? I have NO IDEA!).

He made sure that they were far away from the servers and explained to his 5-year-old that he shouldn't touch ANY of the servers and ESPECIALLY none of the power switches.

In fact, he had them right near the door... (can you see where this is going...?)

The boy didn't touch any of the server power buttons... No, that would be entirely too easy to explain. Instead he hit the BIG RED BUTTON that was near the door... The button that shuts down power to the ENTIRE SERVER ROOM!!!

Phone lines immediately started to light up wondering why Exchange, File Servers, etc. weren't available... Imagine trying to explain THAT to the CEO!

-JFV

Answer 28

I once had a fight with the APC UPS monitoring software. Being a small company, we had a couple of small-ish UPSes and various servers were setup to monitor them. Most of the servers were Linux, but a few were running Windows and so they were the ones used because the APC software is Windows only.

However, the APC software at the time was hard-coded to assume the UPS it is talking to is also powering the PC its running on! This was not the case for this server, but I discovered that too late to tell it to halt. Also unfortunately, the lead programmer was demonstrating the company product to a partner - it was a web-based app, running on the same server I didn't want the APC software to shut down...

Answer 29

I was giving a new sysadmin a tour of a Service Manager app. I said "if you ever needed to stop this service you would click this button, but you should never do it during the day." You would never believe how sensitive her mouse button was!

Two minutes later the service had started up again, and no-one seemed to notice.

Answer 30

Tripping over a tower server that was wedged behind a rack and hitting my head on the back of the main Cisco router on my way down. Thus revealing how loosely the power cords were actually seated in the power supplies on the front of the Catalyst 6500.

Yeah. We've got a hardhat on a hook in the server room now. With my name on it.