Best system administrator accident [closed]

Question

I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.

I'll add my own story as an answer.

Answer 1

Former employer story that's great. Some of the details are changed to protect the innocent. I had a problem employe, call him Fred, who had been having alot of productivity issues, but seemed to have redeemed himself and had earned back some privileges. Only problem was, when his privileges were restored, a bug in a provisioning script gave him some extra privileges.

I was in the middle of a big project, so I asked Fred to package up a Windows hotfix that was needed for an application. (This was in the pre-blaster days when people didn't patch as religiously as they do today). So Fred runs a test on out in our lab and everything works fine.

Fred then asks a couple of questions:

"Who should I push it to?" (Mind you, this is a patch for some custom VB app)

"Everyone", I respond

"Ok, what time should it start?"

"How about 2AM?", I answer. (Figuring I'd have time to look over everything before I left for the day!)

So what happens next? He setups up a job with our software distribution app to push to everyone, and is even kind enough to check the boxes for every platform that the product supports. Then, sets the start time for 2AM, as in the 2AM which took place about 12 hours in the past.

The result? Everything reboots and trys to install some VB5 runtime patch. At about 2:45 PM on a Friday afternoon. Everything.

Everything? Like 40,000 PCs? Yes. 3,000 Windows servers? Yes. 300 HP, Sun and IBM Unix boxes? Yes. An AS/400 cluster? Yes.

The only thing that didn't reboot were the Windows DCs, because the AD guys disabled our application for some reason. Holy nightmare. After a week of mopping up, I couldn't believe that I was still employed.

The punchline? Fred got a huge promotion into a job where he couldn't hurt anything anymore.

Answer 2

Everyone 'rm -rf /'s at some point accidentally. Mine was trying to delete some of the extra files in my home directory 2 days before my last data structures assignment was due.

Professionally I've been capable enough to not have any catastrophic screw ups so far.

Answer 3

There was the time I accidentally deleted the "bin" user on a Unix box. Of course, deleting a user causes its home directory to be removed, as well.

Can you guess what bin's home directory is?

/bin

Answer 4

Ten, plus years ago I was working on a project that required a SOCKS proxy. I had been using a program called WinGate that in addition to SOCKS proxy, provided a nice little Internet gateway functionality with NAT, DHCP and a few other niceties. This was before Windows had Internet Connection sharing, so WinGate let you share your dial-up modem with your Ethernet network.

I installed the software and started work on the SOCKS client functionality. Later that day, we lost internet connectivity. All of a sudden, it just stopped and nobody could access outside the company. We called our ISP and everything looked fine on the connection. The router was working fine. We just couldn't figure out what went wrong. I pitched in at one point as I had some knowledge of TCP/IP, but I didn't make any headway.

The next day our IT guy figured out that the DHCP server had given the address of the router out to someone's machine, and everyone was using it for the default gateway which didn't go anywhere. Later that day our IT guy came into my office and I asked, "So did you figure out who gave out the wrong IP address?" He said, "Yeah, it's you!"

WinGate had defaulted to running a DHCP server and had given out the router address to the first client whose previous address had expired. I was pretty red-faced for a while.

Answer 5

A couple of companies ago we had a Windows NT 4 box as the main server running everything, as a backup it had a mirrored hard drive.

I accidentally deleted a few important files, no problem just restart the box, select disk 2 from the SCSI menu and we are back up and running on the copy in under a minute.

Then I started the command to rebuild the mirror drive. It turns out that although Windows now had new C: and D: drives the clever mirroring software wasn't going to be fooled by that. It used the SCSI ID numbers for the source and target, and happily copied 1->2.

Thank you Adaptec!

Answer 6

While setting up a static IP address in /etc/network/interfaces on a Debian box, somebody accidently switched the IP addresses on the IP address line and the gateway line.

Guess what happens when you "steal" the IP of the core switch?

Answer 7

We built turnkey IVR systems for clients on Unix boxes. One time the developers had all their code in /devel. They asked me to remove the development directories and box and take the servers to the airport on a Sunday afternoon (my day off!). In my hurry, I deleted /dev/*. Instantly saw my mistake, sat and pondered for a minute. Not sure if the system would die if the kernel had no hooks to system devices, so I looked at the /dev directory on an identical machine and in order did mknod [c|b] major minor to restore keyboard, tty, scsi drives, fd0 and null then made a floppy on the other machine /dev and mounted and copied it locally to get the rest.

Still no idea what would have happened if I left things alone, but I'm pretty sure it would have been unhappy on reboot :)

Lesson learned - development directory doesn't get to be called /devel.

Answer 8

This happened when I had just started my first support job out of uni, I was connected in to a customer's 2003 server trying to get on to one of the user's machines after they had complained about connectivity problems.

Talked her through some basic troubleshooting and noticed she had a static IP so started talking her through setting this to DHCP. I opened up the properties on the LAN connection on the server to use while I talked her through what to do. After getting her to try and set it back to DHCP it still had a static IP so asked her to disable the connection and re-enable it.

Now by this point I was doing everything I was telling her to on the server without actually changing any settings, right up until the point I asked her to right click on the LAN connection and hit disable which I then proceeded to do too.

Took me maybe half a second to realise what I'd just done.

Took maybe 10 minutes for the other engineers to stop laughing at me before one of them had to go drive for an hour to re-enable the NIC at the customers site.

Answer 9

I used to look after a bunch of database servers, each with a well defined development and testing cycle. Our role was to roll the changes the developers supplied, using their documentation from their test environment into the customer's test environment for customer testing before going live. As part of that the customer test environment was built from the most recent backup of the live environment.

This was all neatly documented, along with the process for rolling the change into the live environment after the customer had signed off on the change.

We had a new start in our team and after he'd been with us for a couple of months we let him sit in on a number of change cycles until one fateful night we let him do it himself. The customer testing went smoothly and the customer happily signed off on the change.

The new start then did exactly what he'd done every time he'd rolled the change into the test environment, confident he didn't need to follow the documentation the rest of us did. Step (1), rebuild from previous backup...

The next morning the customer noticed that the previous day's work was missing and it didn't take us long to find out what had happened. Fortunately the databases had change logging enabled so we were able to recover all the activity. The new start did at least learn to value the documentation and follow it in future.

Answer 10

I had a good new one happen to me last week.

I had one of my guys build a temporary DNS server for a test platform we're building, I asked our DNS guys to update a particular test domain to point at this new temp DNS server but the guy updated the live record not the test one.

Suddenly this one server (fortunately a new box so a reasonable spec) serving just about every DNS request for nearly 5m users - 400 million requests on the first day! - fortunately the TTL was only 24 hours so it's mostly drained away now.

Answer 11

Early on when I was a young one, I was trying to be 'helpful' and tried to copy 250 MB of data over a 128 kbit/s line to 86 different sites at the same time... during business hours. While I was doing this, I overheard people asking why everything was taking so long.

Needless to say, I killed the transfers, and (luckily) no one knew it was me!

Answer 12

Totally different dimension, but it is still a system administrator accident.

Sorry: You need to understand some Italian slang to get this. It can't be translated. You need to know it by heart

I was asked to fix something on a Solaris server in Napoli, Italy. I needed the root password, and I didn't speak much Italian at the time. The guys did seem reluctant to tell me what it was. Finally one of them half-whispered:

- sticazzi

I said: Aha, 'sticazzi'. How do you spell that?, and gave him a piece of paper + pen.

A year later I met M.*o B.* again (Hi! - if you read this). At the time my Italian was far better. I told him I now know some more italian.

That was a hard laugh.

The moral of the story: If need to ask for the root password in a language you do not know, once it's given to you better laugh, blush and look insulted at the same time.

Answer 13

End of week, everyone almost out of the building, I go into the server room to load new tapes into the autochanger, for the weekend-long full backup. The AC is too cold I think, and turn it off (the server room was just a room with a wall mounted AC - no funds for anything serious). So I load up the tapes, make sure the TBU read the barcodes OK, and head out.

The next day, I wake up in the morning, with a hangover (hey, it's weekend!), look at my phone and see a bunch of SMS messages "$server going down". Then another one "main UPS going down".

I grab the keys, drive to the offices, and open the server room, to find it's around 60c in there, and all the equipment is off.

Ended up dragging a few fans to drive the hot air out, before I even could start the AC working, not to mention the UPS and the 40+ servers and comms equipment. And spending the weekend in the office of course. And thanking all deities for smart UPS units that can pull everything down nicely if the ambient temp is too high. I always keep a hoodie around since, and never turn the AC off

Answer 14

This didn't happen to me, but I guess it's a really nice story.

These guys were working with one of those old Solaris full-tower servers which, as I am aware, were holding databases for several Informix database this company had. This was a basic-utility company so you can imagine how much data that means.

There was a point where several configurations through servers were copied on a floppy disk and then passed on from server to server. After working with a server, they would just eject the floppy disk and move on to the next one.

Accompanied by another person in the sysadmin group, this guy was working on these configurations as the they talked about random stuff. He finished his step so he pushed the button to eject the floppy.

-"WAIT! Don't release the button!"

When he looks again, he had hit the reset button on error and not the eject button. At the moment he released that button, the whole database system for the company would immediately power down. (I thought these buttons were instantaneous... but this is how the story goes.)

So, every sysadmin stops what he's doing to call department managers and "tell everyone to log off the system. Now." while this guy looks everything happening attached to a server by his finger.

Answer 15

A long time ago, I decided to change the mount point of my data partition. So I created a new directory, changed the mount point in /etc/fstab, and deleted the directory it was previously mounted on.

The thing is I only realized that the partitions was still mounted on the old directory when nautilus showed me a progress bar (for what should be a 4Kb deletion). Thankfully I was able to cancel it before a great damage was done, but I did lose some files.

Answer 16

On my first installation task (many years ago, in DOS age) I accidentally delete almost all system files and half application files on computer which belongs to director of public institution. But it wasn't my fault. I try to delete non important files in C:/TEMP folder to free some space. Delete begins...after a few moments I see some familiar names from root and DOS folder scrolling up on screen...Hitting hard Ctrl+Break...but too late...

That was the harder way to learn what cross linked files problem on FAT file system is.

Answer 17

We have a cold-testing facility for our engineers in northern Minnesota. About 10 years ago the T1 we had up there went dead. We had moved servers down from that facility to our main datacenter because we had installed the faster line so just about everything was useless up there. Come to find out that some farmer in central Minnesota had run through the fiber with some piece of farm equipment. We were none too happy that the fiber was even accessible to that piece of equipment and not buried much deeper...

Answer 18

Several years back, our iSeries administrator at the time was doing some cleanup in the area where our IBM iSeries servers where sitting in the computer room. This was around 8:30 in the morning. Just as I started to get going with whatever I was working on at the time. The screen went blank a few seconds later the phone calls started coming in.

Come to find out, when he moved a table the power cord was wrapped around the leg just enough that it came out when he moved the table.

About two hours later after the system recovered itself from the power down people were able to work again.

Answer 19

Oh, one day I deleted a PostgreSQL database inadvertently and recovered it from log files ;)

Answer 20

During maintenance at a co-location I pulled our primary DNS power cable. I was replacing the secondary at the time and must have yanked the cable before I closed the rack. All of our sites started dropping fast and I had to go back to the co-location to plug the stupid thing back in.

Answer 21

In my early time of system administration I invented some new method of doing inventory process (stock taking) for our retail shops. I took a lot of laptops and connected barcode scanners to them and made the process ten times faster than usual as when we did it by writing all the articles with pen on pappier. I also bought some Symbol PDT DOS handheld terminals. To extend the lifetime of the batteries for Symbol terminals I made my own battery packs and connected wires manually. That night and the next morning I was so proud of myself and I was proud as a peacock walking around the office saying how smart I was.

The nightmare started when I was sending data up to the server to make a calculation and comparison of stock and lists. One of the Symbol devices with an extra battery pack had been flashed because one of wires had lapsed and the device left without energy for a long time.

Now all the work of around 100 employers fell into the water. What is the purpose of 13 or 15 devices and their list if I did not have all of them? How could I know what of inventory was missing.

To closer describe my disaster, we had only a few days off in the year. It is when we close our shops and make stock taking, and that event costs our company a lot of money and effort.

Lucky for me our director and chef of that retrial has been reasonable and accepted inventory lists as they were at computer for that year.

After that I always make two copies of data while work is still in progress and just after we finish inventory process and of course I do not brag anymore.

Answer 22

Thankfully I was able to easily recover from what I am about to share with you. So you have heard of the infamous

rm -rf /

under linux, right? Did you know that Windows has the same command? From the C:\ prompt, this is it:

deltree /y /s/b \

.

My problem was that I typed this in and knew it was wrong, so I went to hit the backspace key, but fat fingered it and hit the enter key instead! It took me literally only 2 seconds to realize what I had done so I furiously started pressing ctrl-c repeatedly to abort the operation. By the time I had stopped it, half of the file system was gone.

Backups to the rescue, my friends! Other than a reboot, there was no other down time. In once sense, I was really lucky that day because I had great backups in place.

Answer 23

We had a bit of a mess up a few years back. Mid-morning, the users started reporting loads of errors about locking when accessing our SQL Server-hosted app. The app grinds completely to a halt - nobody can do anything. Rather than take the time to find out what's causing it, we do an emergency reboot and everything starts working again. Then I start nosing through the various logs to see what might have triggered it, and just before everything went belly-up I find an open named transaction against the main table without a corresponding COMMIT.

Turned out my colleague had written some SQL in Query Analyzer to correct some erroneous data in the main table, and he'd placed it inside a transaction. But, instead of just hitting F5 to run it, he'd highlighted the whole thing and then hit F5. Except he hadn't quite highlighted everything...he'd missed out the end where it actually COMMITTED the transaction...leaving the table locked.

Answer 24

In the early days of the Internet I ran everything on SGI Challenge S servers. At one point, without my knowledge, the "art department" ordered a demo rendering print server from IKON. Walked in one morning, Challenge acting funny, admin calls into the server room, we go through routine diagnostics, etc finally I say it HAS TO BE the power supply. Of course we have no spare. I walk back into the main office - see the loaner machine and realize - it's also an SGI - open it, unscrew power supply, reboot server - bingo! We order a spare overnight, rep shows up in the AM to ask how we like the demo, we have to hummada hummada for 30 mins til FedEx shows up and we re-swap power supplies and roll the demo box out the door. All in a days work.

Answer 25

I'm a bit of a novice/hobbiest sysadmin with only 30-40 sites hosted on my server so this wasn't too bad. I was removing execute permissions on all files in the directory /bin/xxx and they all started with .

So taking the obvious action, I ran

chmod -R a-x .*

Wow. When you remove execute permissions on your bin directory, it's quite a pain to cleanup. The data centre techs had to boot into a live CD to fix. The best part was I had to walk them through how to fix it. The worst part is they still knew enough to laugh at me :P

Answer 26

Picture a cup of coffee. It's a full cup, with sugar. Picture it seriously misplaced on a rack's retractable keyboard tray. A rack full of servers. The tray gets somehow pushed into the rack. The cup enters the rack and then topples.

That was my fault, and I was a seasoned admin by then, so I have no excuses. There was a bathroom nearby and I was able to mop up most of the mess with paper towels. Luckily not enough coffee got inside the servers, so I shut them down and cleaned them good. Only 400 users affected. Phew!

Then there was another accident, let's call it so, that happened to a friend of mine. He has dedicated the past 10 years building his own company. He has ~15 employees, and all the company's data was in this one server. This included all past and present projects, lots of costumer data, information he had been contracted to keep safe, all contact information, etc. All nicely encrypted with LUKS. I had been pestering him for a long time to make him start doing backups, but he never did. Too busy, short of funds, you get the idea. He was confident his RAID1 would save him. His last backup was 8 months old. That was his server uptime too. He had changed his LUKS password right before the last reboot, 8 months before this. Now he rebooted his server and then realized he had not written the new password down, and he didn't remember it. All he could remember was that it was very long, and it had several words approximately arranged in some way with some sort of capitalization and possibly symbols thrown in.

You can imagine the degree of demoralization among his employees and the rage of costumers who had to resend their information for processing, thereby learning their data was "temporarily" unavailable. To make a long story short, it took me about 40 hours of work, 14 days of runtime and a specialized program to generate and test more than a million passwords to finally find his LUKS password.