Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.

I'll add my own story as an answer.

share
3  
See also serverfault.com/questions/5066 –  Zoredache May 13 '09 at 18:26
14  
This really is more of a poll then a question. This should probably be set to community wiki. –  Zoredache May 13 '09 at 18:31
7  
Yup, this should definitely be a community wiki. In the intent of the question, though, my favorite story is the 500 mile email one - ibiblio.org/harris/500milemail.html - although, obviously, that wasn't me. –  Mihai Limbăşan May 13 '09 at 20:26
show 2 more comments

locked by sysadmin1138 Jan 6 '12 at 3:44

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as not constructive by Mark Henderson Jan 4 '12 at 6:29

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

86 Answers

Not me, but someone I work with. They created a policy on the AV server that contained a * in the process field. In layman's terms: do not allow read, write, execute to any process that contains the name *.

This policy then was replicated to 1,500 servers, which in turn shutdown RDP and any other process. To fix it meant to mount every server hard drive one by one and remove the policy. 48 hours with a team of 15.

share
add comment

This didn't happen to me, but I guess it's a really nice story.

These guys were working with one of those old Solaris full-tower servers which, as I am aware, were holding databases for several Informix database this company had. This was a basic-utility company so you can imagine how much data that means.

There was a point where several configurations through servers were copied on a floppy disk and then passed on from server to server. After working with a server, they would just eject the floppy disk and move on to the next one.

Accompanied by another person in the sysadmin group, this guy was working on these configurations as the they talked about random stuff. He finished his step so he pushed the button to eject the floppy.

-"WAIT! Don't release the button!"

When he looks again, he had hit the reset button on error and not the eject button. At the moment he released that button, the whole database system for the company would immediately power down. (I thought these buttons were instantaneous... but this is how the story goes.)

So, every sysadmin stops what he's doing to call department managers and "tell everyone to log off the system. Now." while this guy looks everything happening attached to a server by his finger.

share
add comment

Picture a cup of coffee. It's a full cup, with sugar. Picture it seriously misplaced on a rack's retractable keyboard tray. A rack full of servers. The tray gets somehow pushed into the rack. The cup enters the rack and then topples.

That was my fault, and I was a seasoned admin by then, so I have no excuses. There was a bathroom nearby and I was able to mop up most of the mess with paper towels. Luckily not enough coffee got inside the servers, so I shut them down and cleaned them good. Only 400 users affected. Phew!

Then there was another accident, let's call it so, that happened to a friend of mine. He has dedicated the past 10 years building his own company. He has ~15 employees, and all the company's data was in this one server. This included all past and present projects, lots of costumer data, information he had been contracted to keep safe, all contact information, etc. All nicely encrypted with LUKS. I had been pestering him for a long time to make him start doing backups, but he never did. Too busy, short of funds, you get the idea. He was confident his RAID1 would save him. His last backup was 8 months old. That was his server uptime too. He had changed his LUKS password right before the last reboot, 8 months before this. Now he rebooted his server and then realized he had not written the new password down, and he didn't remember it. All he could remember was that it was very long, and it had several words approximately arranged in some way with some sort of capitalization and possibly symbols thrown in.

You can imagine the degree of demoralization among his employees and the rage of costumers who had to resend their information for processing, thereby learning their data was "temporarily" unavailable. To make a long story short, it took me about 40 hours of work, 14 days of runtime and a specialized program to generate and test more than a million passwords to finally find his LUKS password.

share
show 2 more comments

End of week, everyone almost out of the building, I go into the server room to load new tapes into the autochanger, for the weekend-long full backup. The AC is too cold I think, and turn it off (the server room was just a room with a wall mounted AC - no funds for anything serious). So I load up the tapes, make sure the TBU read the barcodes OK, and head out.

The next day, I wake up in the morning, with a hangover (hey, it's weekend!), look at my phone and see a bunch of SMS messages "$server going down". Then another one "main UPS going down".

I grab the keys, drive to the offices, and open the server room, to find it's around 60c in there, and all the equipment is off.

Ended up dragging a few fans to drive the hot air out, before I even could start the AC working, not to mention the UPS and the 40+ servers and comms equipment. And spending the weekend in the office of course. And thanking all deities for smart UPS units that can pull everything down nicely if the ambient temp is too high. I always keep a hoodie around since, and never turn the AC off

share
add comment

Totally different dimension, but it is still a system administrator accident.

Sorry: You need to understand some Italian slang to get this. It can't be translated. You need to know it by heart

I was asked to fix something on a Solaris server in Napoli, Italy. I needed the root password, and I didn't speak much Italian at the time. The guys did seem reluctant to tell me what it was. Finally one of them half-whispered:

- sticazzi

I said: Aha, 'sticazzi'. How do you spell that?, and gave him a piece of paper + pen.

A year later I met M.*o B.* again (Hi! - if you read this). At the time my Italian was far better. I told him I now know some more italian.

That was a hard laugh.

The moral of the story: If need to ask for the root password in a language you do not know, once it's given to you better laugh, blush and look insulted at the same time.

share
add comment

Adding a bypass rule to a firewall in order to speed up some BitTorrent downloads. It turns out the system that the bypass rule used wasn't too stable, and it took down the firewall. This was a border firewall for every school's Internet connection in the city. To make matters worse, the reboot was just enough to cause the firewall's hard drive to die. Amusing? Not so much. Spectacular failure? Definitely.

share
show 1 more comment

I was reloading a system for someone, and during the manual backup process I asked him the question "Do you have any other programs you use?" and "Is there anything else important you do on the computer?"

He said "no" SEVERAL times.

I was convinced and formatted the drive.

About 30 minutes later he said "oh my god" and put both hands on his head.

Turns out he had been working on a book script for over 10 YEARS in a specialized program. This was back when programs used to save user data in its program files directory and I missed it.

Whhhhooooops.

He wasn't mad at me, but it was a sobering feeling.

share
7  
You've taught him a valuable lesson: Backups are important –  MikeyB Sep 10 '09 at 5:00
5  
Just reading this really hurts. So painful. Sure a lesson, but... –  mafutrct Jun 1 '10 at 10:36
show 4 more comments

I'm a programmer, so all of my mistakes belong on Stack Overflow. However, below are some of the system administrator errors I have witnessed.

  1. Revoke logon permissions from ALL users on a Windows NT domain. (Other than the builtin administrator on the PDC, sadly only the contractor that set the domain up knew the password, and they were long since gone) I don't actually know how this was achieved. I do know that I got to sit and chat with my fellow developers for a few hours.

  2. Accidentally delete the Member Servers OU. That was another few hours chatting while a restore from tape was done.

  3. Our admin intended to give all domain admins permission to use CD & floppy drive access. (We used SecureNT to control access to removable media at the time.) Sadly he got the group membership backwards and instead gave all users of removable media full domain administrator rights as well. I found this because some tables turned up in a production SQL database that had been created by a user that shouldn't have been able to. When I told the administrator in question I enjoyed watching his face change from, no, that's the right way round, down to, oh **. Thankfully there was no serious harm done.

share
add comment

I work for a wireless provider in North America, and had done some training for a person in my group to run through work orders. I had stayed up the first couple of nights (we do everything during the maintenance window), but he was doing fine and said he's got to learn it on his own, so I let him and left my cell phone and pager on. I logged in and checked the configuration when I got up at 8 a.m. the following morning.

The change was that we were adding a new pool of IP addresses for BlackBerrys, the pool we were adding was about 10000 addresses. To do this, we add routes on the router that point to the processor address on a blade that does all the call processing (essentially it works like a proxy). Also, we log into the processor and configure the IP pool, and link the IP pool to be used for our wireless users. However for testing, we normally configure this on one processor (actually boot up a phone and test all the features), and then just move the configuration to the actual processor we want it on.

Fast forward two weeks, and I get a call from our control center that there's been a lot of call in's about some intermittent BlackBerry problems, and the few BlackBerrys they've looked at seem to be cycling through a common pool, but weren't really sure what was happening. It only took me about 5 minutes to realize that this was the new pool my colleage had just added two weeks before. It also didn't take long to see that the router had two routes in it, one going to the test processor, and one going to the proper call processor. This being what it was, he forgot to delete the route to the test processor, and it superceded the proper route.

Essentially a BlackBerry would connect to the network, connect to the proxy to get its IP address, the proxy would give it an address from the pool with the incorrect route, and the BlackBerry would try and talk to the RIM relay, and the response would be routed to the test proxy and never make it back to the user, essentially meaning no connectivity.

We got lucky though since BlackBerrys have a behaviour that if they can't contact the relay, they will disconnect / reconnect to the network, but nonetheless some RIM devices were without service for up to several hours until they were able to cycle onto a working pool. I thought back, and when I double checked the work, I had only check the proxy configuration which was new to this guy, I never checked the routing configuration since this guy was previously with the backbone team and routing was his thing. Oops!

I fixed it and called him up that afternoon, his day was going well, but I started with I'm sorry, but I'm about to ruin you're entire week. A year later the story still comes up around beers.

share
add comment

Early on when I was a young one, I was trying to be 'helpful' and tried to copy 250 MB of data over a 128 kbit/s line to 86 different sites at the same time... during business hours. While I was doing this, I overheard people asking why everything was taking so long.

Needless to say, I killed the transfers, and (luckily) no one knew it was me!

share
add comment

I'm a bit of a novice/hobbiest sysadmin with only 30-40 sites hosted on my server so this wasn't too bad. I was removing execute permissions on all files in the directory /bin/xxx and they all started with .

So taking the obvious action, I ran

chmod -R a-x .*

Wow. When you remove execute permissions on your bin directory, it's quite a pain to cleanup. The data centre techs had to boot into a live CD to fix. The best part was I had to walk them through how to fix it. The worst part is they still knew enough to laugh at me :P

share
add comment

In the early days of the Internet I ran everything on SGI Challenge S servers. At one point, without my knowledge, the "art department" ordered a demo rendering print server from IKON. Walked in one morning, Challenge acting funny, admin calls into the server room, we go through routine diagnostics, etc finally I say it HAS TO BE the power supply. Of course we have no spare. I walk back into the main office - see the loaner machine and realize - it's also an SGI - open it, unscrew power supply, reboot server - bingo! We order a spare overnight, rep shows up in the AM to ask how we like the demo, we have to hummada hummada for 30 mins til FedEx shows up and we re-swap power supplies and roll the demo box out the door. All in a days work.

share
add comment

I had a good new one happen to me last week.

I had one of my guys build a temporary DNS server for a test platform we're building, I asked our DNS guys to update a particular test domain to point at this new temp DNS server but the guy updated the live record not the test one.

Suddenly this one server (fortunately a new box so a reasonable spec) serving just about every DNS request for nearly 5m users - 400 million requests on the first day! - fortunately the TTL was only 24 hours so it's mostly drained away now.

share
add comment

I used to look after a bunch of database servers, each with a well defined development and testing cycle. Our role was to roll the changes the developers supplied, using their documentation from their test environment into the customer's test environment for customer testing before going live. As part of that the customer test environment was built from the most recent backup of the live environment.

This was all neatly documented, along with the process for rolling the change into the live environment after the customer had signed off on the change.

We had a new start in our team and after he'd been with us for a couple of months we let him sit in on a number of change cycles until one fateful night we let him do it himself. The customer testing went smoothly and the customer happily signed off on the change.

The new start then did exactly what he'd done every time he'd rolled the change into the test environment, confident he didn't need to follow the documentation the rest of us did. Step (1), rebuild from previous backup...

The next morning the customer noticed that the previous day's work was missing and it didn't take us long to find out what had happened. Fortunately the databases had change logging enabled so we were able to recover all the activity. The new start did at least learn to value the documentation and follow it in future.

share
add comment

This happened when I had just started my first support job out of uni, I was connected in to a customer's 2003 server trying to get on to one of the user's machines after they had complained about connectivity problems.

Talked her through some basic troubleshooting and noticed she had a static IP so started talking her through setting this to DHCP. I opened up the properties on the LAN connection on the server to use while I talked her through what to do. After getting her to try and set it back to DHCP it still had a static IP so asked her to disable the connection and re-enable it.

Now by this point I was doing everything I was telling her to on the server without actually changing any settings, right up until the point I asked her to right click on the LAN connection and hit disable which I then proceeded to do too.

Took me maybe half a second to realise what I'd just done.

Took maybe 10 minutes for the other engineers to stop laughing at me before one of them had to go drive for an hour to re-enable the NIC at the customers site.

share
add comment

We had a bit of a mess up a few years back. Mid-morning, the users started reporting loads of errors about locking when accessing our SQL Server-hosted app. The app grinds completely to a halt - nobody can do anything. Rather than take the time to find out what's causing it, we do an emergency reboot and everything starts working again. Then I start nosing through the various logs to see what might have triggered it, and just before everything went belly-up I find an open named transaction against the main table without a corresponding COMMIT.

Turned out my colleague had written some SQL in Query Analyzer to correct some erroneous data in the main table, and he'd placed it inside a transaction. But, instead of just hitting F5 to run it, he'd highlighted the whole thing and then hit F5. Except he hadn't quite highlighted everything...he'd missed out the end where it actually COMMITTED the transaction...leaving the table locked.

share
add comment

We built turnkey IVR systems for clients on Unix boxes. One time the developers had all their code in /devel. They asked me to remove the development directories and box and take the servers to the airport on a Sunday afternoon (my day off!). In my hurry, I deleted /dev/*. Instantly saw my mistake, sat and pondered for a minute. Not sure if the system would die if the kernel had no hooks to system devices, so I looked at the /dev directory on an identical machine and in order did mknod [c|b] major minor to restore keyboard, tty, scsi drives, fd0 and null then made a floppy on the other machine /dev and mounted and copied it locally to get the rest.

Still no idea what would have happened if I left things alone, but I'm pretty sure it would have been unhappy on reboot :)

Lesson learned - development directory doesn't get to be called /devel.

share
add comment

On Linux and FreeBSD hostname -s will "Display the short host name. This is the host name cut at the first dot".

On Solaris 9, hostname -s will SET the hostname to be '-s'.

So, my fellow admin ran a script to audit all of our 120 systems, including 10 Mission Critical Oracle Database servers running on Solaris 9.

for HOST in `cat all-hosts`; do
ssh $HOST "hostname -s"
done

All of our Oracle servers failed instantly. The speed of this failure was really quite amazing, It took about 20 seconds for us to recover from this mistake, but it was already too late. Everything was down.

The irony is that our datacenter suffered from a major power failure just a few days earlier, and we were updating our "power down/power up" spreadsheet to ensure faster recovery for any future power failures.

share
show 2 more comments

Former employer story that's great. Some of the details are changed to protect the innocent. I had a problem employe, call him Fred, who had been having alot of productivity issues, but seemed to have redeemed himself and had earned back some privileges. Only problem was, when his privileges were restored, a bug in a provisioning script gave him some extra privileges.

I was in the middle of a big project, so I asked Fred to package up a Windows hotfix that was needed for an application. (This was in the pre-blaster days when people didn't patch as religiously as they do today). So Fred runs a test on out in our lab and everything works fine.

Fred then asks a couple of questions:

"Who should I push it to?" (Mind you, this is a patch for some custom VB app)

"Everyone", I respond

"Ok, what time should it start?"

"How about 2AM?", I answer. (Figuring I'd have time to look over everything before I left for the day!)

So what happens next? He setups up a job with our software distribution app to push to everyone, and is even kind enough to check the boxes for every platform that the product supports. Then, sets the start time for 2AM, as in the 2AM which took place about 12 hours in the past.

The result? Everything reboots and trys to install some VB5 runtime patch. At about 2:45 PM on a Friday afternoon. Everything.

Everything? Like 40,000 PCs? Yes. 3,000 Windows servers? Yes. 300 HP, Sun and IBM Unix boxes? Yes. An AS/400 cluster? Yes.

The only thing that didn't reboot were the Windows DCs, because the AD guys disabled our application for some reason. Holy nightmare. After a week of mopping up, I couldn't believe that I was still employed.

The punchline? Fred got a huge promotion into a job where he couldn't hurt anything anymore.

share
add comment

Thankfully I was able to easily recover from what I am about to share with you. So you have heard of the infamous

rm -rf /
under linux, right? Did you know that Windows has the same command? From the C:\ prompt, this is it:
deltree /y /s/b \
.

My problem was that I typed this in and knew it was wrong, so I went to hit the backspace key, but fat fingered it and hit the enter key instead! It took me literally only 2 seconds to realize what I had done so I furiously started pressing ctrl-c repeatedly to abort the operation. By the time I had stopped it, half of the file system was gone.

Backups to the rescue, my friends! Other than a reboot, there was no other down time. In once sense, I was really lucky that day because I had great backups in place.

share
add comment

In my early time of system administration I invented some new method of doing inventory process (stock taking) for our retail shops. I took a lot of laptops and connected barcode scanners to them and made the process ten times faster than usual as when we did it by writing all the articles with pen on pappier. I also bought some Symbol PDT DOS handheld terminals. To extend the lifetime of the batteries for Symbol terminals I made my own battery packs and connected wires manually. That night and the next morning I was so proud of myself and I was proud as a peacock walking around the office saying how smart I was.

The nightmare started when I was sending data up to the server to make a calculation and comparison of stock and lists. One of the Symbol devices with an extra battery pack had been flashed because one of wires had lapsed and the device left without energy for a long time.

Now all the work of around 100 employers fell into the water. What is the purpose of 13 or 15 devices and their list if I did not have all of them? How could I know what of inventory was missing.

To closer describe my disaster, we had only a few days off in the year. It is when we close our shops and make stock taking, and that event costs our company a lot of money and effort.

Lucky for me our director and chef of that retrial has been reasonable and accepted inventory lists as they were at computer for that year.

After that I always make two copies of data while work is still in progress and just after we finish inventory process and of course I do not brag anymore.

share
add comment

Very stupid mistake. I was writing a script on my Linux workstation that processed a number of files, but it didn't matter what kind of files it were, as long as it were a lot of files. So I decided it was a good idea to copy /etc to a directory I was doing my tests in. When things went wrong, I deleted the copy and copied /etc to my test directory again. That went well, for a some time, and then I typed

rm -rf /etc 

instead of

rm -rf etc/  

OK, nothing to worry about, I could still do things on my workstation and thought I could revive it by copying it from another workstation, or something. Or, reinstall at the end of the day. First, get something to drink, and because of the corporate policy, I locked my screen. Damn, I need my password to unlock and that's in /etc/.....

Stupid mistakes:

  • doing too much a root (I had a good reason for it O:)).
  • typing /etc instead of etc/
  • using /etc for testing purposes
share
2  
next time - pick something like /tmp :) –  warren Sep 29 '10 at 2:05
add comment

Typed kill 1 as root. init and all of her children died. And all of their children. etc, etc. Oops.

What I meant to type was kill %1

After I realised what I did I ran to the control panel of a BIG wool bale sorting machine and hit the emergency stop button. This stopped the machine ripping itself to bits, as I had just killed the software which controlled it.

share
1  
That one made me laugh! Having worked with large machinery in the past I can appericate control systems. –  SpaceManSpiff Jun 21 '09 at 0:12
1  
On BSD, kill -1 1 caused init to reread inittab or /etc/ttys. Leaving off the "minus" had consequences... –  kmarsh Jul 15 '09 at 17:53
2  
Thank goodness for that emergency stop button! :-) –  staticsan Aug 24 '09 at 23:23
show 1 more comment

When I was first hired as sysadmin by the lead admin...within the first week we received a brand new Dell server...Windows Server 2003...it was his little baby until I was secretly called to the server room at midnight one Saturday night to clean numerous instances of malware from it because he was SURFING THE WEB with it before deployment WITHOUT ANTIVIRUS!!!

Malware cleaning is something that I have had much experience with, but since this was a server I did a format and reinstall to be extra safe.

I never said a word to him about it. He knew he had messed up royally.

share
add comment

During maintenance at a co-location I pulled our primary DNS power cable. I was replacing the secondary at the time and must have yanked the cable before I closed the rack. All of our sites started dropping fast and I had to go back to the co-location to plug the stupid thing back in.

share
add comment

Oh, one day I deleted a PostgreSQL database inadvertently and recovered it from log files ;)

share
add comment

While setting up a static IP address in /etc/network/interfaces on a Debian box, somebody accidently switched the IP addresses on the IP address line and the gateway line.

Guess what happens when you "steal" the IP of the core switch?

share
add comment

Maybe more of a late night brain fart than anything else.

One of the developers was having trouble with running a Java profiler on a Solaris box. The profiler was complaining that there were two copies of Libc; one in /lib and one in /usr/lib. So after a few lds we moved the one from /lib as everything was pointing to /usr/lib, or so they said.

But suddenly nothing worked. No ls, no cd, no cp or mv. After about 20 minutes of 'oh crap, oh crap' we figured out that one of the developers had a currently running copy of Emacs on that box and we were able to open the backed up /lib copy of Libc and write it back out with the original name. And voila! Everything worked. Lesson learned; leave Libc where it wants to be and don't make changes on developer requests at 2 A.M.!

share
add comment

More of a personal scripting thing than a system administration thing, but...

I was writing a Perl script to act like a macro that would retrieve now playing information from Banshee and enter it character by character as keyboard events using the program "xte". This way, I could have it work within programs without any special interaction, it would be just like I typed it.

Well, I coded the thing almost perfectly. I decided to test it out in some random game. The keypress to bring up the chat was shift + enter. Now in order to do this I needed to have it hold down shift, press enter, then release shift. Unfortunately in my haste I forgot "release shift". I ran the script and this led to the somewhat hilarious side effect of my shift key being locked down. I thought "no problem, I'll just go to the terminal and manually type in the line to release shift". Unfortunately, as everyone knows, Linux is case sensitive. It would not accept the command in all caps as I had to enter it. I couldn't "counter-shift" or anything like that.

This led to a five minute scavenger hunt of me visiting websites and using the mouse to copy+paste individual lowercase letters into the terminal to form the command I needed to turn it off.

share
show 2 more comments

A couple of companies ago we had a Windows NT 4 box as the main server running everything, as a backup it had a mirrored hard drive.

I accidentally deleted a few important files, no problem just restart the box, select disk 2 from the SCSI menu and we are back up and running on the copy in under a minute.

Then I started the command to rebuild the mirror drive. It turns out that although Windows now had new C: and D: drives the clever mirroring software wasn't going to be fooled by that. It used the SCSI ID numbers for the source and target, and happily copied 1->2.

Thank you Adaptec!

share
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.