I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.
I'll add my own story as an answer.
Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.Sign up to join this community
I'm looking for amusing stories of system administrator accidents you have had. Deleting the CEO's email, formatting the wrong hard drive, etc.
I'll add my own story as an answer.
I work for a wireless provider in North America, and had done some training for a person in my group to run through work orders. I had stayed up the first couple of nights (we do everything during the maintenance window), but he was doing fine and said he's got to learn it on his own, so I let him and left my cell phone and pager on. I logged in and checked the configuration when I got up at 8 a.m. the following morning.
The change was that we were adding a new pool of IP addresses for BlackBerrys, the pool we were adding was about 10000 addresses. To do this, we add routes on the router that point to the processor address on a blade that does all the call processing (essentially it works like a proxy). Also, we log into the processor and configure the IP pool, and link the IP pool to be used for our wireless users. However for testing, we normally configure this on one processor (actually boot up a phone and test all the features), and then just move the configuration to the actual processor we want it on.
Fast forward two weeks, and I get a call from our control center that there's been a lot of call in's about some intermittent BlackBerry problems, and the few BlackBerrys they've looked at seem to be cycling through a common pool, but weren't really sure what was happening. It only took me about 5 minutes to realize that this was the new pool my colleage had just added two weeks before. It also didn't take long to see that the router had two routes in it, one going to the test processor, and one going to the proper call processor. This being what it was, he forgot to delete the route to the test processor, and it superceded the proper route.
Essentially a BlackBerry would connect to the network, connect to the proxy to get its IP address, the proxy would give it an address from the pool with the incorrect route, and the BlackBerry would try and talk to the RIM relay, and the response would be routed to the test proxy and never make it back to the user, essentially meaning no connectivity.
We got lucky though since BlackBerrys have a behaviour that if they can't contact the relay, they will disconnect / reconnect to the network, but nonetheless some RIM devices were without service for up to several hours until they were able to cycle onto a working pool. I thought back, and when I double checked the work, I had only check the proxy configuration which was new to this guy, I never checked the routing configuration since this guy was previously with the backbone team and routing was his thing. Oops!
I fixed it and called him up that afternoon, his day was going well, but I started with I'm sorry, but I'm about to ruin you're entire week. A year later the story still comes up around beers.
Back in the day, when I was very green, I needed to install AV software on my users PC's, as no-one seemed to have it. So I spent a bit of time figuring out how to do a remote install, rather than poking around 40 or 50 desktops. The remote installation ran perfectly and everything seemed fine, until various managers dropped by my office to complain that they couldn't log in.
It turned out that a few individuals had Symantec AV installed on their machines, and this did not coexist at all well with the McAfee software I was using and would lock up the machines after a login attempt.
Fortunately, it was possible to remotely disable the service if you got to the machine before they tried to log in, so I managed to get points for fixing it instead of having to rebuild all of senior managements PC's...
My aunt asked me to fix their computer. They said it wouldn't boot up and its been like that for 2 weeks. I suspected it was either the BIOS or the OS.
I sat down in front of their computer. I crouched down to push the power button. I look up.
The BIOS passed. That's good.
The OS booted. That's good.
I moved the mouse around thinking maybe there's a problem with the input devices. There was no problem with the input devices.
I opened up her word processor. It ran.
I print test the printer. It printed.
By this point, I stood up and told my aunt (who was watching me) that there is nothing wrong with the computer. She claimed that it wasn't like that before I sat down.
I can now claim to my family that I am so good, that I can fix any computer just by sitting in front of it.
Not a huge problem, but certainly an 'egg on my face' morning about 10 years ago. I had been going through the old hardware inventory and re-imaging the disks ready for the hardware to be offloaded. Trying to find the most efficient way possible to do this, I had built a CDRom with a copy of Norton Ghost and the image to apply. You powered on the machine, and while it was POSTing, put the CD in the drive. The machine would boot off the CD and re-image itself automatically. Worked well.
The problem came when I had been making copies of the CD so I could get more machines going in parallel. I finished burning the last CD, switched off my desktop computer and went home for the day. Well you can guess what happened the next morning. I came in, switched on my PC and went and made a coffee...
When I came back for some reason my machine was off the domain and not accepting my password...
I had just worked out what had happened and started swearing when the other guys arrived for the day. Yep, they didn't let me live that one down for a while.
More of a personal scripting thing than a system administration thing, but...
I was writing a Perl script to act like a macro that would retrieve now playing information from Banshee and enter it character by character as keyboard events using the program "xte". This way, I could have it work within programs without any special interaction, it would be just like I typed it.
Well, I coded the thing almost perfectly. I decided to test it out in some random game. The keypress to bring up the chat was shift + enter. Now in order to do this I needed to have it hold down shift, press enter, then release shift. Unfortunately in my haste I forgot "release shift". I ran the script and this led to the somewhat hilarious side effect of my shift key being locked down. I thought "no problem, I'll just go to the terminal and manually type in the line to release shift". Unfortunately, as everyone knows, Linux is case sensitive. It would not accept the command in all caps as I had to enter it. I couldn't "counter-shift" or anything like that.
This led to a five minute scavenger hunt of me visiting websites and using the mouse to copy+paste individual lowercase letters into the terminal to form the command I needed to turn it off.
When I was first hired as sysadmin by the lead admin...within the first week we received a brand new Dell server...Windows Server 2003...it was his little baby until I was secretly called to the server room at midnight one Saturday night to clean numerous instances of malware from it because he was SURFING THE WEB with it before deployment WITHOUT ANTIVIRUS!!!
Malware cleaning is something that I have had much experience with, but since this was a server I did a format and reinstall to be extra safe.
I never said a word to him about it. He knew he had messed up royally.
Longer ago than I'd like to think, I was the company's technical person and worked with some consultants installing their application. The hardware was a DEC VAX and used an HSC50 storage server. The consultants took much of the day with their install, and after they left, I decided to back up the system disk to an empty disk using the HSC50's bit-for-bit copy utility. After the copy was done and I tried to reboot, I discovered that I had reversed the names of the source and target disk, and so had backed up the blank disc bit-for-bit to the system disk.
I was able to rebuild VMS on the system disk, and reinstall much of the application, but I think it never worked as well. Since then, if I was doing a copy/backup/etc., I would write-protect the source disk before continuing. (Now that write-protect switches are no more, I look at the command before I hit Return.)
Done by one of my employee's... Perfect example of why you clearly label your servers:
Sent my employee out to the colo to rebuild the secondary MSSQL database server (which had no current data on it). Primary one was actively in use. You can probably predict the rest of this story... Once there, he rebooted the server, started the install and reformatted the drives, only to have me call him and ask him why the primary database server was no longer responding. (doh)
Mine happened just 6 months ago. We had just switched to a new server for a PHP/MySQL web application. Since I got to choose the OS, I chose the one I'm most familiar/comfortable with: Ubuntu.
We had a number of backup scripts that would be run by cron hourly, daily, etc. The transition went perfectly. There were only about 2 minutes of down time while I transferred the MySQL DB from the old server to the new one and switched IPs.
A few weeks later however, I was working in MySQL at the command line and was deleting some old test records that were no longer needed. Since I'm a programmer first, sysadmin second, I've gotten into the habit of typing my semi-colon (;) first and then typing in the command. Well, as I was about to add the WHERE clause to my DELETE query, I accidentally hit the enter key. ...oops.
Query OK, 649 rows affected (0.00 sec)
"No big deal," I thought. "The hourly backup just finished 4 minutes ago. There might be 3 records lost in all. I quickly went to the backup directory and restored. Problem solved.
...Then I noticed the timestamp on the backup. It was 17 days old. There were no other backups. I had just wiped out everything in the system entered less than 17 days before.
It turns out there's a bug in Ubuntu's cron daemon that causes it not to run a script file with a dot (.) anywhere in the name. It doesn't raise an error, so there's no evidence of a problem. It just refuses to run it. All of our backup scripts had dots in their names. They worked perfectly before, but not now.
Lessons I've learned:
I got called to investigate an alert coming from a Windows machine that was indicating that the monitoring system had no license file. I opened up the command prompt and started to investigate the problem and found that the basic windows commands were not even there.
A sysadmin who had run a script remotely had written a script which used the del command to delete a folder specified by a root and subfolder with the folders specified in Environment Variables. If the Environment Variables were not set, it silently deleted the whole partition.
When told, the sysadmin was so surprised that they confirmed the action by running the said script on their own notebook, thus trashing it too.
The amazing thing was that Windows was running fine, until we rebooted the server. Only the stingy monitoring software complained.
It was the secondary Active Directory server for a political party. Oops.
Ok. To get
& on a US keyboard, press Shift-7. To get it on a Swedish keyboard, press Shift-6. So, what do you get when you press Shift-7 on a Swedish keyboard? You get
Years ago Swedish layouts were not that common. My personal preference was to use the US layout. One day I wanted to delete a bunch of files and subdirs in a directory.
rm -fr *
But is was too slow, so I quickly hit:
Ctrl-C rm -fr * &
Or did I? Well I did not. It took me a few seconds to realize I was on a Swedish keyboard. See above to decode what happened. And that disaster was a fact.
That was the day when I learned the command:
I managed to get basically eventually from the disk to tape, only that it took all night. Next day I learned that the system was about to be reinstalled anyway.
I was lucky, but I learned a few things.
Mine was a tag team effort.
I was instructed by management to log one of our DBA's into a server so he could do some sort of cleanup. He ran his query and immediately both our pagers went off, which prompted expletives from both of us.
As it turns out, the cleanup was actually a drop of the database, and was supposed to be done on one of the development servers. However, the instructions that I received led me to believe this was a minor cleanup task that was supposed to occur in production.
Fortunately, we were able to restore from backup with minimal data loss.
Lesson learned: Make sure you ALWAYS know EXACTLY what you're supposed to be doing when messing with production servers. If there's uncertainty, it's best you get clairification.
When most of the server fleet was still Windows NT, the primary remote method in use was pcAnywhere. We had a "well-known" bug, that sometimes the servers would suddenly restart when using pcAnywhere, and end-users were told about this well-known bug.
The bug was that pcAnywhere (at least whichever version we were using) had a "reboot host" button next to the "disconnect from host" button. So every now and then... :D
Adding a bypass rule to a firewall in order to speed up some BitTorrent downloads. It turns out the system that the bypass rule used wasn't too stable, and it took down the firewall. This was a border firewall for every school's Internet connection in the city. To make matters worse, the reboot was just enough to cause the firewall's hard drive to die. Amusing? Not so much. Spectacular failure? Definitely.
On Linux and FreeBSD
hostname -s will "Display the short host name. This is the host name cut at the first dot".
On Solaris 9,
hostname -s will SET the hostname to be '-s'.
So, my fellow admin ran a script to audit all of our 120 systems, including 10 Mission Critical Oracle Database servers running on Solaris 9.
for HOST in `cat all-hosts`; do ssh $HOST "hostname -s" done
All of our Oracle servers failed instantly. The speed of this failure was really quite amazing, It took about 20 seconds for us to recover from this mistake, but it was already too late. Everything was down.
The irony is that our datacenter suffered from a major power failure just a few days earlier, and we were updating our "power down/power up" spreadsheet to ensure faster recovery for any future power failures.
I'm a programmer, so all of my mistakes belong on Stack Overflow. However, below are some of the system administrator errors I have witnessed.
Revoke logon permissions from ALL users on a Windows NT domain. (Other than the builtin administrator on the PDC, sadly only the contractor that set the domain up knew the password, and they were long since gone) I don't actually know how this was achieved. I do know that I got to sit and chat with my fellow developers for a few hours.
Accidentally delete the Member Servers OU. That was another few hours chatting while a restore from tape was done.
Our admin intended to give all domain admins permission to use CD & floppy drive access. (We used SecureNT to control access to removable media at the time.) Sadly he got the group membership backwards and instead gave all users of removable media full domain administrator rights as well. I found this because some tables turned up in a production SQL database that had been created by a user that shouldn't have been able to. When I told the administrator in question I enjoyed watching his face change from, no, that's the right way round, down to, oh ****. Thankfully there was no serious harm done.
Not me, but someone I work with. They created a policy on the AV server that contained a
* in the process field. In layman's terms: do not allow read, write, execute to any process that contains the name
This policy then was replicated to 1,500 servers, which in turn shutdown RDP and any other process. To fix it meant to mount every server hard drive one by one and remove the policy. 48 hours with a team of 15.
I had an employee complain that his laptop was slow, so I checked the hard drive fragmentation and it was (and is to this day) the worst I had ever seen. Attempts to defragment the drive were fruitless because there was not enough free space. I tried cleaning up temporary files (not sure why I didn't just move stuff to the server temporarily) and stupidly deleted his entire outlook.pst thinking that it was a backup of his e-mail and not his actual e-mail. He forgave me, but never let me forget it.
(This happened many years ago shortly after I graduated university. I'm much more competent now.)
I had one not so long ago. During some Oracle ODBC bridge deployment, I had to modify the path on about 500 user posts.
It's a quite simple operation, really. Too bad I forgot about those quotes. People started ringing after they had some strange garbled messages (the ODBC install failing), and seemed to think rebooting the machine would be just wat it needed.
Of course, some other previous installation PREPENDED (!!!) some program files path in the system variable (with spaces and all, without quotes), so the new path stopped just there, at c:\Program (of course, the existence of %ProgramFiles% remained completely ignored). No system, no system32, no shell. So no logon scripts either.
People who rebooted didn't have any network access anymore, and no automated script could repair the damage. Of course, as soon as I went to some complaining user, looked around and checked the path, I got that.. sinkin' feeling.
In about 30 minutes, I had another script, with the most standard path values, ready to be mailed to everyone (e-mail still worked). Users even phoned back to be sure the patch was real, as they are not used being send cryptic exe's with strange reasons to apply them, and most of them weren't even aware of what was happening.
The first version was messy (a new semicolon at each execution), but it logged every possible path value available, so I quickly had data with possible paths, so I just had to create something smart to check them all, end get the path nicely in place.
All in all, it lasted only about 45 minutes, and I was luckely the one who put everything back allright. But still, when a corrupted path pops up now, I'm still ready to take the blame ;)
Was adding RAM to an email server in a cabinet with about 8 machines. On the top row was a power strip with a "lightswitch" style on-off switch. As we reached in to pull out the machine, my arm flipped up the light switch, cutting power to all of the machines ( There was no UPS on the boxes ). All of the machines came back up except 1. Spent the next 4 hours ( 2am - 6am ) fixing that box.
1) UPS is good
2) Only use power strips and UPS's that protect their on-off switches from accidental tripping.
My best one came at a time when our backup server was in administrative limbo - my boss was "debating" whether or not it should remain in the office, off-site from our server room (and not doing backups for some reason) or whether it should be installed in the server room to save massive amounts of bandwidth. I seem to recall that this limbo state existed for several months.
Our web server had a RAID 5 array for storage of websites. It seems that it had been running in degraded mode (without informing me for reasons unknown or which I can't remember) for some time before the second of three drives failed. I got to pull an all-nighter putting the server back together. Our customers were Not Happy that their websites had disappeared and they needed to restore from their own backups. Especially the ones who didn't have their own backups.
Questions my boss asked me were "How could a RAID array fail like that? I thought they weren't supposed to!" and "Why didn't we have backups of our webserver?"
However, the lesson had not gone unheeded. My boss was cooperative when I suggested that the upgrades to our mail server should include a RAID 1 array with a hot spare (instead of arguing with me over the extra cost, which he would normally have done). And of course, the backup server was doing its job properly in short order.
How about learning the difference between Exchange Server 2007 "Remove Mailbox" and "Disable Mailbox" feature? Especially when I'm removing everyone's old mailbox to deal with a corrupt database?
Restore on an exchange server... not fun... Having to restore an exchange server AND active directory... double not fun.
Doing it at 11:00 AM Friday morning... Priceless.
I was new to RAID 5 and was still learning about how it worked. At the time I was the only IT guy in a very small company. All the files everyone accessed were stored on only one server. The server was getting low on space and had only 3 drives in the RAID array, so I thought adding in a 4th would increase space and responsiveness. I did this during business hours. I hadn't learned the concept of after-hours maintenance.
The array started rebuilding, and it said it would be done in 36 hours. I thought that was way too long. I found a slider that controlled rebuild priority, and it was set to the lowest setting. I set it to medium. The time went down to 8 hours. The hard drive lights were blinking a bit faster, but I still thought that was still way too long for only 80GB of data. So I set the priority to high. The hard drive lights went solid, and I thought "that's more like it!" Then the GUI I was using stopped responding. It connected to box remotely. I tried to bring it back up, but it couldn't find the server.
I started to hear people down the hall complaining that they couldn't get on the server. I went to the server to log in to see what was going on. It took 5 minutes for the blank screen to change to the background. It was another 5 minutes before the login prompt came up. Each key press took 5 minutes to register. I had set the priority so high that the server wouldn't respond to anything. It took 2 hours for the array to rebuild. Luckily it was an hour before lunch, so no one really cared that much. My manager at the time was a really cool lady and said it wasn't a big deal. The head design engineer did give me a mean look though. I was sweating bullets for 2 hours. Lesson learned.
I was trying to free up some space on the primary partition of the site's RedHat 5 web server. I was relatively new to Linux but had been using DOS for ages.
I managed to move the entire /bin folder to another partition, taking out the production website, and leaving myself without any accessible system commands. I freaked out, I couldn't rename, copy, move, anything because I'd moved all those helpful executables.
Thankfully I was able to use a boot disk and undo my handiwork.
Ha, My first really big accident was when i was writing a small SVN Admin panel on our development server, completely insecure software that was only to be used for updating the internal "Development" website.
Sometimes the SVN repo would get corrupted so i had written a button that would call a PHP file, which would clean out the entire SVN directory requested, and looked something like this..
<?php $directory=$_GET['dir']; $result = shell_exec("sudo rm -Rvf /".$direcory); echo $result; ?>
For those who don't see it -- the i misspelled "$directory" in the shell_exec, causing the system to run "sudo rm -Rvf /" .... At first i thought the web page was just taking its time deleting all the files in the repo. After about 10-15 minutes i had discovered i had destroyed over 1/2 the file system.
Maybe more of a late night brain fart than anything else.
One of the developers was having trouble with running a Java profiler on a Solaris box. The profiler was complaining that there were two copies of Libc; one in
/lib and one in
/usr/lib. So after a few
lds we moved the one from
/lib as everything was pointing to
/usr/lib, or so they said.
But suddenly nothing worked. No
mv. After about 20 minutes of 'oh crap, oh crap' we figured out that one of the developers had a currently running copy of Emacs on that box and we were able to open the backed up
/lib copy of Libc and write it back out with the original name. And voila! Everything worked. Lesson learned; leave Libc where it wants to be and don't make changes on developer requests at 2 A.M.!
Very stupid mistake.
I was writing a script on my Linux workstation that processed a number of files, but it didn't matter what kind of files it were, as long as it were a lot of files. So I decided it was a good idea to copy
/etc to a directory I was doing my tests in. When things went wrong, I deleted the copy and copied
/etc to my test directory again. That went well, for a some time, and then I typed
rm -rf /etc
rm -rf etc/
OK, nothing to worry about, I could still do things on my workstation and thought I could revive it by copying it from another workstation, or something. Or, reinstall at the end of the day. First, get something to drink, and because of the corporate policy, I locked my screen. Damn, I need my password to unlock and that's in /etc/.....
/etcfor testing purposes