You sure witnessed it with your own eyes (or are going to) sooner or later: that dreadful project/system/situation where something got SO screwed up you just can't believe it actually went like it did.

Mismanagement? Misbudgeting? Misunderstanding? Just silly, plain ignorance? Name your cause, it sure happened (and keeps happening a lot, sadly; see here).

Describe it here, for amusement (although somewhat of the cynic kind) and learning (hopefully).

Some rules:

  • This is not the place for random (even if utterly devastating) admin errors, so please avoid "argh I mistyped that rm -r" or "OMG I JUST COPIED THE CORRUPTED DATABASE OVER MY LAST GOOD BACKUP" (been there, done that); those things are better here. This is about "what sort of drugs was exactly under the influence of who designed/implemented this system?".
  • One WTF per post, so they can get properly commented.
  • Please post something you actually witnessed :-)
  • If it was you who did it, it still qualifies :-)

I'll be adding some material soon, feel free to add your own; and please do :-)

  • 11
    This needs to be a community wiki – squillman Aug 20 '09 at 16:46
  • 6
    To be honest I think it requires too much discussion to be a valid SF question. Perhaps www.thedailywtf.com should be brought into the LoJ? – EBGreen Aug 20 '09 at 16:53
  • What is a community wiki exactly? I've seen some questions flagged as that, how does it work? – Massimo Aug 20 '09 at 17:42
  • 1
    Funny moving from a site with full moderator rights to one with no priviliges whatsoever. Lolz. – BinaryMisfit Aug 20 '09 at 17:47
  • 3
    So, I created and sysadmined this website for asking technical questions and having a community of users answer them. Then someone posted a question that was totally anecdotal in answers. Clearly violating the rules! And the @#%#$ using the site actually voted it up even though some voted it down! My inner pedant was dying a little more each day this question was left in the questions...WTF?! :-) – Bart Silverstrim Sep 10 '09 at 11:56

22 Answers 22


Email response from a Microsoft support engineer to a reported issue:

"As far as my opinion of your issue, I have one word: WEIRD."


  • 3
    At least you got a response. Not everybody does. – John Gardeniers Aug 24 '09 at 23:20
  • 2
    "Reality is never weird. Calling reality "weird" keeps you inside a viewpoint already proven erroneous. Probability theory tells us that surprise is the measure of a poor hypothesis; A good model makes reality look normal, not weird; [..] But it will also hinder you to keep thinking How bizarre! Spending emotional energy on incredulity wastes time you could be using to update. It repeatedly throws you back into the frame of the old, wrong viewpoint. It feeds your sense of righteous indignation at reality daring to contradict you." - lesswrong.com/lw/hs/think_like_reality – TessellatingHeckler Mar 26 '11 at 20:30

I was called from a company I never heard of before, which had been tasked with implemented an Exchange 2003 mail server for a customer and had no clue at all about how to do it; nothing too strange, right? I work as a freelance consultant, so I'm perfectly fine doing jobs you don't know how to do for you (and getting your money for it).

So I went at the customer site, and discovered something quite strange: every single server in the network was a domain controller; all 15 or so of them.

Then I discovered something even stranger: no one of them was properly replicating with any other, Active Directory overall behaviour could only be described as "erratic", users were having about any network issue you can imagine and Exchange just refused to install with unknown-to-mankind errors.

So I had a look at the network configuration on the server, and I saw... it was using the ISP's public DNS servers. Then I look at another server... and it was the same. Then I look at a DC... same thing. Then I asked... and it was officially confirmed: each and every computer on the network (about 1500) was using the ISP's DNS instead of a rightful domain controller.

I proceeded in explaining DNS is quite critical for proper Active Directory operation, and was able to reconstruct the back story:

  • Someone originally set up the AD domain correctly, using a DC as the DNS server for every computer.
  • He/she/it didn't know anything about forwarders and/or firewall configuration, so computers were unable to resolve Internet public names.
  • So came the idea of using the ISP's DNS servers on the computers; they configured it on everyone of them.
  • They started having lots of "can't find a domain controller" error (who would have guessed?).
  • They thought the problem was caused by not having enough DCs, so they proceeded to promote every single server to that role.
  • Needless to say, this only worsened things, as those new DCs too used the wrong DNSs, so they were also unable to replicate.
  • This went on for months, they just "got used" to the network being totally unreliable.
  • On top of this, they try to launch Exchange setup, which crashed miserably; only then they decided to call some external consultant, and until them they had absolutely not clue something was totally, definitely wrong in their network setup.
  • 3
    I love that stuff. "We couldnt login anymore because the server is unreliable, so everyone has local accounts now". uhhuh. – SirStan Aug 20 '09 at 17:20
  • 3
    This is a prime example of the Tragedy of the Temporary Fix. – Ernie Aug 24 '09 at 16:57

Once upon a time I had a client that was a small business (10 people) with an electronic health record. (Not a medical doctor). I noted one day that the backups had been failing. Upon testing, the tape drive was not working at all. I mentioned this to the owner, who said he was well aware that the drive was bad, but was too expensive to replace.

Sure -- thats not very WTF.

The WTF is he had his staff rotating the tape daily, taking it to a safety deposit box, and all that jazz for the 6-9 months since it died.

"Don't tell the staff, it might worry them"

  • 6
    Oh. My. Goodness. – Wesley Aug 20 '09 at 17:13
  • 7
    thedailywtf.com/Contact.aspx – LRE Aug 20 '09 at 20:02
  • 3
    dont waisting everyone time to change tape and storing it waste more money that just fixing the damn drive? – Robert Sep 16 '09 at 1:21
  • @Robert: I guess this WTF comes from actually ;) – Olivier Pons Oct 28 '11 at 7:45

I was working as a sysadmin for a Big Government Agency (one of the main bodies of Italy's governement), and had been managing their data center for some months. One evening, my phone rings and my boss tells me something Very Bad is happening: total power outage.

Ok, we have UPSes, right?

Yes, but they won't last long, so better go there and shut down everything until power returns.

I go there, make my way through the dark corridors, arrive at the server room... and am greeted by what can only be described as pure hell. Literally. The room was so hot you could have baked cakes in it. UPS power was ok, but half of the servers had already shut down from overheating and the remaining ones were screaming in agony.

The reason?

Servers were on UPS power... air conditioning was not.

  • 15
    There's no feeling quite like walking into a 100 degree room full of amber and red blinking lights with lots of beeping alarms and server fans shrieking at 100%. – kmarsh Aug 20 '09 at 18:05
  • 2
    Sadly I've noticed a lot of large companies are setup like this. Several times when Rackspace went to Generator power the temp that the servers reported would go from 70 to 100+ in minutes. – mrdenny Aug 20 '09 at 23:16
  • 3
    Truly, we have a reasonable way to simulate hell on earth now...hot, wailing, dark, and you want to leave as fast as possible, but can't. – Avery Payne Jul 29 '10 at 18:31

HP ProLiant ML370 G3 fan failure... The fan sensors on the motherboard of this model tended to go bad after 5 years. The server can't boot when the wrong combination of fans is not detected. I had to walk the customer through jumpstarting the machine with a shop-vac (to get the fans spinning at boot) and this is how they kept the server running until I could arrive with a new system.

enter image description here enter image description here


I used to be an accounting software consultant for Dac-Easy accounting. One time I was called into the main office of a local business and was told by the accountant that if I couldn't resolve why the program was full of accounting errors each weekend, they would have to find another application and consultant. Going through the administrative log files, I discovered all of the entries were usually made on Friday or Saturday nights. I then found out that the owner's wife was logging in to the accounting system computer from home using PC Anywhere and trying to balance the accounts with her checkbook after several glasses of wine. Once the numbers looked good, she'd log off.

  • 2
    I can guarantee that was your fault, dammit. – LRE Aug 20 '09 at 20:14
  • 5
    How many glasses of wine until the numbers look good? I bet there are a lot of people who want to know. – kmarsh Aug 21 '09 at 17:47

I lifted a computer I received from our parent company's office when I heard something rattling around inside. When I opened the case, I found it 1/2 full with Frosted Mini-Wheats. I figure a mouse was living in it or using a food cache. Likely entry point was the gap between the case and the DIN keyboard jack.

Not exactly what you asked for, but a definite WTF.

  • 3
    I worked for a seed company years ago. One fall wee had machines in the seed lab doing the weirdest things. Of the 4 boxes out there 2 had mouse nests inside the case. Mouse pee is bad for motherboards. We all got MUCH better about puting the card slot covers back on after that. – Laura Thomas Aug 20 '09 at 16:54
  • And here I thought card slot covers were there just for airflow... – Ernie Aug 24 '09 at 16:58

Another customer, another horror story.

In the main post I talked about having erroneously overwritten a good backup with the corrupted database it was meant to replace; it happens :-(

So a restore from backup was neeeded. Luckily, there actually WAS a backup there: it was done daily, on a central backup server with a Really Big tape library attached to it; this server managed backups for the whole company, it was Really Expensive and had a Real Backup Software installed on it.

So far, so good. We look up the backup job, load the proper tape, start the restore operation, the tape is loaded, the restore starts... and just nothing happens.

We try again, some thing.

We unload, reload, reboot, try restoring previous backups... nothing changes.

We assume some long operation was going on, and leave it on the whole night... next day, still nothing changes.

Ok, time to call the Real Backup Software vendor's support... but it can't be done, we are on Sunday. We try look up the vendor's support site, but a special access code is required, and only one manager has it... the same manager which will be Really upset on discovering the system is still down on Monday, when he comes to work.

Another day of pain, and I discover the bug is well-known, and it has been fixed by a vendor patch, which (obviously) nobody bother applying. So I go to apply it... but it can't be done: management doesn't want to risk breaking anything unless the vendor confirms the patch can be safely applied; the fact that the backup server wasn't able to restore anything apparently didn't look enough "breaked" to them.

Only after four total days, various support calls and the vendor sending a support engineer onsite we were finally able to apply the patch and restore the backup; the backup server hadn't EVER been able to do restores, but nobody ever tested it, so nobody noticed.

  • 1
    Well, as they say, backup is easy, it's the restore that can be tricky... – sleske Aug 24 '09 at 22:27
  • 1
    "Nobody cares about backups. They only care about restores." – Stefan Lasiewski Jun 13 '11 at 5:24
  • @StefanLasiewski Actually and sadly in our world the converse is generally true :| – Shadok Feb 8 '12 at 16:34

Terminal Emulation For Fun And Profit

I work with an older system -- essentially a bunch of text terminals which connect to a Unix box, with a few Windows computers thrown in just to complicate things.

A few essential background facts

  • Primary software application uses it's own extended termcap file. System-wide termcap and terminfo are ignored.
  • Primary software application allows various keyboard mappings to be selected according to environment variables.
  • All text terminals have programmable keyboards.
  • Windows PC's connect using proprietary client software with no built-in support by primary software application.
  • Proprietary client software allows various keyboard mappings to be selected on load.
  • Only half the staff use standard QWERTY layout.
  • At least one, possibly more, of the system administrators who have worked on this system have lacked a proper understanding of terminal emulation.

I think you can see where I'm going with this.

There'd been half-hearted attempts at adding proper terminal support to the terminfo and termcap files, but these are only partially functional. The proprietary termcap file used by the primary system application works, but it's mostly irrelevant since the $TERM is never set properly anyway.

Logging in, each user has to basically select where they're logging in from and what keyboard layout they want to use -- there's no autodetection involved. This will set $TERM to ANSI, regardless of which terminal is being used on the client side, and an environment variable set so the primary software application remaps some of the keys appropriately.

In the case of people using the embedded keyboard layout on the text terminals, a script is then run to reprogram some of the keys to the embedded layout (as physically printed on the keyboard itself), but not all of them. On the Windows PC's, a client keymap is loaded up to remap most of the function keys to ANSI. Can't remap all of them, or else some of the other keys will stop working.

Logging into anything else from any of the terminals is an exercise in failure, and can only reasonably be done directly on the server itself. Since everything works, it's difficult to justify the time and effort required to fix it. But adding anything newer than the 20-year old text terminals we're stuck with now would essentially require a rewrite of the entire system.

I cry a little inside every time I see it.

  • 1
    You need a sign on the server room door that says "Beyond this point lies madness". – Ernie Aug 24 '09 at 21:00

I knew someone who decided to reorganize the files on their computer by putting all the .exe's in one folder.

  • 4
    Ah, yes, that one! Quite similar to "I deleted all that stuff in C:\Windows which was totally useless and taking up so much space"... – Massimo Aug 20 '09 at 17:57
  • 3
    I do that to Windows systems running Deep Freeze for some feeling of catharsis. – Bart Silverstrim Sep 10 '09 at 12:14

A network with ~60 (SIXTY) PCs.

A security-fanatic boss.

Some new switches with VLAN capabilities.

A "network reorganization plan" involving ~20 (TWENTY) VLANs.

Thanks to some unknown upper power, I left before all of this could get actually started...


We gave one of our Unix admins a Windows box. This was shortly after we had given Windows admins Linux boxes to admin in an attempt to increase our familiarity across different operating systems. I as a Windows admin trying not to screw up my Linux box I had asked a ton of questions of the Unix team. I expect this was the objective of the exercise. After a stern lecture about making sure I sized all my volumes right and not putting everything on one volume, or on a single partition, I went forth and built a perfectly adequate box, that is still running in production two years later.

When the Unix guy who had given me the lecture about different volumes and not building things so that a full disk could crash the OS built his Windows box, he put everything on C:. When I said "I can't back that up with any policy we have and when those logs fill your box will crash." He said "Well I figured it was Windows, everything on C: makes sense." He didn't make another partition at all. I was seriously boggling at this. Absolutely WTF were you thinking sort of thing. Apparently it didn't occur to him that some best practices are the same regardless of OS.

  • 10
    Arguably, often "everything on C:" does make sense in a Windows world. As Unixy types understand / as c:. The "correct" Unixy fix would have been to make extra partitions and mount them as folders, rather than drives. So Partition 1 = / (C:) Partition 2 = /home (c:\documents and settings\) Partition 3 for c:\windows Partition 4 for c:\program files Partition 5 for shared data (if it was a file server) – SirStan Aug 20 '09 at 16:57
  • A person could believe that if he hadn't spent a bunch of time in desktop support of windows machines. – Laura Thomas Aug 20 '09 at 18:02
  • 1
    One of the problems here is that for MOST people, everything on C: makes perfect sense for the very simple reason that at least 99% of Windows machines only have a C: drive. Sure those are workstations, not servers, but it pretty much locks in the mindset for those who haven't learned better. – John Gardeniers Aug 24 '09 at 23:10
  • 1
    I guess I'm missing the WTF there since most machines have just a C: drive and the OS logfiles can be capped easily (as can Linux systems, I suppose). There is a difference that I think stems from filesystem layout...Linux using mountpoints to create a giant single filesystem while Windows used discrete drive letters for volumes, and you had some applications that barfed if they installed on anything other than C: (or other complications can ensue later). I've seen plenty of Windows systems that ran out of space because there were two partitions on a drive, though... – Bart Silverstrim Sep 10 '09 at 12:05

Massimo's reply reminds me of another WTF...

A small office has Windows 2003 server with Exchange, Symantec AV, etc. The company usually uses us to do all their IT work.. but this new hotshot is in the office and bought a new PC. We had them on a unified Dell Optiplex platform, this guy decided to replace 3 of them with "great machines" that were on sale at Best Buy.

Regardless -- they somehow manage to join it to the domain.

First call:

We can't find outlook. The machine came with Word (Microsoft Works), so where is everything else.

Second call:

How do we use the network antivirus?

This is the wtf. I went onsite, and 7 of their machines had been configured as Symantec Antivirus SERVERS, instead of installing the client. Apparently the gentleman didnt like that we had locked the settings of the client, and decided to "reinstall" the client so he could change when it scanned. Apparently the first CD for Symantec AV that we had left out somehow persuaded him to install the "server", rather than the client.

  • It's easy to install the server instead of the client. It's the default choice on Endpoint Protection, for some reason – Matt Simmons Aug 20 '09 at 17:26
  • 5
    Installing the server is the default because you are not expected to install clients from the CD. When you install the server it publishes a share that can be used to install clients that then automatically know what server to connect to and stuff like that. You are supposed to use those install files, or push the installation from the server, not stick the CD in each client computer. – Catherine MacInnes Aug 20 '09 at 18:48
  • I'll get all my people's machines on AD soon, damnit :-) – Matt Simmons Aug 21 '09 at 14:09

A client called me to help with some server builds. During the build I was informed that these were replacement servers for the bottom two from the rack. I thought it was strange that the rack position was relevant and asked why. Turns out the server room flooded after the water pipe that runs through the middle of it burst.

So what is a water pipe doing in the middle of the server room you ask? Well, the ladies bathroom is next door and the server room is conveniently out of sight. What better place to put the main feed to the bathroom?


Oh, this is an easy one...

I had a Linux system installation for a company situated in a warehouse in rural Kentucky. This system is the accounting/inventory application for the organization. I was replacing a 1998-era SCO server with a new CentOS 4 server. As I result, I expected the worst when it came to connectivity and clients. You know, old PCs, bad cabling, etc. I shipped new switches, cables, thin clients, etc. to the site...

However, I did NOT expect the wiring "closet" to be in the BATHROOM! Actually, the cabling was located in the space above the dropped ceiling directly above the warehouse bathroom toilet. This included the DSL modem, 10-megabit HUB and a 66-block for the phones...

enter image description here enter image description here


Same customer involved in the DNS From Hell thing.

Their Exchange server kept running for years gracefully and even survived some disk failures (thanks, RAID, and thanks more, backups). One day, I was called again for an unrelated problem, and they told me en passant something was troubling them: some times before, they discovered everyone was able to open everyone else's mailbox; they thought it was strange, but didn't do anything, because they don't know how to fix that, and anyway e-mail was working and (almost) no one had disccovered this.

I was quite shocked here: everyone was able to open any one else's mailbox, and they thought this was ONLY A MINOR NUISANCE and didn't bother doing anything for THREE MONTHS. Any ordinary people would have yanked the network cord from the server immediately, but not them.

The source of the problem was... strange: someone had set "Everyone/Full Control" at the Exchange Organization level in Active Directory, so everyone immediately became a full Exchange administrator; their only luck was that very few people noticed it.

As of today, nobody knows yet who actually did that.

  • 2
    I've managed a number of small networks where a requirement was just what was implemented here -- specifically because it is more secure than sharing passwords. Jo the owner can open Steve the receptionists email (does my gender selection offend anyone? :)) if Steve is out. It does make some sense in some scenarios -- but typically in a "everyone knows what color underwear everyone else wears" type of office. – SirStan Aug 20 '09 at 17:17
  • I had worked with a small business where there are fixed, shared, role-based user accounts and they actually wanted it to be this way. But they were 15 employees in a small business. The customer of this post had 1500 employees, they definitely did't ask for full e-mail sharing, didn't want it, but just didn't care about it for THREE MONTHS. – Massimo Aug 20 '09 at 17:25
  • 1
    I'm wearing black underwear right now. Does that now make Server Fault that type of office for us all? – Wesley Aug 20 '09 at 20:52

Worked as a full-time programmer and part-time admin for a small company making doctor's office accounting (read: customer billing and insurance claims) software. Got a call from a big customer that their server was down; it was the new product running Altos Unix SysV/386. Since I was all they had, they sent me and the new gal.

Got there, and there was a room full of data entry operators typing in WordPerfect documents as fast as they could. There was a long-haul line going out of state where another set of operators were doing the same.

The documents directory had about 10,000 word perfect documents, all of them following the same format. ALL IN ONE DIRECTORY. So-and-so has been injured and suffers from such-and-such. Seems that this doctor was the go-to guy for lawyers filing injury lawsuits.

I tried to boot off of floppy and got only lots of seeking noises. I pulled the floppy out and it was completely covered with cobwebs, I'm talking handfuls here. It was a doctor's office, so I demanded swabs and alcohol (it was the only Unix boot disk we had, and the only chance of fixing things on that service call), and got started cleaning up the floppy, drive and server.

There were two problems, one was the server was full of dirt from the carpeted office and its corner-of-the-room location (on thick plush pile carpet), and was overheating in the 80 degree office. The filesystem needed serious FSCK'ing and I divvied up the 10K files into a new set of alphabetical subdirectories and suggested they break it down further.

The doctor was a perfect ***hole who ripped us a new one when we came in the door. When we left we explained that buying the WordPerfect for UNIX option from us didn't mean we had sized the server for intracontinental document creation; also, he needed an A/C, a carpet-free server location, a UPS, an external RAID and a better directory filing protocol. Suddenly he was all ears as this was how he made all his money. I gave all the information to Sales but left that job soon afterwards for other reasons.


My best WTF originated at my current employer, when I first started here.

Pretty much the first couple months on the job involved a constant state of panic and outright horror as I discovered the state of the servers. At my current level of experience, I might have just recommended setting fire to the place and collecting the insurance. But the very best part was one company policy:

There was no accounting for dialup customers. At an ISP that got half its income from dialup customers. This had actually been a policy that had been in place for some 6 years by the time I started, and it just boggled my mind that nothing had been done about it. The standard line was "we just trust our customers not to use too much." The fact that the place was running on a shoestring budget and didn't have a real admin at the switch for all that time didn't help matters I don't think.

The end result of this policy of course, was that spammers had basically commandeered the dialup pool. Nearly every night, they would use one account to dial into multiple modems until the whole pool was full, then blast out spam as fast as they could. The version of AAA RADIUS didn't help matters in that there was no quick "allow only one user at a time" switch at the time. I hacked something together in Perl to disconnect anyone with two logins until such time as the RADIUS server could be replaced, and that fixed that problem sometime in the first week or two. Part of the other problem was that I wasn't allowed to have enough time to fix things (see: shoestring budget) since I was also doing tech support at the same time. But part of the reason I'm still here was that I had free reign over it all. I don't think I've ever really been told not to go ahead and implement some sound idea. The only restriction I've ever had was money.


Ten years ago, I interviewed for a company whose primary server room's cabling looked like spaghetti soup. The networking equipment was in 3 racks near one corner of a 20ft x 60ft room. There was a cascade of network cables of all sorts of colors tumbling down the racks and ending in a pool of tangled cables about 7ft wide and spilling out about 5 ft from the racks. There wasn't any documentation and no one knew what went where.

The company had a 95% market share in a field that dealt with safety of life and property. All I can say is that that can give a real pause.

BTW the company is still in business and has apparently cleaned up a lot of things since then.


Was in a Demarc closet not so long ago and found this sweet setup on a site:

Elite fiber cable install= TWIST TIE

alt text http://locobox.googlepages.com/Fiber.jpgalt text http://locobox.googlepages.com/Fiber2.jpg

  • 3
    I've been told recently that twist ties are more "green" than cable ties, being reusable and all. Nonetheless tight coils on an optical cable are a big no-no. – kmarsh Sep 10 '09 at 13:03

A few years ago, the company I was working for acquired one of our competitors and I had to audit their machines (Centos) to ensure that they would meet our standards in terms of configuration (their sysadmin had declined the opportunity to join our company in favour of starting his own company that developed turn-key solutions).

In /etc/rc.local (the home of all great WTFs) I found:

swapon /dev/sda3


route add -net netmask eth1

Note: eth1 was configured in ifcfg-eth1 with a netmask of

Apparently real sysadmins don't use those cliched config files like /etc/fstab and /etc/sysconfig/network-scripts/*

  • 4
    If you are coming from a Debian, Slackware or Solaris background, finding ALL THREE network scripts in CentOS/RedHat and fixing them can be confusing. – kmarsh Sep 10 '09 at 13:08

Earlier this year we hired a new girl. I was asked to make sure that all the company fonts were installed on her computer. I knew they were but decided to pick my battles and just install them again. I got a call from the girl about 5 minutes afterwards and she wanted me to come over and show her how to use the new "software" I installed. I walked over, opened Word and walked her through how to change fonts....which is what I installed.

This girl had a PHD!! She no longer works here but not because she didn't know how to use fonts.

  • Off topic: Can someone hold a PhD and still be called a "girl"? =) – Wesley Aug 20 '09 at 20:53
  • Well to be frank, she should have been using a stylesheet that dictates the use of fonts for you through, well, styles, so that every document she produces would have been formatted correctly automatically. Although one does wonder how she formatted her thesis... – Mark Henderson Aug 20 '09 at 21:15
  • 1
    I don't know this person, but the only time I've heard of this level of cluelessness (not just technical/computer cluelessness) in a PhD was when the document was bought online. – Ernie Aug 24 '09 at 17:11
  • @Nonapeptide: in this case yes. – AppsByAaron Aug 24 '09 at 19:24
  • 2
    I think she used LaTeX to make her PhD thesis.. :) – Arie K Sep 10 '09 at 12:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.