While not exactly a task I was given, I had to recover from forced downtime.
I used to be the SA for a rather large financial site.
We knew our systems inside and out, had centralized logging and great tools to sift through them.
All of a sudden (naturally very close to a deadline), all the cellphones go crazy with alert messages. Check to see that the site is down, and all SA staff drop what they're doing and start investigating.
Apache logs were fine, database wasn't throwing any errors and the caches were spinning just fine. Plenty of spare resources, network was fine and no recent deployments.
10 minutes later, I find out that one of the developer had gotten access to the site and added a die(); in an obscure module related to page generation.
In other words, the software did what it was told to do, and there was no log information that would've helped.
The GM for the company that ran the site came up with a huge grin and said he wanted to clock us.
I told him to screw off, not touch my production servers, that we did have great disaster recovery plans, but his developer's incompetence made sure none of those plans kicked into gear.
If he wanted to check our response time, he should at the very least have discussed it with the CTO and asked if he could do it "some time today" or "this week". That way nobody would have been pissed off, and we wouldn't have wasted any time arguing about it.
The whole event was one of the most unprofessional ones I've encountered so far.