We have a small engineering team and the need to handle 24x7 emergency support (i.e., responding to outages in our service APIs, so more of a NOC responsibility than customer support). Today our engineers have an on-call phone, but too often we miss initial reports because the phone isn't loud enough, isn't in the right room, etc. I'm not confident we can flush out all the issues that would ensure a single person with a single phone will always answer.

We assume it would be best to hire a company as first line of defense outside of working hours to field calls and escalate internally as needed.

Basic searches for outsource NOC/customer support yields dozens of companies that look more or less identical. Any recommendations for an approach or a specific organization that offers these services? Of course we're trying to put this together as inexpensively as possible.

link|improve this question
I must be missing something. If the first call is getting missed, what's to prevent the escalation failing for exactly the same reasons? Simply having an outsider taking the initial call won't really make any difference to the problem resolution. At least not the way you've described it. – John Gardeniers Nov 9 '09 at 18:59
If you are currently insourcing, are you compensating appropriately for it? – DanBig Nov 9 '09 at 19:09
At the moment we have a single on-call phone carried by a single engineer, rotating periodically. Our customers have that, and only that, phone number. If they call and the phone isn't answered, there's no response. If we had a NOC service, they could walk through an escalation list rather than a single phone number. We'd still have an engineer on call, but more than one backstop in the event on-call fails. @Dan, unfortunately no, we don't have any special compensation plan for emergencies. – Thomas H Nov 12 '09 at 4:36
feedback

2 Answers

We went through this same scenario and went with an inexpensive answering service. We gave the service a simple script to run through to assess the severity of the problem. The hope was to avoid waking someone up unless it was really necessary. We had a primary and a secondary Blackberry which rotated among the team. The service was instructed to get the customer's contact information and then call the primary. If the primary went unanswered after some number of tries, call the secondary. If the secondary didn't answer, the service called the manager's cell phone. Whichever of the 3 of us the service reached first would return the customer's call.

In practice we were lucky if the service got the product name captured correctly. We couldn't rely on them for problem severity assessment. They did get the contact information for us. Fortunately our customer base was honest enough that they didn't wake us up for silliness (most of the time).

What sort of SLA do you have in place with your customers? That will also factor into how robust an off-hours call mechanism is required.

link|improve this answer
feedback

Why outsource? Why not insource? Since the problem will be escalated to you anyway, this seems to be the most logical solution.

Just rotate weekly and make one of you an on-call person that will get woken up during the night. If you are lucky, he might be the only one who gets woken up.

This solution has an added bonus that it stimulates you to maintain a reliable service, so you don't lose your night sleep. Plus, a customer that is calling gets a knowledgeable person to answer the phone instead of a call center drone that runs him through a script.

Of course, this needs to be appropriately compensated. In my company on call person gets 20% of a daily wage as an "inconvenience" bonus plus a normal overtime pay for the actual hours he gets to work extra.

link|improve this answer
Lost sleep is a great incentive to improve quality. – Eric H Nov 7 '09 at 21:39
2  
We insource today (will update the question). The problem is that the engineers on call don't always take the call - phone not loud enough to wake them up, phone in the wrong room, etc. I understand the impersonal helpdesk, but at least it's a reliable first response. – Thomas H Nov 9 '09 at 18:42
Is your goal really solving your customers problem or providing a "reliable first response" for your management to put in contracts? If its the former, how far do you think giving lase hope to your customer will really get you? If its the latter then it really doesnt matter who you take, just take the cheapest ;) When I call, I want problems fixed so that I can go back to sleep, now, not when someone wakes up in the morning. Why are your engineers so sloppy with the phone? Are you paying them enough? Do you have too many problems so they don't get any sleep? This is your real problem. – Aleksandar Ivanisevic Nov 10 '09 at 14:11
Thanks for the feedback. The goal is both a reliable first response and a quick resolution. Our customers desire that initial acknowledgment, rather than having to repeatedly dial an on-call number. The middle-of-the-night problems are rare, which may be contributing to the "sloppiness", but at the end of the day it seems difficult to completely avoid Murphy's Law. – Thomas H Nov 12 '09 at 4:42
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.