3
votes

I'm working with a team of fairly smart individuals with a reasonable skill set for the task they're given (supporting the operations leads). No doubt due to the circumstances and the training drilled into them, they show little initiative and are afraid of doing anything wrong.

Now the company has decided this team will join in the on-call rotation. In the first meeting to discuss this, from the look in their eyes and their questions it was clear that responsibility for decisions during emergencies was very unsettling for them. I fear they may freeze up in such a situation.

How do I get them to open up and take the situation under control when an emergency hits? Anyone have direct experience with this?

Edit: This is about system administration (jboss, websphere, mysql, etc), but I realize this may be more applicable to a management forum. However since I'm looking for practical experience from the trenches, I guess serverfault still applies. My main concern is not having systems back up as soon as possible.

Also, since the department is fairly young, we don't really have a proper overview of what is most likely to go wrong. This makes matters even worse, of course.

Edit2: All very good answers, I'll just pick the one that did it best for me

9
  • I've made this a little more generic, purely because this doesn't only apply to foreign workers. At $DayJob I also have English co-workers who seem unwilling to make decisions in an emergency situation for the same reason that they are afraid of the consequences if they make a bad call.
    – Ben Pilbrow
    May 29, 2011 at 20:15
  • Do the people want to be doing on-call rotation, or is it being forced on them, e.g was it part of the job when they signed up for the job?
    – Ian Ringrose
    May 29, 2011 at 21:46
  • 8
    This is a management issue and has nothing whatsoever to do with system administration.
    – John Gardeniers
    May 29, 2011 at 21:54
  • @John Gardeniers, I somewhat disagree. I suspect most people are hesitant to take control when an emergency hits because they are unprepared for how to respond properly. I think the proper solution is to train them how to handle the event.
    – Zoredache
    May 30, 2011 at 5:35
  • See: What's your checklist for when everything blows up and My server's been hacked EMERGENCY. Both these question cover some aspects of how to respond during an emergency.
    – Zoredache
    May 30, 2011 at 5:37

3 Answers 3

Reset to default
3
votes

Find and possibly send them to a training on Incident command system. At the Cascadia IT Conference 2011 Brent Chapman gave a great talk about "What We Can Learn from the Fire Department". Some of the slides, and materials from a presentation of his, at a different venue can be found here.

Basically the fire department and other emergency response people have a very structure set of techniques to handle making decisions and getting things done in an emergency situation.

How do I get them to open up and take the situation under control when an emergency hits?

One of Brent 's recommendations was to have people be responsible for coordinating some major project or upgrades. The point is to put them in charge of major things in a more controlled environment, then they will be better prepared to handle operate in the more chaotic situation when things break down.

1
  • Very interesting, I'll look into this!
    – Joris
    May 30, 2011 at 9:59
6
votes

The best thing you can do is have good runbooks. "If A happens, do B" Every alert from monitoring should have a procedure, even if that procedure is "Call Bob at: " Bob should know that if he wants the calls to stop he needs to create procedures or tools.

For NOC operations it is often helpful to "apprentice" new team members, have them work the desk with an experienced person for 2-3 weeks.

Make sure everyone on the team is aware that if the first line support people escalate, even incorrectly they should assist them. Be willing to harshly punish any 2nd tier support people who are unresponsive or mistreat the NOC personally when they escalate. (This is a major problem, the second tier person is going to resent any intrusion in their off time, but if they are not responsive the NOC will become "gun shy" and will stop escalating, or will avoid escalating to that individual.)

1
  • 2
    totally agree; I have successfully used exactly this approach myself. TO reiterate; the person who does not write or who writes crap processes must also be the person to receive the 3am phone calls. And yes 2nd level support must be responsive; remind them the way to stop calls is training and procedures.
    – Karl
    May 29, 2011 at 20:24
0
votes

Apart from the other answerers spot-on suggestions have a second person, more accustomed to decision making, on-duty too during a transition phase for the purpose of being thoroughly consulted. The person doing the primary support shouldn't feel talking to the secondary is "escalating". Instead make it clear that questions are expected.

Are they all being handsomely payed for this? Apart from the fact that most people's attitudes on things change radically whether they are being rewarded for their efforts or not do note that in some countries being on-call without pay is downright illegal.

1
  • This is an interesting idea, I'll propose it on the next meeting. I'm not aware of their level of compensation, but considering they are hired because they are cheap, I fear for it. Most on-call pay seems to be very limited in my experience :(
    – Joris
    May 30, 2011 at 15:31

Not the answer you're looking for? Browse other questions tagged .