5
votes

So the organization I work for as a sysadmin is considering extending our hours to 24/7/365 - can anyone who works for or has worked in an extended hours shop maybe provide some basic info on your organization size, support hours, & call/incident volume so I can do a little comparison?

A lot of my coworkers seem to think that extended hours coverage is unnecessary due to our relatively small size. We do have 20+ international offices, but they are mostly quiet in terms of requesting technical assistance. Here are some more stats on my employer:

2000 employees

250 production servers worldwide

~20 incidents daily (sysadmin)

7-8 systems/network admins

We are also unsure if covering a 24 hour schedule will be difficult with only about 8 people (considering vacation and sick time) - so if you care to share how many people are in your group, that would be helpful as well.

7 Answers 7

3
votes

Whether or not this will work will depend a lot on whether you're dealing with automated alerts, a staffed operations center, or waiting for people to call for support. Our shop is a 24/7/365 shop, but I work in production systems, not in desktop support/client support. We don't get calls directly from users, so that helps a lot. We also support our own home-grown apps, and the development teams are invested in keeping the systems error free (from a code standpoint, the servers are up to us). Depending on what system a certain team supports, after hours calls can range from 2-5 a month to 2-5 a week. This can also vary depending on if a new code release caused issues.

As for compensation, our system was in place when I started working here, so my compensation was set with that in mind. If you are going to be expected to offer support after hours, you should be entitled to additional compensation. At the very least, if someone is on call, and has to deal with an after hours issue, they should be able to come in late the next day, or work from home if possible.

2
votes

One of the place I worked at had a 24/7 monitoring center. It took basically 10 staffers to fill all the slots, to have "spares" and to have a manager in charge. The biggest problem was the graveyard shift (2400-0800) since there was basically one guy coming off shift (in the entire building) and one guy coming on shift. Any delays meant automatic overtime for the guy on shift and if the midnight guy was a no show, a nice 16 hours shift.

We also had terrible turn-over rates on those jobs and much difficulty recruiting for the night/evening shifts... Of course the non-peak shifts were only staffed with "operator" level employees with technical back-up on call. Basically management was unwilling to pay well enough to get highly technical resources to do night shifts, especially if they had families.

1
vote

One thing you're not asking - when you have an incident, how long is acceptable until someone actually looks at it? And will this impact your regular maintenance windows?

I saw a shop go from exclusively 8/5 to 50% 12/5 and 50% 24/6 over a gradual process (a couple years) without adding staff. The new issues created by this were somewhat mitigated by getting more redundant hardware, better monitoring (especially alert delivery).

The real problem is that everyone had come to expect a large maintenance window during convenient hours of the day. This was handled by having more people work from 8am-8pm, more people scheduled to work Saturdays and Sundays, and three-day weekends being considered an opportunity do to maintenance that had previously been done after-hours on normal days.

Another problem is that everyone had come to expect that only unusual problems, extremely severe problems, or problems with a certain small handful of systems, needed to be handled during the middle of the night or the weekend. This was handled by never defining these requirements, so that a severely sleep-deprived individual could choose to ignore a particular issue if he was unable at the time, without fear of repercussions. (Although it could cost the company a great deal).

People became increasingly burnt out and disgruntled. But IT management was able to hide these problems from their management, looking like heroes able to do more work with less staff. Upper management never understood that the decreased maintenance windows for hardware maintenance were a serious problem, because IT management would not admit that this caused an issue.

The environment became increasingly pathological and I ended up leaving due to IT management's habit of lying about requirements and issues all the way up the chain. The last I heard, the remaining staff is feeling more burned-out, more ill-treated. People are losing their families becuase three-day weekends are considered a time to do extra work, and because of the expectation of working such long hours that their family is asleep when they return home. (only to do more work anyway).

However, one group started offering secret "comp days", i.e. if Saturday is scheduled for 18 hours of work, then Monday could be scheduled as a weekend day. This had to be done in secret for fear of management finding out that extra work requires extra people.

The company's profits are increasing, and the salary of the IT management is increasing.

Vacations were surprisingly not a problem due to careful scheduling and a diverse staff that did not share many religious holidays.

It seems like an overall success to me.

0
votes

I admin 75-100 servers (physical and virtual) in a 24x7x365 environment. It's a financial services company with worldwide clients, so the service itself can have no downtime. We've got 4 physical sites (2 datacenters, 2 offices) with a small userbase (around 20 users total, varying from very technical to very non-technical). There's around 40TB of SAN storage and 5 WAN links.

I feel overworked.

0
votes

I've worked in multiple shops with multiple ways of handling things. The first was by far the scariest but seemed to work...

We were a small shop of 4 admins that was on call for our customers that bought the extend 24x7 support contracts with their appliances. The 4 admins would rotate on-call nightly M-TH and then F,Sat,Sun was 1 guy which rotated through the 4 admins. Typically, if you were on call on Monday, you had the F,S,S shift but then weren't on call again until Wednesday the next week, and then Tuesday the following week etc. The calls after 8 PM went directly to a paging agency that would field the calls and page the on call person after filling out a trouble ticket. At first, it seemed like it was going to be a complicated system and we would never be getting any sleep. But for the most part, we had very few pages. Depending on the stability of the code or a flare up the issues might be in the range of 1-2 a night. But typically it was 4-5 per week. Compensation was strictly in time off. If you worked all night, you weren't expected to be in the next day. But being such a small shop, you never were fully away from work.

My second experience with an on call 24x7 situation has been MUCH better. We have a 24x7 call center/NOC staff that takes all customer calls and server pages. Based on the issue, they will call the on call person for the appropriate group. My group does week long rotations. With the week schedule and the size of my team (6 guys) I'm on call every month or so. I've been paged 2x this year while on call so its a pretty good average. The compensation is the same as the last, if you work well into the night, sleep in, take your time getting into work.

0
votes

Once you offer it, people will start using it.

24/7 coverage is challenging. 168 hours a week, every week. I've done it in other jobs but not in IT. In theory you can do it with 5 or 6 people, but 8 is more realistic. You also have the challenge on dividing up the shifts and weekends. Plan carefully.

0
votes

We have an 8 person shift layout.

  • Two people on each shift
  • Each shift is 12 hours with 1 hour for lunch
  • Each shift is 7 to 7 (7AM to 7PM, 7PM to 7AM)
  • Team A works days from Monday to Wednesday and every other Sunday
  • Team B works nights from Monday to Wednesday and every other Sunday
  • Team C works days from Thursday to Saturday and every other Sunday
  • Team D works nights from Thursday to Saturday and every other Sunday
  • Teams rotate every 4 months