3

How does your company handle monitoring and administering servers after-hours and during holidays? Do you require that some come into the office? If not, how do you insure that staffing is available to handle any issues that come up?

Presumably it would be possible to give out mobile data cards, but is this a practical solution in the real world? Are employees compensated for this in any way?

8 Answers 8

4

Simple! Draw straws... the guy who draws the "short straw" loses. ;-)


On a more serious note. We have a monitoring system that pages all of the sys-admins whenever something goes down.

We have a good bunch of people who are on top of things. Nobody is compensated for logging in with VNC and restarting the mail server but people ARE compensated when say a hard drive crashes and it requires a trip down to the colo and a RAID rebuild or something.

Most companies I know of do pay for the sysadmin cell phones or gives a good discount. Depends on if the phone is for personal use as well. I've also seen company issued laptops and in some rarer cases broadband internet cards.

In the REAL world... there's a PC around or a hotspot (be sure to THINK and encrypt your stuff traffic) around somewhere.

The reality is... right now most people are happy to be gainfully employed right now and when you love your job... its just PART of the job.


The only REAL problem I've heard of (in regards to perks) was with an employee who was absolutly OBSESSED with his Red Swingline stapler. Management wouldn't listen, took the stapler away, and as a result the employee lost it and burned the whole place down!

(tongue in cheek reference) ;-)

3
  • Thanks for the information. I tend to agree with the overall compensation strategy (regular salary + exceptions for exceptional conditions). PS - The cockroaches in the cellar here aren't really that bad.
    – jsight
    Jul 15, 2009 at 18:09
  • +1. As for the sysadmin cell phones for personal use: the reason companies have cared about this may be going away. The IRS requires companies to report on cell phone usage by employees for both business and personal use, causing a management nightmare. The current administration has encouraged the IRS to do away with this rule, and the IRS is obliging. Cell phone ploicies should be a lot clearer from here on out.
    – RascalKing
    Jul 15, 2009 at 18:21
  • We do essentially the same thing. I am in New York City, which means that it is almost impossible to be more then 5 blocks from a hotspot (usually a Starbucks). So my company pays for a monthly subscription to a service called Boingo, which allows you to log on to most pay per use wireless hotspots. That way, when I am on call, all I have to do is carry a laptop (or even my iPod touch) to be able to log into servers remotely. Jul 15, 2009 at 18:24
2

Right now, I'm the only admin at my company, so when there is an issue, I get an SMS & email from the Nagios server that monitors the affected host.

I'm soon to be hiring a junior admin, and when that happens, we're going to get to do on-call rotation (once the person is trained and ready, anyway).

My cell phone is paid for (as will the new admin's), and we have a certain flexibility in terms of when I come in and leave. As long as I don't abuse it, it works fine.

We're a small company with 17 people and somewhere between 50-70 servers that runs 24x7x365 automated processes.

6
  • I'm in almost the exact same boat. =p being on call 24x7x365 sucks, but its worth the job security, especially in times like these.
    – grufftech
    Jul 15, 2009 at 18:09
  • What do you use for generating SMS alerts from Nagios?
    – RainyRat
    Jul 15, 2009 at 18:11
  • 1
    Not sure what he does, but I use the built in Nagios SMS notification function along with the email address for my phone which is [email protected], I have never run into a cell phone provider that doesn't have a similar structure for sending SMS via SMTP. Jul 15, 2009 at 18:22
  • Indeed, I do what Catherine suggested, which is send an email to [email protected]. Incidentally, I have two nagios servers on WAN separated networks which watch each other and alert in case one dies. That way I'm not in the cold if one of the WAN connections fails. Jul 15, 2009 at 20:44
  • Something to be aware of when using your cellphone. The cellphone companies have been known to filter certain messages that 'look spammy', and this happened with some of our nagios notifications. The company first claimed they didn't do blocking, then admitted it, but refused to remove the block. Using dedicated pagers works much better and we have never run into this kind of issue.
    – Mark
    Jul 15, 2009 at 21:22
2

When I ran a group of admins the issue of who was oncall was always a concern. At the time I felt that it was all part of the job of being an admin, but alas, in hindsight, while I loved the job, it didn't mean they did.

Not paying a stipend or some other acknowledgement of required ad-hoc hours (either onsite or remote) is just a slap-in-the-face. Regardless of budgeting, folks asked to carry pagers and response in a timely manner should be given credit either as pay, flex hours, paid cell plans, etc.

Where I work now it's a flat-fee stipend. You can spend it on your cell plan, internet connection, laptop, etc... if you do your job well, 'in theory' you wont get calls that could be avoided, just 'acts of god' ;-)

Luckily all the stuff I do now does not require on-site presence (whew).

1
  • "At the time I felt that it was all part of the job of being an admin, but alas, in hindsight, while I loved the job, it didn't mean they did." - Exactly. :) Thats a very good insight.
    – jsight
    Jul 21, 2009 at 22:43
2

We have a central monitoring server which notifies administrators of issues with their machines. We are all strictly speaking on call 24x7. So when the blackberry goes off we look and decide if we have to do something about it. In theory we're paired up with another administrator and should be able to back the other up. This works better or worse depending on the paring. So when I'm on vacation my buddy should cover my stuff and vice versa.

Blackberries are provided by the university, but we get taxed on them, so they're not totally free, but pretty cheap. We also have partial reimbursement for our broadband internet expenses. We have a VPN too. So really I don't have to be at work to work except in the most extreme situations.

We don't have an official plan for compensating people for time worked outside normal hours. In practice if you work a couple late night hours you can email in that you'll be in after lunch. We have a ton of flexibility in when we have to be here, which makes up for the lack of on call rotations.

During working hours we have a couple people who's job it is to process route and sort tickets. They also watch the queues in the evenings. For serious emergencies there is a phone, that gets called, (and is usually in the possession of one of the managers) and then the right resources are dispatched. Sad for those of us who got woke up to say "Uh- that's not mine but I can fix it if you can't find the right guy" but it works out pretty fair in the end.

1
  • 1
    +1 for pairing up to have pre-arranged backups
    – Ophidian
    Jul 16, 2009 at 3:06
1

We give pagers out and have an on call rotation. The person who is on call can take a wireless modem if required, but everybody has a home internet connection also that is used. Re: compensation, it depends on the situation. On option is to have compensation be salary based (same regardless of number of hours worked), and let potential employees know this up front and to take the on call time into account when negotiating salary.

2
  • Thanks for the insight. All of ours have access at home as well, but holidays tend to put a kink into things.
    – jsight
    Jul 15, 2009 at 18:05
  • Holidays you keep an eye on to ensure that it isn't the same person being on call during them. And for times you can't be on call (some other engagement), arrange to swap with somebody else. Most of the time an arrangement can be made that is acceptable to all.
    – Mark
    Jul 15, 2009 at 19:53
1

We use a company paid cellphone that is able to do tethering, So laptops can connect to the internet through their phone (most employees just use their home connection since it is faster)

The cellphone receives TXT messages & Emails from Nagios with alerts.

As an IT company with today's VPN capabilities we do not require any technician / engineer to actually come into the office after-hours or during holidays unless solving the issue requires them to.

4
  • We've considered those approaches, however, we've had some doubts about the quality/performance of a tethering connection. Is the tethering performance adequate for sys admin needs (mostly ssh, with occasional VNC or RDP)? What network are you using?
    – jsight
    Jul 15, 2009 at 17:56
  • As long as its on 3G or CDMA its reasonable, i pull 1.5-3mbps down on my phone, more then enough to do VNC, RDP, SSH, anything i would usually need to do at 2AM when X service fails for whatever reason. I use Verizon myself, with a VX6800. Tethering on Verizon isn't as nice as AT&T, however i find that Verizon's signal is a bit better, which is my primary focus, is being able to get online when & where i need to.
    – grufftech
    Jul 15, 2009 at 18:02
  • @GruffTech - Thanks, thats exactly the sort of information that I need. :)
    – jsight
    Jul 15, 2009 at 18:04
  • Also, most smart phones (my VX6800 and the iPhone, several others as well) offer applications that allow me to RDP right from my phone, allowing me to most of the time fix things without needing my laptop handy. (Often times if something crashes when I'm at lunch or something like that.)
    – grufftech
    Jul 15, 2009 at 18:08
1
  • We run a monthly on-call rota

  • We use Citrix and VPN extensively so 95% of problems (almost everything has Remote Console) can be resolved from any internet connected computer in the world, removing the need to physically attend an office

  • We have housed all our servers (000's) at [insert large Telco here] data centres, who provide 24/7 hands & eyes support when required

  • We utilise [insert large Telco here] Helpdesk to deal with 1st line user issues 24/7

  • We utilise [insert large Telco here] ticketing system that has escalation paths, and is tied into the on-call rota

  • We issue data cards and laptops to staff that will be in remote locations during their on-call period

1

I've worked at "it's part of the job, suck it up" companies. It got old after a while.

My current employer does a voluntary rotation. If you take the rotation, you get one week in (how ever many people are in the rotation). You get a small extra pay for being the "on-call" person that week, and time-for-time for anything you have to do after hours.

It only works if time-for-time is tracked (which is easy, since 95% of everything we do is billable back to a customer) and you are actually permitted to take it.

My current employer pays for phone/Blackberries through a corporate plan, but that's due to the nature of our business and isn't special to on-call staff.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .