6

We are a small company, so we have a small closet with one server rack. This closet has very good ventilation, so that's not a problem - but today the air went out on this side of our company suite. It didn't occur to me until just now, that means the closet is very hot, and indeed I came in to find the servers all with fans on full blast and the temperature in there quite hot, and I haven't yet checked if all the servers are up.

I opened the door, but for security purposes I can't leave it open so I'll have to close it when I leave in an hour or so. All the servers are running Windows; there are a couple towers (those are probably cut off by now from the heat, I would imagine) and basically the rest of the rack is full of rack servers.

I need to keep these servers running in the morning. They contain all our vital functions - network data storage, email, intranet, accounting, etc. Not much can be done without them up.

What do I do??

2
  • If you only have an hour to sort this solution out, and you can't leave the door open (which would be my first choice), then I doubt there's much you can do on such a limited time frame. Others opinions may differ though. Apr 6, 2011 at 0:55
  • 1
    For what comfort it may be worth the morning after - datacenterknowledge.com/archives/2008/10/14/… - Servers can survive a remarkable amount of heat (and humidity changes) with “no consistent increase” in failure rates...
    – voretaq7
    Apr 6, 2011 at 18:01

8 Answers 8

4

We had a similar issue; server room ambient temperature was about 33 degrees if I remember. Was like that for at least 2 or 3 days. No issues with the servers, various temp readings on the server's chassis/core/etc. were maybe 8-10 degrees hotter than normal, but no errors, no shutdowns. We ended up putting a big fan facing out from the server room to draw out the hot air, however.

Monitor the temperatures closely, react accordingly (turn them off or at least non-critical servers).

EDIT

Asked co-worker, he said the hottest it ever got was 33 (and dropped to 28-29 when we put the big fan in the door).

1
  • I just went and found a fan. The servers have been surviving for 7-8 hours without air, so they should be okay. Your story is much less extreme than the others, thanks for the reassurance, haha. :) - now I'll see if they stay up under load tomorrow...
    – Ricket
    Apr 6, 2011 at 1:22
8

Your options are limited as you've painted yourself into a corner. Here are your options.

  1. Leave the door open and run numerous room fans on them to circulate air. Leave servers running for access to everything. Pray servers don't die or someone doesn't steal them. Prepare for fitfull nights sleep.
  2. Close door and leave servers running. Pray harder that servers don't die. Prepare for not sleeping at all tonight. At least servers won't be stolen, besides, who can carry something that hot?
  3. Close door/power down servers. You'll sleep well tonight. Servers won't die and nobody will steal them.

To me your only real option is to power everything down. If your company cannot operate in any way, shape or form without their computer systems up and running, then it's time for them to invest in redundancy.

You could also decide on leaving just one server running. You could even mention that to management. "I can leave one server running, which shall it be?" and it makes them feel like they're part of the solution.

2
  • 1
    +1. I especially like letting management make the hard choice. It will put you in a better place when you have the "server room REALLY needs cooling" discussion tomorrow. ;-)
    – tsykoduk
    Apr 6, 2011 at 1:36
  • 1
    I'm really curious to know how this morning went and if the servers survived the night. I hope OP updates us.
    – GregD
    Apr 6, 2011 at 16:54
1

You could look into renting a portable HVAC or you could just turn off all the servers before you leave. If they over heat and cause component failure, they will end up being down longer than if you risk leaving them up.

1

How long has the ventilation been out? If the time between it dropping off and when you opened the door is longer than you'll be gone, the temperature already found it's max and the servers may live the night.

Try to cut down heat where you can anyways:

  • Turn off any box you can
  • Turn off any software you can
  • ensure your backups are up to date

Bonus: find random lowly paid minion, offer pizza, overtime, and a sleeping back to stay with the open door.

0

They might turn themselves off. Check to see if thermal shutdown is enabled (if you can get to that setting without rebooting.)

0

It's past your one hour deadline by now, but I would look into increasing the physical security of the office suite. If you're lucky, you might be able to secure the space with just a bike lock or padlock, and that will allow you to leave the door open over night. If possible, find a box fan or floor fan to point at the door. Even better if the fan is in the room blowing warm air out. The downside is this won't work if you have other staff there over night, locking cleaning staff out is likely the preferred option here.

Failing that, I would stay over night (or make my boss do it - who ever has ultimate responsibility for these machines should be the one holding the stick here).

In a nutshell: you would probably be okay letting them go in locked closet over night, but the odds that some could go really wrong are significant. Do you really want to leave this to chance?

0

What do I do??

Since your immediate options are limited to fanning the closet and hoping for the best, and you've done that, I would think about what to do tomorrow. If you have to keep the servers in a "closet", and assuming they do something important for your company, use the next week or so to work out a risk management strategy. Figure out what the worst case scenario would be (ie: servers stolen with credit card and social security numbers, liability exceeding 10M). Find small, affordable steps that can guard against whatever that worst-case is, while protecting the servers from damage. Some options might include backup power for a small HVAC system serving the closet. or even an installed ventilation fan connected to a thermostat (would do more than sitting a fan near the closed door). It may be a good time to talk to management about these things.

-4

The inevitable answer, backported from the future when this will seem common-sense:

Use a public cloud such as AWS, Rackspace, etc.

4
  • While humorous, this doesn't really help the OP.
    – GregD
    Apr 6, 2011 at 1:56
  • No, but it is a suggestion for an action plan beginning the day after. Once this concrete problem is solved today, start migrating to a public cloud tomorrow.
    – yfeldblum
    Apr 6, 2011 at 12:53
  • 1
    Migrating to the "cloud" comes with it's own set of problems and issues, which could very well make the A/C going out seem like a walk in the park.
    – GregD
    Apr 6, 2011 at 13:20
  • The inevitable answer from a couple decades ago when everything was "cloud-based" in the sense of terminals on a mainframe: Use personal computers such as the Apple II or an IBM PC. (oh how we've come full-circle)
    – Ricket
    Apr 6, 2011 at 15:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .