Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We are a small company, so we have a small closet with one server rack. This closet has very good ventilation, so that's not a problem - but today the air went out on this side of our company suite. It didn't occur to me until just now, that means the closet is very hot, and indeed I came in to find the servers all with fans on full blast and the temperature in there quite hot, and I haven't yet checked if all the servers are up.

I opened the door, but for security purposes I can't leave it open so I'll have to close it when I leave in an hour or so. All the servers are running Windows; there are a couple towers (those are probably cut off by now from the heat, I would imagine) and basically the rest of the rack is full of rack servers.

I need to keep these servers running in the morning. They contain all our vital functions - network data storage, email, intranet, accounting, etc. Not much can be done without them up.

What do I do??

share|improve this question
    
If you only have an hour to sort this solution out, and you can't leave the door open (which would be my first choice), then I doubt there's much you can do on such a limited time frame. Others opinions may differ though. –  Mark Henderson Apr 6 '11 at 0:55
1  
For what comfort it may be worth the morning after - datacenterknowledge.com/archives/2008/10/14/… - Servers can survive a remarkable amount of heat (and humidity changes) with “no consistent increase” in failure rates... –  voretaq7 Apr 6 '11 at 18:01

8 Answers 8

up vote 3 down vote accepted

We had a similar issue; server room ambient temperature was about 33 degrees if I remember. Was like that for at least 2 or 3 days. No issues with the servers, various temp readings on the server's chassis/core/etc. were maybe 8-10 degrees hotter than normal, but no errors, no shutdowns. We ended up putting a big fan facing out from the server room to draw out the hot air, however.

Monitor the temperatures closely, react accordingly (turn them off or at least non-critical servers).

EDIT

Asked co-worker, he said the hottest it ever got was 33 (and dropped to 28-29 when we put the big fan in the door).

share|improve this answer
    
I just went and found a fan. The servers have been surviving for 7-8 hours without air, so they should be okay. Your story is much less extreme than the others, thanks for the reassurance, haha. :) - now I'll see if they stay up under load tomorrow... –  Ricket Apr 6 '11 at 1:22

Your options are limited as you've painted yourself into a corner. Here are your options.

  1. Leave the door open and run numerous room fans on them to circulate air. Leave servers running for access to everything. Pray servers don't die or someone doesn't steal them. Prepare for fitfull nights sleep.
  2. Close door and leave servers running. Pray harder that servers don't die. Prepare for not sleeping at all tonight. At least servers won't be stolen, besides, who can carry something that hot?
  3. Close door/power down servers. You'll sleep well tonight. Servers won't die and nobody will steal them.

To me your only real option is to power everything down. If your company cannot operate in any way, shape or form without their computer systems up and running, then it's time for them to invest in redundancy.

You could also decide on leaving just one server running. You could even mention that to management. "I can leave one server running, which shall it be?" and it makes them feel like they're part of the solution.

share|improve this answer
1  
+1. I especially like letting management make the hard choice. It will put you in a better place when you have the "server room REALLY needs cooling" discussion tomorrow. ;-) –  tsykoduk Apr 6 '11 at 1:36
1  
I'm really curious to know how this morning went and if the servers survived the night. I hope OP updates us. –  GregD Apr 6 '11 at 16:54

You could look into renting a portable HVAC or you could just turn off all the servers before you leave. If they over heat and cause component failure, they will end up being down longer than if you risk leaving them up.

share|improve this answer

How long has the ventilation been out? If the time between it dropping off and when you opened the door is longer than you'll be gone, the temperature already found it's max and the servers may live the night.

Try to cut down heat where you can anyways:

  • Turn off any box you can
  • Turn off any software you can
  • ensure your backups are up to date

Bonus: find random lowly paid minion, offer pizza, overtime, and a sleeping back to stay with the open door.

share|improve this answer

They might turn themselves off. Check to see if thermal shutdown is enabled (if you can get to that setting without rebooting.)

share|improve this answer

It's past your one hour deadline by now, but I would look into increasing the physical security of the office suite. If you're lucky, you might be able to secure the space with just a bike lock or padlock, and that will allow you to leave the door open over night. If possible, find a box fan or floor fan to point at the door. Even better if the fan is in the room blowing warm air out. The downside is this won't work if you have other staff there over night, locking cleaning staff out is likely the preferred option here.

Failing that, I would stay over night (or make my boss do it - who ever has ultimate responsibility for these machines should be the one holding the stick here).

In a nutshell: you would probably be okay letting them go in locked closet over night, but the odds that some could go really wrong are significant. Do you really want to leave this to chance?

share|improve this answer

What do I do??

Since your immediate options are limited to fanning the closet and hoping for the best, and you've done that, I would think about what to do tomorrow. If you have to keep the servers in a "closet", and assuming they do something important for your company, use the next week or so to work out a risk management strategy. Figure out what the worst case scenario would be (ie: servers stolen with credit card and social security numbers, liability exceeding 10M). Find small, affordable steps that can guard against whatever that worst-case is, while protecting the servers from damage. Some options might include backup power for a small HVAC system serving the closet. or even an installed ventilation fan connected to a thermostat (would do more than sitting a fan near the closed door). It may be a good time to talk to management about these things.

share|improve this answer

The inevitable answer, backported from the future when this will seem common-sense:

Use a public cloud such as AWS, Rackspace, etc.

share|improve this answer
    
While humorous, this doesn't really help the OP. –  GregD Apr 6 '11 at 1:56
    
No, but it is a suggestion for an action plan beginning the day after. Once this concrete problem is solved today, start migrating to a public cloud tomorrow. –  yfeldblum Apr 6 '11 at 12:53
1  
Migrating to the "cloud" comes with it's own set of problems and issues, which could very well make the A/C going out seem like a walk in the park. –  GregD Apr 6 '11 at 13:20
    
The inevitable answer from a couple decades ago when everything was "cloud-based" in the sense of terminals on a mainframe: Use personal computers such as the Apple II or an IBM PC. (oh how we've come full-circle) –  Ricket Apr 6 '11 at 15:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.