18

Property management at my organization has informed me that our building will be losing power for 4 hours tomorrow. I need to be prepared for this event (we're a small organization, i'm young, therefore I am I.T). What sorts of things do I need to be aware of.

I am planning on going in and shutting down all machines and printers. Will this cover me? We have a managed switch. Does it need to be shut down? Do I need to disconnect plugs in case of a surge?

Seems like I'll be covered all around if I just unplug everything. Thanks for any insight though.

Improve this question

7 Answers 7

Reset to default
32

Before the outage:

  • Power everything off - workstations, servers, printers, switches, the works.
  • Turn off your UPS' so they don't panic when power is lost.

After outage in this order:

  • Turn on UPS
  • Turn on networking (router, switches etc)
  • Turn on servers
  • Turn on workstations
  • Turn on everything else

Have a test plan ready so you can test important functionality is working

  • Internet connectivity
  • Email, printing etc

If possible, have a laptop with a separate network connection handy (ie: you can get to the internet without your work router working). That way you have a way to ask for help here if something goes wrong with the networking when it comes back up. :)

You should be fine though - the fact that you took the time to ask here shows you already have the requisite "clue" required for IT support!

Improve this answer
5
  • Thansk for the detail.
    – user8539
    Jun 6, 2009 at 18:02
  • 2
    +1 for the laptop with its own internet access. In addition to calling for help, you can use this to test that outside items are available beyond your network and inside resources aren't. That was a part of your test plan right? Riiight?
    – Rob Allen
    Jun 6, 2009 at 18:36
  • 4
    +1. A couple of things: if you have the login for the managed switch, make sure you log in and save the configuration. Also, see if you have the cables required to connect to the console port of both your router and switch. Many of them require serial (DB-9) connections, so if you have a recent laptop, you may need a USB->serial adapter. Good luck.
    – Murali Suriar
    Jun 6, 2009 at 18:43
  • 2
    +1 and also figured I'd repeat what some of the posters have already said below about unplugging some of your valuable hardware from the WALL if you don't know exactly what the facilities people are going to be doing to the power lines. Better safe than sorry!
    – KPWINC
    Jun 7, 2009 at 6:25
  • Yeah, definitely isolate as much from the supply as you can. However, don't have anything running on the UPS when it's isolated, as you NEED the ground connection if anything is online.
    – Chris Thorpe
    Apr 24, 2011 at 6:31
12

Having just done through a datacenter shutdown in the last week, this is fresh on my mind ;). Yes, shutting everything down needs to be done. Some things can tolerate having the power yanked out from underneath them, and they typically can be identified by not having a power switch on them. Depending on what the heck the facilities people are doing, you may want to physically unplug higher value assets from the wall.

Sometimes you can get sizeable power-surges when power returns, and that can kill gear. Also, be aware of your start-up load. EVERYTHING powering on at once may be enough to pop breakers and bring everything down again. That sucks. Try for a staged power-on to get around this.

EDIT: We had a case where the transfer switch for the generator didn't fire, and the UPS ran out of batteries. When utility came back, everything powered on at once. 20 seconds later the UPS hit Overload and dropped load from the room again. By then enough techs had arrived on site (it HAD to happen during a weekend) that they managed to get enough stuff turned off at the rack-switches that when utility came back again the room didn't drop again. That yo-yo cost us a lot of hardware, and it was week recovering from some of the damage.

Improve this answer
2
  • 3
    Big up to that staged power-on
    – Andy
    Jun 6, 2009 at 18:20
  • 1
    Definitely agree- make sure that you turn on the servers one at a time, and let them get the disks fully spinning before you bring up the next one. Otherwise, you can easily pop the breaker- I've done it myself.
    – Tim Howland
    Jun 7, 2009 at 3:32
5

The only thing I'd add is to make sure you have all your support contracts and replacement parts easily findable. I'm sure you do, but double and triple check that. If you've had servers that haven't powered down in a long time you'll occasionally find issues with all the moving parts (hard drive bearings/motors, fan motors and bearings and some really silly stuff that we normally don't think about).
If you can be there for the duration, make up a list for your important servers including their service tag, configurations, where the backups are and any people that might be responsible for apps on each machine. Tape that up on the side of the server in question, or have it in a binder in the datacenter for when you start trying to bring things back up. You don't want to have to keep running all over your office if something is broken.

Improve this answer
3

First of all, you've got good backups, right? :) Always a good idea to make sure that you're all backed up, just in case a disk decides to not spin up, etc. when you bring everything back online. Nothing like breathing easy after a power outage then realizing that one critical SQL server never came back. :)

Second, you're right, gracefully shut down and turn off all servers, etc.

If it's not a modern building and/or you don't have a UPS in-line with surge protection, (which it sounds like) or anything like that, it's always better being safe rather than sorry, and I would suggest unplugging equipment. Bringing power back online could result in a power spike.

Also, a little FYI; bring your core servers down last, (such as DNS, etc.) You don't want to bring down your DNS servers and find yourself unable to resolve the rest of the servers if you're shutting them off remotely :)

When bringing everything back online, bring the network equipment up first, then your core critical servers, then finally your app/file servers.

Good luck!

Improve this answer
2

Shut everything down, including switches and routers, and either unplug or turn off at the wall socket.

If you're worried about surges, is there a budget to get a few small UPS to cover the most critical equipment?

EDIT: I like neobyte's suggestion to have a "test plan" for when you bring stuff back up. Even if you just have one in your head, and a firm order to follow to restore services quickly you'll be fine. Make sure people know that you're working to a plan and hopefully they'll understand why switching on their workstation before you've looked at the servers and switch isn't going to help!

Improve this answer
1
  • We have a small UPS covering the vital stuff. I think I will do as suggested and disconnect everything from the walls as well.
    – user8539
    Jun 6, 2009 at 17:58
2

Inter-server dependencies / server boot order will be best laid out ahead of time, too. For example, if it's a windows network, power up DC's first, then database servers, then exchange, etc. This is a great list for your documentation, in any case.

Minimizes restarts and logfile spam.

Improve this answer
1

Too late for a rehearsed shut-down scenario .. I suggest you do all you described, then be around when power comes back ..

don't forgett to post a note explaining to your collegues what happened .. most of them will have "forgotten" to turn of their computer and might want to know why the system needs to be booted...

Improve this answer
1
  • Great suggestion to leave an explanation for colleagues. Thanks.
    – user8539
    Jun 6, 2009 at 18:00

You must log in to answer this question.