Advice on moving a machine room to a new location?

Question

Our company is moving to new offices in a couple of months, and I am responsible for looking after the move of the development servers in the company. most of the dev equipment is in 5, 42U cabinets + rack for switching/routing equipment. How do most people do this sort of thing? Move the cabinent whole or extract the indvidual components and move the racks empty.

any advise on prep and shutdown before the move would be welcome

Answer 1

We have done a couple of moves with racks. Here's what I took away from the experience:

Before:

• label everything. Both ends. And the rack slot it's plugged into too.
• diagram everything.
• if you have important data, schedule a full backup to finish 6-12 hours before the move. At a minimum, validate the last full backup you have.
• plan to take everything out of the rack during the move. At a minimum take out half the mass, from the top down -- this means leave the top empty Don't expect movers to move a top-heavy rack. In a rack with a UPS at the base that means you are taking out probably 2/3 of the servers. Anything which is left racked MUST have rails and be secure in the rails (less than 0.5mm movement, and tolerant of any orientation -- including upside down and face down -- was our margin). We've moved full racks, and half-full racks, and the racks always need to be tilted/rotated, and it is downright scary to watch your livelyhood being dangled over a concrete loading dock.
• in our experience, front and rear doors are more decorative than load-bearing and leaving them on can make moving the racks very awkward because all the good hand-holding points are unavailable. Plan to remove the doors during the actual move. This forces the movers to lift, pull, push, and tug the rack by its strongest parts. Usually side panels can stay on, but be prepared to remove them if the movers think having them gone would help.
• hire someone with an insured truck to to the transportation. DO NOT do it yourself. You might think you are insured, but your insurance company will probably think otherwise. The second last thing you want is to be involved in a three-way blamestorm between your employer's insurance company and your insurance company.
• hire someone insured to physically pick up, move, put down, and (during an accident) drop your gear for you. DO NOT do it yourself. The last thing you want to do is hurt yourself or be involved in a co-worker's injury.
• make sure your insured someones specialize in high-tech moves and can supply blankets, anti-static bubble-wrap, and if necessary crates.

During the move:

• backups are good, right?
• do the wire disconnect yourself. Pull ALL wires that are disconnectable. Each wire should be labelled and go into a box that itself is well labelled.
• let the strong, well insured guys do the server extractions from the rack. Supervise the wrapping personally.
• stay out of the strong, well-insured guys' way when they do the loading and unloading.
• supervise the unwrapping and re-insertion into the racks. If there is ANY question about a server's condition, mention it, and note that it was mentioned.
• If the job is taking a while, have coffee and/or doughnuts available for the guys. If appropriate (and if they've done a good job), cold beer for after the job goes well too.

Physical bring-up:

• double your time estimate for getting things back on track.
• have a build-up plan. A good build up plan includes a checklist of all services, servers, and a test plan.
• validate, validate, validate.
• check any servers which may have been damaged in transit as noted above.
• don't let management cheap out when it comes to food if you are working through meal times. Hungry techs make mistakes. If the bring-up is going to be long and complicated, schedule time to get out of the room for 90 minutes to go get a real meal.

After:

• a cold beer for you may be appropriate, too.

Answer 2

I'm with Kevin on unracking and disconnecting. Racks that I've worked with that are built to ship loaded typically have additional bracing and must be attached to a fixed base (like a pallet) during shipping. Usually the servers have some kind of locking mechanism, too. If you're not moving something that isn't built to be moved loaded don't.

I would add this to Kevin's answer:

Part of documenting is knowing why the various connections and cables are there. You can make all the notes, take all the pictures, etc, that you want to, but at the end of the day if you don't understand why something is there you're running a risk of not knowing what to test to establish, after the move, that you "got everything".

This is a golden opportunity to, as you're documenting the "what's plugged into what" of the move, ask yourself "why is this plugged into this"?

I've had too many cases of the "I'll just patch this here and temporarily change a VLAN assignment" that turns into a semi-permanent configuration that no one documented come back to bite me when moving equipment years later.

Before any equiment goes into a personal vehicle, find out how the company's insurance is handling claims in the event of damage during transport.

Back everything up before you begin moving. Assume that all the equipment will be destroyed during the move because (if you're driving on public roads, at least) it could be. It goes w/o saying that the backup media should not move with the infrastructure gear!

Answer 3

Extract the individual components and move the racks empty. This also has the advantage of allowing you to reconnect things in an optimal configuration and get your wiring exactly the way you want it. Just be sure to document everything before, during and after. Take digital pictures to help.

Answer 4

We relocated two racks of servers about 60% full about 3 years ago.

It was the most frightening thing I have witnessed in a long time.

Everything was OK in the end, but the movers had to tilt the rack to get it out of the door at our old location. They were very big and very strong, and lucky for me the rack did not drop -- but I can tell you, I was stressing just watching it.

If you must to move the racks full, make certain that you can roll the racks through the doors with out having to tilt them at every door that you will encounter during the move. if you have to tilt, the forget it and start unloading servers :-)

I would personally advise on either emptying the racks first, or like Kevin suggested, getting new racks at your new location. This would also perhaps allow you the flexibility to move the systems over in stages, rather than all at once. Of course that might not work given your specific environment.

Answer 5

Buy an awesome label maker, and label everything like you're going insane. Then take it apart and move it piece by piece.

That is all.

Answer 6

You might also consider a partial move. For example, you might move backup equipment first - if it's a live backup, you might even redirect users to the backup systems while the primary systems are being moved.

You might also start the move with one team's systems, or part of several teams' systems first. Fix the problems that those systems experience while the team can still work with their machines in the old room.

Answer 7

before you move the equipment, prepare what you can ahead of time. if you want a tidy rack, plan where each server will go and cut new network cables to length and have them labeled. install any extra switches and pdu's as well. this way when moving you only have to take power cords and servers.

when moving, if you are moving production equipment, domain controllers, email servers, terminal servers or anything that users generally expec 24x7x365 uptime on, get those shutdown last and booted up first. Move equipment later in the night and if you have a lot to move, get a couple extra hands (friends will work for pizza) and borrow/rent a van. If your boss is paranoid about security, contact the local sheriff about a police escort.

get your business critical equipment up first and tested. secondary equipment like backup servers or backup dc's can wait a bit. doing work overnight gives time to bring things up and tested without interfering with the

these are a few things i learned moving 2 racks worth of servers in one night.

Answer 8

I am all for the already existing answers about labeling and unracking. The other thing I would add is if it is possible don't move all 5 racks of gear on one day. That way you only have to trouble shoot one rack worth of gear in a single day.

Answer 9

I just did this a few months ago. Here's my tips:

1. Get a ton of sleep. I ended up working 32 out of 36 hours starting at 8am Friday and ending at 10pm Saturday.
2. We were fortunate enough to have most everything setup in an HA cluster, so we could move half the equipment to the new DC at our leisure during the day, switch DNS to the new DC, then move the remainder of the equipment as DNS is propogating.
3. If you're going to change IP's, crank down the TTL on your domain ahead of time so propagation happens faster.
4. Make a very detailed checklist of steps laid out in chronological order. You will be physically and mentally exhausted, having a checklist leaves you little room for mistakes.
5. The most important part: have backups and service plans on everything. We had no bumps or slips during our move, but we lost 2 hard drives, one stick of RAM, and one DRAC card in the move. Probably vibration from being moved, but it could have been simply power cycling it all.

I did nothing but prep work for two weeks prior, and it paid off big dividends.

Answer 10

I've previously had to deal with powering off a data center with well over 200 systems, but without the move part. (To install a load transfer switch and to replace all of the batteries in the UPS room) I'll have to deal with moving between buildings sometime next year, though, but it's only 7 racks.

So, on top of what others have already said:

• Of course, there's the backups, and testing the backups. If you can, get extra hardware, or repurpose whatever test hardware so you can pre-position critical services. (DNS, NTP, syslog, anything else other systems might need to come back up.) You might also considering boxing up whatever spares you have (cables, drives, power supplies, etc) so they're ready to ship, too.

• Have all of the power tested before you're ready to move -- and make sure it's not just 'yes, the hot's hot', as your UPSes will balk at an open neutral, and then you've got to try to track down an emergency electrician.

• If you have a lot of network connections going between racks (eg, to a single 'networking rack', consider having all of the network drops already prepared for the racks ... punch down (and test!) patch panels that you can slap in the rack and cable up to, and avoid dealing with the long network runs on top of everything else.

• Look for single points of failure, and make sure you have a spare or can bypass it -- we had our terminal server die, which meant we had to take the few wyse terminals we had to build two crash carts that we could drag from server to server to check on the ones that weren't coming up clean.

• Make sure you have all of your vendor's support numbers available. If you have a high level support contract, you might see if they'll send out an engineer with spare parts to be there in case things go wrong. Some might even require their personnel to be there to oversee things or they'll void your support contract.

• Recruit a manager (someone who's got a vested interest with enough clout in the company, but isn't directly needed for bringing everything back up) to be in charge of keeping other management folks from bugging your team. They don't seem to get that when you're trying to deal with things that went wrong, every time they ask you how long it's going to take, they're breaking your concentration. The manager can also be responsible for making sure that people have food and their preferred caffeine source, and be able to pull people out when they're dead on their feet and need they're doing more harm than good (although, that take-away's from a mail system rebuild with some of us working 12-16hr days for 16 days straight)

• If your sysadmins have long commutes, get the company to rent some nearby hotel rooms for them so you're not losing 2-3 hrs per day to commuting. (again, from the mail system rebuild).

• If you're really paranoid, I'd ask for two moving trucks or take two trips, and don't transport primary systems with their backups. Or if time's an issue, break down the most critical racks first, and send them out in the first trip while someone else finishes breaking down the less critical racks for a second trip.

• For the dollies, the moving company should be prepared (as others have said, use a moving company), but you can also rent the ones for moving refrigerators from a rental supply company if they're just being used for insured transport, and not prepared to move a half-full rack.

• If you have enough people to be working staggered shifts to get things done, make sure there's some overlap so people know what's really going on, and not just blindly following a checklist.

• If you have cabinets and not two post racks, and the doors close cleanly without crushing cables, I'd probably try to leave as much cabled up as I could, other than what had to be pulled to lighten the top of the rack. (okay, the one time I disagree with what's already been said)

• ... and then plan on some recuperating time. Thanks to long days preparing for it as management only giving us two weeks notice (which they had known about for months, just didn't bother telling the sysadmins), our Xyplex going out, and spending hours in the machine room having to shout over the fans of the two EMC cabinets, I got sick and was out for a week. It'd probably also be worth holding back a sysadmin or two on Sunday, so they'll be fresh to work on Monday dealing with whatever trouble tickets come in from the users.

addition:

So, after doing another move, a few things to consider:

• If you're going to be moving during a cold time of the year, insist that the moving company bring a heated truck, and won't leave stuff out on a cold loading dock where you'll have sudden thermal change that might crack solder joints. (we moved mostly full racks, and wrapped them in bubble wrap and moving blankets to insulate them)

• check on the size of the truck the moving company's going to use, so they don't bring something too large than can easily get to your loading dock.

• place cones or other items the night before to make sure that no one decides to park where they'd impact the ability for the semi that the moving company brought from being able to get to the loading dock.

• if one of the hardware vendors insists on bringing in their own moving truck, and having a packing crate delivered to move the rack in advance, ask them if their moving company is planning on bringing in a truck that can handle items as tall as their crate. (luckily, we were only moving within our campus, so they relented to letting the rack move uncrated)

• if you have strange requirements (like the ramp for the raised flooring extends through the doorway, and some of the racks are too tall to make it through the doorway if the ramp's in place) make sure to get the actual people who are going to do the packing up of the racks to come and survey the situation, not just the manager. (we lucked out, as they were moving a machine room on the floor above a week before us ... so we happened to find that the way they were planning on lifting the racks down from the raised floor wasn't going to work, so two of us tore out a section of the raised floor before the day of the move)

Answer 11

Move it empty (or get new ones installed and cabled in the new location first). Servers are staggeringly heavy. Even if you wanted to I doubt you'd be able to move 42U of densely-packed electronics in one go.

Answer 12

If you are going to move the entire rack, consider something this: http://powermate.info . We purchased one that can handle up to 1500 lbs, and it is a lifesaver for those short doors!.

Answer 13

Print your move docs (and digital pictures) and burn them to a CD for paranoia's sake. Put an extra copy somewhere safe. Your laptop might fail.

Also, make a poster-sized master checklist for each end of the move and put them on the wall at the appropriate site. Then when someone asks for status, all you have to do is point.

Use color-coding and icons wherever possible on all your labeling/docs. Think Garanimals.

Don't forget to apply labels to the racks themselves where you've removed something.