269

You know, you see pictures like below and sort of chuckle until you actually have to deal with it.

I have just inherited something that looks like the picture below. The culture of the organization does not tolerate down time very well, yet I have been tasked to 'clean it up'. The network functions as it is, and there doesn't seem to be rush to get it done, but I will have to tackle the bear at some point. I get the ugly eye when I mention anything about weekends.

So my question goes, is there sort of a structured approach to this problem?

My Ideas thus far:

  • Label, Label, Label
  • Make up my patch cables of desired length ahead of time
  • Do each subnet at a time (appears that each subnet are for different physical locations)
  • Replace one cable at a time for each subnet
  • It's easier to get forgiveness than permision?

A Rat's Nest

25
  • 95
    Things like this were the reason that I insisted on seeing the infrastructure before accepting any offer.
    – gWaldo
    Aug 24, 2012 at 16:11
  • 99
    My suggestion would be to carefully document absolutely everything and keep it to yourself. Then unplug one end of every cable one weekend. Call your boss and renegotiate for a much, much better compensation package. :D Aug 24, 2012 at 16:40
  • 6
    make sure that any devices that support it are set up for dual-connections, both switch-to-switch and switch-to-device. that way you can do a lot of this work without needing downtime. i cleaned up a similar mess with only 2 hours of downtime to a couple of servers and one router (and no switches). but the whole repatching project took me 24 man hours, 20 of which were during business hours.
    – longneck
    Aug 24, 2012 at 17:10
  • 21
    As a note for future changes, make sure you set up a standard for adding/removing/changing cables. Even if you can't get everything organized immediately, you can get everything organized progressively by making sure that all changes to the network happen neatly and consistently. Also, make sure everyone working with you is on-board with the new standard operating procedure. It doesn't matter how nice you make things, everything will end up a mess again if the people who are working in the system are unaware of or don't like the new system.
    – zzzzBov
    Aug 24, 2012 at 17:15
  • 46
    Looks like we got ourselves a new 404 image for the site
    – squillman
    Aug 24, 2012 at 23:53

15 Answers 15

152

In no particular order here are some suggestions that have been helpful to me over the years-

  1. Can any of the equipment in those racks be eliminated, upgraded or consolidated? It's hard to tell what's there, but in my experience these kinds of messes tend to be aggravated by gear that should have been pulled out years ago.

  2. Once you've got some idea of the minimum set of equipment then consider how best to lay it out. The criteria here may vary, but grouping by technology type or business function might make sense. Clearly the proximity of high density devices (i.e. switches) and patch panels and such will immediately be apparent.

  3. Use cable management!!! There are both horizontal and vertical cable management solutions. Use both - horizontals around patch panels and other significant concentrations, verticals next to switches and to facilitate risers.

  4. It's always surprising, but how power cables are routed should be considered. UPS units in the bottom of racks, PDU selection and diversity all need to be considered before pulling a cable.

  5. Keep inventory of common cable lengths. It's late at night and you want to go home. A 3' cable is what's necessary but the closest you have handy is 5'. This is how these kinds of messes develop.

  6. Documenting is part of the game, but the importance of labeling cannot be overstated. With clear labels and efficient/clean cabling the number of mistakes will be vastly decreased and troubleshooting simplified.

  7. Limit who can pull cables!!! Differing styles and degrees of attention to detail can yield chaos pretty quickly.

9
  • 1
    As much as I like the idea of getting another company to do the work, I beleive I would have hard time getting the budget passed by the board since the mindset will likely be "we already have the skillset in house". Asside from that, these are great suggestions. Aug 24, 2012 at 18:54
  • 29
    +1 for inventory of cables. Unless you are very good and have excellent test equipment, buy certified cables of the needed lengths. Saves much grief
    – Dave M
    Aug 24, 2012 at 19:30
  • 3
    I completely agree on the premade cables. Especially with the advent of cat6 and beyond the tolerances are much tighter around all terminations. I can crimp a beautiful looking patch cable and can make neat 110 punchdowns on a panel, but neither necessarily implies that I've built a completely functional network until it has been actively certified. While it might seem to cost a bit more, the truth is that buying these elements as premade (and standardized) as possible saves a great deal in terms of headaches and mystery outages in the future.
    – rnxrx
    Aug 24, 2012 at 20:10
  • 10
    If you're going to re-cable this whole mess, it seems to me to be the perfect opportunity to work out a color-coding scheme. You know (or will soon know) what you have, and can figure out what categories make sense and assign colors to each, and post the color key on a laminated piece of paper nearby so everyone else who works with it can be consistent with your scheme. And you can leave the current cable colors/tones (black, white, and gray?) as "non-categorized" while you use (e.g.) blue, red, green, yellow for new cables. It'll also be easy to see what's been replaced and what hasn't.
    – iconoclast
    Aug 27, 2012 at 17:04
  • 1
    Sorry - my wording may have been unclear. "These elements" referred to the patches themselves. In the case of fiber, however, the use of MPO cables is a big win. Pre-made cables with modular connectors, snap-in breakouts, etc. No help as far at twisted pair, but potentially a godsend for horizontal scaling of fiber and the use of 40GE or 100GE.
    – rnxrx
    Aug 27, 2012 at 19:37
142

Please call a cabling contractor in to spend a day or two onsite to "dress" your cables. I used to spend time dealing with this type of work on my own, but realize that cabling contractors are faster and more organized.

A good cabling contractor is better at this than you are!

They will have the right resources to tag, test, get custom lengths, dress cabling and install the right type of management infrastructure.

It's worth the time and expense in order to end up with a cleaner, more manageable solution.

If cost is an issue, then you can reduce the hired man-hours by bringing the contractor to handle any structured cabling needs (re-terminating patch panels, testing existing runs, etc.)... Looking at that photo, there will be a need... You can buy or have them provide a variety of patch cable lengths and perform that work on your own, tagging and labeling as needed.

I went from this before-to-after by using this approach. It took 6 hours of contractor help, and 8 hours of my work to get the result.

enter image description here

enter image description here

20
  • 90
    That's pretty... Aug 24, 2012 at 16:51
  • 6
    moonraker. what a lovely name for a cisco :)
    – petrus
    Aug 24, 2012 at 16:52
  • 3
    The contractor did the patch panels, the 2-post rack installation, vertical and horizontal cable managers and ladder work. I performed the patching and color-coded the subnets.
    – ewwhite
    Aug 24, 2012 at 17:04
  • 8
    Two problems with this because its a one off a) often pro cabling jobs will leave it hard to make changes b) cables get messy because of changes often in a rush, so it requires a change in procedure for tidiness to last
    – JamesRyan
    Aug 24, 2012 at 21:43
  • 6
    @JamesRyan I don't see how my example above would be difficult to manage or modify. As for quick changes, it's possible to perform them outside of the cable management system, but it means that those changes will have to be corrected eventually. That's a discipline/organizational issue, not a problem with using a contractor.
    – ewwhite
    Aug 25, 2012 at 11:03
69

A co-worker and I recently cleaned up a mess that was pretty bad (I might post pictures later if I get some time) and I wholeheartedly disagree with the contractor approach. You will learn significantly more about the system itself and what deficiencies it has if you do the work yourself. Also when you make a mistake, as you or any other mortal is likely to do, you will have a much better idea of how to fix it. Having said all of this here are my suggestions.

  • Take your time: this is a marathon not a sprint.
  • First remove all the items that are not plugged in on one end. This took a surprising amount out in our situation.
  • Label, label, and label again.
  • Do a switch at a time.
  • Take your time and check your work.
5
  • 45
    +1 for the "remove all items that are not plugged" Aug 24, 2012 at 18:46
  • 2
    re "You will learn significantly more about the system itself and what deficiencies it has", what if you DO NOT want to learn about the system deficiencies? :) Really you will sooner or later but it's about choice... Aug 24, 2012 at 20:33
  • 1
    The sooner you get neck deep in the problems the sooner you can proactively fix things. I have a huge preference for doing things myself whenever I can. You have fiber to run between buildings, hire a contractor. You need to splice a couple cables or buy pre-made lengths, do it your self. You will be happier in the long run when you have a more intimate knowledge of the system.
    – PlTaylor
    Aug 24, 2012 at 20:52
  • 5
    I don't see why it has to be an either/or proposition. Why couldn't you work with the contractor, and learn from him, and benefit from his expertise. (BTW, +1 for the excellent points you make, even though I only partially agree.)
    – iconoclast
    Aug 27, 2012 at 17:13
  • 1
    Nothing is black or white and this isn't an either or proposition. My personal bias is to do everything I can that will help my understanding of the system and how to troubleshoot it later. If the cabinet is a mess chances are there is a lot more wrong with it than just a rats nest.
    – PlTaylor
    Aug 27, 2012 at 18:23
57

My sympathies to you. I was tasked with a similar problem for a number of cabinets, equally as horrendous.

The approach I took was as follows:

  1. Use a spread sheet to make a list of which port is connected to which port on which piece of equipment (arduous manual process of cable tracking). Try to use a spread sheet cell to represent a port, and order collections of cells as a visual representation of each piece of equipment.
  2. For each port, also note the VLAN/subnet etc (I did this via a web interface/cli)
  3. Write a small macro to theoretically re-wire your equipment based on cabling distance, or other factor of your choice. The result should give you which switch port maps to which patch panel port (or other equipment). It should also indicate how many cables you need and how long they need to be. I did this in a spread sheet by using cells as switch/patch-panel ports and it worked well. It produced a visual diagram of the wiring by drawing lines between cells and colouring the cells with conditional formatting to show their VLAN number(s).
  4. Once I had the schematic it took less than an hour after working hours to pull out all the cables and patch them up according to the generated diagram. Then another 30 minutes the next day to fix the "why can't I connect?!" problems :-)

There's really no easy way to do it, but at least the above approach saved me from pulling my hair out.

Good luck!

Screen shots:

Before:

Before

After:

After

5
  • 5
    +1 for the spreadsheet (idea and shots).
    – Henk
    Aug 28, 2012 at 18:28
  • 7
    @Jak care to share the spreadsheet with the re-wiring / VLAN reconfiguration algorithm?
    – the-wabbit
    Mar 5, 2013 at 9:36
  • 3
    Maybe a program like Eagle PCB Design program (used to design circuit boards of schematics) can help to layout wiring using the built-in routing algorithm they are designed to avoid crossing wires, which will help solving the nest. other similar program will probably do the job
    – Vitim.us
    Apr 6, 2013 at 23:57
  • 4
    I would love to share the code but I did this as part of a contract so I don't officially own it :-( Its done using VB macros in Excel by calculating the distance between cells.
    – Jak
    Apr 9, 2013 at 15:17
  • Spreadsheets. Our systems professor once said that Excel is the most important piece of enterprise software in the world, there are even banks that run solely on Excel. But this is the first time I see Excel used for cabling itself!
    – Pavel
    Jun 21, 2017 at 8:00
27

All these answers are incorrect because they omit the critical first step: Define the project. Before you can use any of these ideas (many of which are very good), you need to be able to answer the following questions with absolute certainty

  • What systems you are touching-- when and for how long can they go down?
  • Which departments will be hit and when will they agree to be inconvenienced?
  • If there is a critical system, when can I shut it down? is it backed up?
  • How much money can I spend to do this?
  • How many people can I borrow?
  • How long can I take? (meaning both man-hours to do it and time when it must be done)
  • Is there any documentation and should I trust it?

Maybe you know all these things and you didn't go into detail. But unless everyone is on board with your plan, you can end up writing one of those "How I blew up my company" Infoworld columns.

I had this situation once. Turned out we had a box that had to be on and connected. Pull a wire for too long, it would hiccup and crash. If it went down, anyone using it was screwed. Plus, two other systems-- who rarely used it but continually monitored its status-- would freak.

It only took a few minutes with the manual to figure out the order the systems needed to be shut off and a few phone calls to schedule it and one "Don't log into this system remotely over the weekend!" mass e-mail. But that's the difference between nobody noticing and people hating you forever.

Management's responses to definition questions will help you plan other projects. Your work requires them to make occasional choices and you need to learn what, how and why they will choose. You might find you can't hire anyone to help, can't buy 1-foot cables to replace 8-footers, can't shut people down, can't stop taking "can you unjam my printer?" calls while you do it, can't get overtime and can't come in early, stay late or work weekends if it means you won't be there M-F. If so, you know you need a new job.

2
  • 5
    +1 for recognizing that this is a project. It's easy to overlook. Aug 30, 2012 at 15:10
  • 1
    Thank you. All the work you perform in network administration or help desk is a project. "Reset password" or "change toner" don't require much definition... but the fourth time you unjam a printer, it's time to add tasks like "inspect rollers" or "call vendor" to the plan.
    – Geoff
    Aug 30, 2012 at 15:19
20

If you cannot (for whatever reason; it's usually that the company doesn't want to spend the money) get a contractor:

  • Schedule a weekend outage. Nothing will be guaranteed to work during this time. If you can do full shutdowns, so much the better.
  • Get the boss to agree to comp your food, drink, and time.
  • Beforehand, map each cable run, source and destination.
  • Install the cable management
  • Have appropriately-sized cables on-hand beforehand. If you must crimp them yourself, do it before the outage. Label both ends of each cable now.
  • Pre-print labels. (BTW, buy a labelmaker, extra labels, and spare batteries.)
  • Replace one cable at a time (unless impossible).
  • Turn things back on with fingers crossed.
  • Test that things still work. Reboot a client PC and make sure that you can still get to things.
  • Go home victorious
5
  • 7
    "Test that things still work. Reboot a client PC and make sure that you can still get to things." - You cannot complete this step without first documenting what can access what. 'get to things' for an IT person may be completely different from 'do my job' to the actual users.
    – Freiheit
    Aug 24, 2012 at 19:00
  • 3
    @Freiheit Your comment strikes me as a bit pedantic. Was I supposed to have an earlier step that says "document everything that all of your users could possibly need to do their jobs", and replace "can get to to things" with "attempt to do the entirety of their job"? Perhaps that would be the ideal in thoroughness, but it strikes me as excessive.
    – gWaldo
    Aug 24, 2012 at 19:16
  • 9
    "document everything that all of your users could possibly need to do their jobs" is clearly excessive, but "can get to things" is incredibly vague. Especially after working a full day, off hours, and you just want to verify that the job is done and go home. Having a well defined and clear punch list of what things to check is important. It can be as simple as "Login as userbob on secretary workstation and access foo-app on jiggle-bunz-server". Just know what your requirements for "the job is done" are as best as you can.
    – Freiheit
    Aug 24, 2012 at 19:30
  • 1
    Having an acceptance test is good. Having a fallback-plan when that test fails is better. Aug 27, 2012 at 10:50
  • 4
    sorry, jiggle-bunz is the name of my next loadbalancer.
    – ItsGC
    Sep 7, 2012 at 6:09
15

I am amazed no one else has already said this:

Yes document, record, spreadsheet etc etc but photo photo photo and more photos. They won't help you put it back together, but they will save you going insane when you inevitably have to problem solve. Plus when it all goes wrong they will serve as a reminder why never to do it again! :)

I also like the comment of "removed things which aren't plugged in to anything" its amazing how much stuff like that there is!

14

If you can get a contractor to take care of it as ewwhite suggests, then let them figure it out.

Otherwise:

  1. I'd start by identifying exactly where everything is and where it goes.

  2. I'd suggest keeping an inventory (simple spreadsheet should work to start) of each device and it's connections, VLANs, source port, destination port, IPs, and any other pertinent information.

  3. Have your cables pre-made and pre-labeled before you start. Use cables that aren't overly long but still leave room for some movement/ slack.

  4. If you can't do it all in one go, start device by device.

  5. Use vertical and horizontal cable management as needed, and velcro to secure sections (generally every one foot).

  6. Something that may help, is to use a different color cable for your each logical subnet to help visualize things in the future.

  7. Test to make sure things are as they were before you started. It's useful to have a pre-made document of tests to perform so you can go through, perform the test, and document the result.

After you are finished, make sure to keep up with the cable management and avoid quick runs to get something up and going. You'll almost never come back to properly finish the job. You may also want to develop a standard policy/ procedure for all cabling. That way, moving forward, you won't create these sort of issues. The person inheriting it from you will thank you :)

11

On labeling cables

  • I learned one important rule of labeling cables: Don't label your cables with the device/port, unless this is an absolute requirement and you're given the resources (label printer, blanks, where and when you need them.) If you do, the moment you have to repatch a cable you will be required to relabel the thing.
  • The best labeling scheme that I worked with involved labeling each cable with the same tag on both ends, perhaps followed with an 'A' or 'B' tag. That way, when you run across all of your panels and devices, you can match end-to-end connectivity with relative ease.
  • In the past I've worked with custom made cables, which actually had an offset length printed in ft and/or m. on the cable at regular intervals. This helped in actually identifying cables in a big ball of spaghetti. However, prefab cables are much more reliable but sadly lack that length indication.
2
  • 2
    You absolutely should label the cable with device/port but only the device/port name at the end of the cable which is less likely to be unplugged when re-routing. If you are routing cables from servers to switches, label the cable with the server name and interface number (of the server, not the switch) on both ends. If you are routing cables from switches to patch panels, the choice is more difficult and depends on your recabling habits - just define the end you are not going to plug when re-routing and use its name as the patch cable label. Again, on both ends.
    – the-wabbit
    Mar 5, 2013 at 9:43
  • 1
    I really like your idea of mid-span cable labels. I always label both ends of each cable with a serial number and the length. Having the length labeled is handy when you're reaching into the cable box. I use a Brady ID Pal cable label maker. Be sure to get the "industrial" label tape -it actually sticks more or less permanently, unlike typical label maker tape.
    – Dan Pritts
    Mar 4, 2016 at 17:49
10

Wow is all I can say. Here are my suggestions for all it's worth:

  1. Don't look to tackle this in a specific time frame. With a mess like this there's no need to put undue pressure on yourself to have it cleaned up by X date.

  2. Map out and document all of the physical connections (port x on device X connects to port x on device y, etc.).

  3. Map out the network logically (subnet A is for office A, subnet B is for DMZ, etc.).

  4. Figure out how you're going to organize/route the cables that takes into account the new layout in existence with the current layout. A cabling contractor can help with this. I'm not sure I'd want them doing the actual work though as they won't have any knowledge of network topology/dependencies and if things get connected incorrectly they'll be no help in figuring it out.

  5. If your proposed layout allows for you to swap/reroute one or a few cables at a time then do it that way. Find a few minutes every day where you can swap/reroute a handful of cables into the new layout. You'll need to plan for small windows of downtime/unavailability for those systems whose cables are being swapped out. As you stated it's easier to ask for forgiveness then permission but you don't want to over do it.

7

Just thought I would post with something NOT to do.

I was in a similar situation a while ago although I only had two racks to deal with and not quite so many wires.

I thought I had a great idea which was okayed by the boss - the idea being that TCP/IP on windows is tuned to retry and timeout after 5 secs so if a cable is unplugged for a second or two it's not a big deal.

So I commandeered an assistant and we would get the new cable in place all nice and tidy, then simultaneously we would swap both ends with the old cable.

This was when we discovered one application we needed that was dependant on a flat file database was very sensitive to this and it's db got trashed so we had to restore that from backup!

5

My only suggestion would be about roll back process.

If you can yourself or one of your internship guys would take a picture of every single step. Or record video of important steps in case of there is a problem and want to rollback process.

I will be helpful also for further references if somebody else will join your team and tell him what you did.

4

If your boss is not willing to give you overtime, yet also wishes to avoid downtime as much as possible, it might be a good idea to inquire about coming in around 3pm one day, and staying late. This way, there will be no downtime and the company won't be ponying up for overtime. I've pulled jobs like that in the past, when the company didn't want to overtime me, and it was honestly fun. Bring some speakers with you, and your ipod, and cable like a madman to some metal.

2
  • How would this prevent downtime? Maybe no downtime during business hours, but it's not clear from the question that that's okay. I assumed they did not tolerate downtime at all, but perhaps he meant only during business hours. (At the company where I work, downtime would affect end customers, local retail stores which in some cases are open 24 hours a day, as well as other businesses that communicate with our systems, so any downtime at all is a problem.)
    – iconoclast
    Aug 27, 2012 at 17:20
  • 1
    @iconoclast that is a very good point. I had assumed this was an office-setting where after hours downtime would be fine.
    – acolyte
    Aug 27, 2012 at 18:08
3

I think nobody mentioned this yet:

I would ask my boss whether we could get a full new round of equipment to replace the old installation in new racks. Depending on its age that might even be cost effective.

1
  • +1 : when touching anything 5 or more years old, it should next be placed into the trash bin. With cable end(s) cut off.
    – Krista K
    Feb 10, 2014 at 9:47
2

I have an idea that might also help, I think someone already mentioned color coding your LAN segments, but to make life a little easier you could simply buy whatever patch cables you will need in advance, rather than building them yourself and testing. I use a site called Monoprice, they are awesome, they have various cable length options and the cables are cheap. I will post a couple of links to the site below, hopefully this will help make the job go easier.

Cat 5e Ethernet Cables

1
  • Monoprice is cheap but i've had quality problems with items bought from them. CDW has pretty cheap prices on their house brand cables, which I believe are made by Belkin. I'm sure there are other inexpensive vendors.
    – Dan Pritts
    Mar 4, 2016 at 17:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .