8

We have the opportunity next week to rewire five server racks which is currently in a real mess. Each rack currently has a switch installed up top with lots of cables hanging around.

We are thinking of installing patch panels in each rack, wired back to the first rack into another panel panel, from there patch leads that go into the switches and on the other racks, patch leads going into the servers.

enter image description here

So a server in rack 5 for example would be connected something like this:

[server] -> [patch lead] -> [patch panel in rack 5] ->->->->->-> [patch panel in rack 1] -> [patch lead] -> [switch]

Will something like this work?

Any points, suggestions appreciated.

Thank you in advance!

2
  • +1, Good for organization/separation... Our colo's NOC puts the server patch panels at the bottom of racks, as they have a raised floor and run the cabling under there.
    – jscott
    Jul 19, 2011 at 15:44
  • 1
    Good effort on the mspaint drawing. :)
    – user78940
    Jul 20, 2011 at 14:07

6 Answers 6

4

This will work, but what's the advantage to you over cleaning up existing cables and keeping the switches in the local racks (do you have lots of cross connects between the racks that could be eliminated?).
Remember that patch panels don't magically make your wiring neater: Discipline, maintenance, and lots of velcro ties do that.

Generally, separating your racks can be a good thing, particularly since patch panels usually come with nice solid trunks from panel to panel (less junk under the floor or in your cable trays).
The big downside is that if you lose link on a switch your now have a lot more to troubleshoot (is it the cable from the server to the local patch panel, the panel-to-panel trunk, the cable from the patch panel to the switch, the switch itself, the server itself, etc.).
The smaller downside is having to open two racks to connect a server to a switch. This can be argued as an increase in security however (someone with keys to the switch rack needs to be around to connect new equipment.


Small bit of advice no matter what you decide to do: Document the hell out of your cabling - ESPECIALLY if using patch panels. You will thank yourself later when you need to figure out what path a server takes to get to a switch port. (There are a few questions here on cable labeling schemes - https://serverfault.com/questions/64259/what-is-the-most-effective-solution-you-used-to-label-cables is one of them)

3
  • 1
    +1 For velco ties alone... I want to take the dikes to anyone carrying a bundle of zip ties.
    – jscott
    Jul 19, 2011 at 15:52
  • Thanks for your comment and I agree, having the patch panel does somewhat add additional failure points that would need to be investigated if for example a server lost connectivity. We currently have quite a few cross connects, and will be deploying two SAN units with multipath... that is the main reason why we are considering going down the patch panel route.
    – PeterG
    Jul 19, 2011 at 16:55
  • consolidating cross connects is totally worth any additional overhead IMHO - the less crap in your floor/tray the better off you are. Note that there are also fiber patch panels (in case you're considering a fiber SAN) which may be worth looking in to, but they are HIDEOUSLY expensive, as are all things fiber :)
    – voretaq7
    Jul 19, 2011 at 17:41
4

I've personally been involved in wiring data centers with both methods you've described and I have to say that having the switch in the rack has proven to be a much better solution. Both installation and maintenance are easier with the switch in rack and unless you plan to have server counts well below your in-rack switch port density, switch in rack will most likely be cheaper.

Here's a few of the advantages I've come accross for the switch in rack solution

  • Requires less inter-rack cabling - 1(+backups/bonding) uplink for each core switch instead of 1 per host. This is HUGE.
    • Less time to implement
    • Fewer cables to test
    • Lower cost (assuming you can size your rack switches appropriately)
  • Requires 1 patch cable instead of 2 - Each cable that's run is another place that requires testing
  • Less documentation - At a minimum each cable needs to be labeled and 1 is easier then 2
  • Patch cables fully traceable - Oversights happen and documentation gets missed, in a single rack its much easier to trace out a cable (but still no fun)
  • Easier server removals/moves - Patch panelling requires a lot of trust in your documentation. Switch in rack I pull the cable from the server cut the end off and feed it back up to the switch and remove. No trust/guesswork back at the patch panel
  • Fewer data errors - Patch paneling has 6 points to punch/crimp before reaching a switch, switch in rack has 2. Each crimp/punch point is a place where signal is lost. This is less of an issue with 100Mb then 1Gb+. I've also found it easier to certify an rj45 crimp then a panel punchdown.
  • Quicker Switch Inventory - Printing out a config for a single switch and verifying a rack is much easier then printing out all the configs for all the switches and verifying
  • Less clutter - The patch panel solution requires a lot of cables in a small space and if you're seeing clutter with 1 rack of servers (~40 cables) imagine if you had 160+ all in one place

This isn't to say I'm completely against patch paneling, but I try to limit its usage to places where I can't bring the switches to the equipment. Wiring office cubicles comes to mind as a perfect place to utilize patch paneling, but in the datacenter I'd encourage you to get as close to 0 patch panels as possible.

3

We currently do this exact thing where I work. And, I hate it.

  1. Lots of extra points of failure
  2. We're always short uplinks in the cabinet.
  3. When we need new links, someone is wiring in the cabinet on a ladder, risking all the other links.
  4. We have wire guides in place to clean up the wires below each patch panel and then wire guides on the switch side, wasting space.
  5. tripling the number of cables. Yay. I need more of those!
  6. And the bajillion patch cables run with no rhyme or reason and no way to easily trace them down.

That last point is the worst. When you are in a hurry, the multiple levels of trace-down are hell and your switch is far-far away when all you want to do is check the lights on the server and the switch at the same time.

If you do do it this way, On thing I'd recommend is that you interleave 1µ switches with 2µ wire guides so you don't have a mega-brick of wires running down each side from a block of uplinks to a block of switches. You have to tie the wire down to keep it under control once you have 200+ wires in one channel and then you can't trace ANYTHING down.

If you have SAN fabric that is a second switch then I'd put it at the bottom so your wiring goes different ways.

UPDATE with Advice

We used Panduit pop-in patch panels and the hard-wired both, either are fine if your electrician is good. Lots of stress relief is critical with long-term wiring, bend them, bundle them, and tie them. And, honestly, you have to instill a culture of doing things right... so get vertical wire guides and use them religiously. (Velcro to rack frame is fine!) Draped cables are cables gravity is destroying.

To @voretaq7's point: pre-wiring the entire block of 24 is a good idea you'll find uses for them later (we wound up pulling KVM, using them as cabinet-to-cabinet links and more.).

Work hard to keep things consistent. 1-24 should connect in order to 1-24 on the other end. If you've got the switch ports for it, pre-connect all of them (or half) in order as well. If you run more than data on ethernet, color code every cable for that link. You want to be able to spot the weird ones in a hurry. Consider assigning ports by rack µ number rather than loading from left to right. Anything that helps you avoid having a giant spread sheet full of chained addresses that people can't read in a hurry.

Once things are live no one wants to let you go back and make things consistent, so having easy, natural standards that people can follow in the emergency times without paperwork means you don't fall off discipline every time a server dies.

4
  • 1
    I can agree with you on #1 and #4 depending on the kind of wire guides you use, but the rest sound like local process problems - Especially #3: If you're doing that you're Doing It Wrong (tm) -- Your patch panels should be pre-wired end-to-end, and not need any additional work once they're put in the rack. wiring from a patch panel to a device is exactly the same as from a switch to a device...
    – voretaq7
    Jul 19, 2011 at 21:17
  • In your experience, what sort of failures are the most common in this sort of setup? The last thing we want to do is introduce problems into a environment that has been rock solid but messy as hell. I mean no cables tied down, they are just hanging there from switches to servers but as most would agree, it's a disaster waiting to happen.We are thinking of putting in approximately 2 48 port patch panels into racks 2 - 5 which will be more than enough for our usage now and for the foreseeable future.
    – PeterG
    Jul 20, 2011 at 8:19
  • Just to double check, I've never actually done this before so little uncertain - cabling from the back of one patch panel, to the other back of the next patch panel is possible? Anything to watch out for, take extra care with?
    – PeterG
    Jul 20, 2011 at 8:34
  • +1 to voretaq7. Nevertheless "local process problems" are problems almost everyone in the world has. @PeterG currently has draped cables, so things that work with human flaws are better than draconian rules that people don't follow. The bigger the company, the more you are right, though. ;-)
    – Mark
    Jul 20, 2011 at 13:03
2

It would work, and I find it to be a good idea. That way you separate server racks and telco racks...

2
  • 1
    I would heartily endorse this, with one exception - if you have SAN devices then these should preferably be in the same rack as the server(s) they serve, and connected via cabling within that rack.
    – Mike Insch
    Jul 19, 2011 at 15:47
  • We were thinking more along the lines of having the SAN units in the first rack, so all the wiring for each and multipath can be done there and then just patch each server in. If I think about it it does seem like it will be a very neat solution, but again as some answered it does add failure points etc... Thanks for the comments and suggestions. It really does help! :)
    – PeterG
    Jul 19, 2011 at 17:04
1

I've done both. It sounds like you want to move from top of rack switching to end of row switching.

I think I'd skip the patch panels and re-work and cleanup what you have.

Patch panels aren't necessarily going to clean up your racks.

I'd move the switches to the middle and run fiber/other between the switches to make up the fabric for TOR. You can split the physical network into pieces then and run cable appropriately giving you the "structured cable" without too much hard structure. Just put the switch where you want and run cabling in the same manner you would as a patch panel.

I'd use EOR as you have above but without needing any patch panels. Depending on the density and overall switch topology you could use a switching aggregation layer.

Can you label some of that gear with make/model/port count?

4
  • We did consider leaving or rather moving switches down to the middle of the rack, but as I've mentioned on a earlier answer is that the main reasoning for going for the patch panels is that we'd be installing two SAN units with multipath and I don't want to imagine the cabling mess that will create over five racks.
    – PeterG
    Jul 19, 2011 at 17:01
  • Didn't catch the part about the SAN until now. What is the SAN fabric?
    – dmourati
    Jul 19, 2011 at 18:46
  • @PeterG - I'm not a fan of switches or PDUs "in the middle" of a rack -- This seems like a great idea because it's shorter runs of cable and half your cables will be going "up" and the other half "down", but I've never seen it work out to be pretty in practice.
    – voretaq7
    Jul 19, 2011 at 21:18
  • The SAN will be iSCSI (Equallogic)
    – PeterG
    Jul 20, 2011 at 8:07
0

That will defiantly work, I have done the same thing in our data centre just remember to keep it nice and tidy with some cable management bars and run any cables down the sides of the rack and not in front of the equipment.

One problem I have had is that our cables run under the floor and I decided to put the patch panel at the top of the racks so I have 48+ cables running up the side of the rack to the patch panel taking up valuable space.

If I did it again I would say if you cables are going to run under the floor then patch panel at the bottom, if the cable are run above the racks then patch panel at the top.

I decided to go this way so my Cisco core was together and I could have a nice stack rather than having them connected by fiber. Also it makes patching into a DMZ switch easier.

Obviously you will loose a few U of rack space and document everything and leave some slack cable in case you ever need to move your racks around a bit

1
  • Great, thanks for the tip. Our colo provider has decided to run cabling above the racks so I'll definitely keep the panels up there... on my very primitive sketch I didn't add the cable management bars but we plan to get them in there as well.
    – PeterG
    Jul 19, 2011 at 16:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .