17

I've always assumed that using Y cables from the PDU to redundant PSUs in the same server was totally acceptable practice, however I was recently speaking to someone that said that their colo wouldn't allow it.

I'm no electrician, but it would seem to me that since the PSUs are redundant, they aren't both drawing at the same time. If this is the case, it would seem that using a Y cable on two PSUs doesn't risk overdrawing the rated ampacity of the cable or individual outlet.

Have I missed something here? Why would a colo choose to ban Y-power cables in server racks?

8
  • 1
    Can you link to an example cable?
    – ewwhite
    Jan 7, 2013 at 15:22
  • 9
    Hm, seems like we really need a start-to-finish "Power for Sysadmins" primer here. Some of these answers have opened a whole new can of worms
    – MDMarra
    Jan 7, 2013 at 15:53
  • 3
    I'd strongly suspect they had some idiot in the past take a single C13 port from the PDU and "distribute" it to 5 servers or something that blew out the PDU... And now they have a rule against Wye splitters.
    – Chris S
    Jan 7, 2013 at 16:01
  • 1
    This Q should be added to the "Canonical" list. YMMV... Jan 7, 2013 at 16:15
  • 4
    @DeerHunter I don't think this question should be canoncial in and of itself, but a "Power for Sysadmins" primer sure seems like a good idea for a new canonical. Jan 7, 2013 at 16:36

8 Answers 8

19

I am no electrician either, but I think you will at least lose the possibility of keeping your server up and running when doing so. On the contrary if you connect each PSU to a different power source, your server will still have an availble power source (hopefully).

8
  • 12
    This. Y-cables don't pose an electrical hazard, they pose a redundancy hazard since. Jan 7, 2013 at 15:27
  • So you're talking about having multiple PDUs in a rack? That's not really feasible in a lot of cases, especially if you're using vertical PDUs.
    – MDMarra
    Jan 7, 2013 at 15:27
  • Depends on the rack I suppose. We are a smallish company with 2 tall server cabinets and both accomodate redundant power supplies by having independant power rails on each side of the cabinet. Of course your server room will need to be wired properly in order to take advantage of this as well. Jan 7, 2013 at 15:30
  • @MDMarra: Maybe, you will have a risk if the PSUs are working in load balancing mode and a failure occurs. In this case, the power will increase on the other still active PSU. Look at this post.
    – Khaled
    Jan 7, 2013 at 15:33
  • @Khaled but that still won't result in unsafe power consumption as far as I am aware.
    – MDMarra
    Jan 7, 2013 at 15:37
16

I've never used them because they're a single point of failure, at best.

Every server I deploy into a real datacenter has each PSU plugged into a different PDU in the rack, each of which are attached to a different independent UPS, on different circuits, ideally even fed from different power feeds.

If the UPS, PDU or circuit your Y-cable's attached to goes down, the redundant PSUs are going to be useless, so it seems like a waste at best, and a false sense of redundancy at worst.

EDIT:

I'll just mention that I'm talking about lower capacity 1U or 2U UPSes mounted inside the server rack, rather than the much larger, much more expensive UPS units that take a up rack all to themselves. Those are definitely built to be highly redundant in and of themselves, without the need for a secondary unit.

5
  • Most of the environments that I have worked in have a Symmetra or similar UPS that consumes multiple racks. I don't really see too many places having redundant UPSes of this scale, but I also don't have a lot of exposure to the "power" side of things - thus this question. I also really have never seen a failed PDU (not that it doesn't happen, I'm sure). When you hit that scale, do people really have multiple Symmetras to feed multiple PDUs?
    – MDMarra
    Jan 7, 2013 at 16:06
  • @MDMarra Yeah, see my edit, I got tunnel vision - I've yet to see anyone set up a large UPS system like the type you mention with a second unit for redundancy. If/when additional units of that size are added, it's about extra capacity, not extra redundancy. Jan 7, 2013 at 16:12
  • 4
    @MDMarra I do a lot of datacenter design and building. It rarely is multiple UPS systems for redundancy. The big UPS systems themselves can operate with various levels of failures (batteries, power-controllers) and will fall back gracefully to just rerouting mains power if needed. Mains power will be taken over by generator if needed. So normally the UPS is just a rectifier/surge-protector and covers the take-over from mains to generator. The UPS will have multiple output circuits which feed the PDU's in the rack(s). Each server connects to 2 different curcuits.
    – Tonny
    Jan 7, 2013 at 16:37
  • @Tonny Hmmm, very good info. I wish I'd phrased the question in a way that this could be an answer and not a comment so you could get some rep out of it :)
    – MDMarra
    Jan 7, 2013 at 16:41
  • @MDMarra :-) I'm doing OK withotu the points :-) But a canonical question/answer on best practices for serverroom power seems to be in order. (And let's not forget earthing issues on that one.)
    – Tonny
    Jan 7, 2013 at 18:59
8

The primary reason such items are frowned upon is due to redundancy, lack-thereof. Using such a cable means all of your server's power-inputs are being fed by the same circuit so when that circuit dies (or the PDU it's connected to, I've had that happen) so does the server. Colos strongly recommend Primary and Secondary circuits for just this reason and want to see multi-PSU servers plugged into two circuits.

Way back in the day I had a group of machines that shipped with a single 3-way Y cable and 3 normal power cables for a large (7U if I remember right) 3 PSU system. The data-center I was working in at the time (this was about 1999) didn't have enough power-outlets for that kind of thing, so we ended up using the Y cable; 2 legs of the Y on one UPS, and a straight up power-cable for the 3rd PSU to the second UPS. 3-PSU systems are thankfully much less common now.

PSU Load-balancing, or is it switching?

There are differences to how power-supplies handle loading. As various power-supply benchmarks have shown, peak efficiencies are reached once you get over 50% loading. There are gains to be had for running things all on one PSU, it's more likely to be efficient. It is for this reason that some server manufacturers draw all of a server's current through a single PSU and switch to the other one when a failure happens or a whim strikes; a 230 Watt system will get best efficiency from its dual 400 Watt PSUs by running all the load through only one PSU.

Such switching systems only draw from one PSU, and therefore one circuit if using fully separated power circuits, at a time.

The downside to switching systems is that load can move unpredictably in the community of PSUs connected to a certain circuit. If enough of them throw their weight to a single circuit it can overload it. This is bad power-design, since you want to design things so that you can lose a full circuit and have things stay up, but it's still something that trips up systems engineers.

Load-balancing servers draw equal amounts of current from both PSUs. This gives predictability in circuit loading, though can still cause blown breakers if the Systems Engineers load their circuits over 50% and a circuit dies forcing the PSUs to draw 100% from one circuit, which now exceeds its rating. Again, bad power design but it's a common mistake.

Startup loads

There are two kinds of startup-loading:

  1. Everything runs flat out until BIOS (or OS Boot, or app-load) catches up and things calm down.
  2. Inrush current loading right as things get turned on.

The first is something we're all familiar with. That 120 disk SAS array may draw only 4000 Watts when running normally, but if all the disk-shelves restart at the same time it may draw 6500 Watts.

The same holds true for servers. Fans run at full speed, yes. CPUs run at full speed for a bit, yes. RAM is run at full voltage during post, yes. It's likely to draw as much as it can draw during those first stages of POST but rapidly drops off as the BIOS hands things off to the OS and power regimes take over. A server that normally draws 110 Watts during normal usage may temporarily draw as much as 200W for those first few phases.

It's this temp loading that most people think of when they say things like, "it runs the power-supplies at full on startup". Those 400 Watt power-supplies plugged into a server that draws 230W on a busy day aren't going to draw 400 Watts, they'll draw 230W... combined.

The second isn't well known, but when people run across it they get worried. This is inrush current, and takes a few milliseconds during which draw can be quite a lot higher than it normally is. The inrush current for IT devices with AC to DC converters in them (which is all of them) almost always happens twice:

  • One time when the cable is plugged in, as the pre-power stage gets power. It's this stage that allows the power-button on the front to power on the device.
  • A second time when the main distribution stage powers up and starts the device.

Because of the timings, this only becomes a factor when restoring power to a dead circuit. All those devices powering on at exactly the same time can do weird things to the power on that circuit, and that can cause damage all by itself. Doing a staged startup alleviates this.

This is the other area people think of when they say things like "power-supplies run full-tilt on startup", since each PSU has its own inrush current. But as I said, this lasts for a few milliseconds and comes in two stages.

2
  • Are server PSUs optimized for a different point on the load curve than consumer ones? For the latter peak efficiency is normally at ~50% and falls off on either side of that point. Ex test #3 here is the 50% point. jonnyguru.com/… Similar results were shown for both of the other reviews at the top of the main page; and on most if not all of the reviews I've seen of quality PSUs. Jan 8, 2013 at 19:46
  • @DanNeely You're correct, I was remembering the curves wrong. Will update.
    – sysadmin1138
    Jan 8, 2013 at 20:15
4

Our local Colo did not like them either. We had a shared cabinet and were only allowed to utilize a single PDU port for our dual PSU server. The colo didn't like it for redundancy reasons, but for a non-critical machine it was perfectly acceptable to us. There aren't any major power issues from an electrical perspective.

This is why our colo didn't like it:

  1. If the PDU dies the server is dead
  2. If the cable gets fried your server is dead

Here is why I like it:

  1. Non-mission critical box
  2. Supplied power to both PSUs giving me redundancy on the box
  3. I wasn't all that concerned with potential down-time based on the failure of the PDU, nor was I all that concerned about the cable going batshit crazy. (see #1).

The colo didn't have any issues with them electrically speaking. The draw was identical, the box only used what it needed, regardless of the number of PSUs consuming power. However, shortly after buying this cable with my Dell box Dell did stop publicly offering the cables for sale.

5
  • 4
    If the PSU dies the server is dead - Did you mean PDU there?
    – MDMarra
    Jan 7, 2013 at 15:41
  • IMO, the colo/datacenter shouldn't mandate what is in essence an operational issue. If I choose to use a Y-cable then that's my choice, regardless of the redundancy or lack thereof.
    – joeqwerty
    Jan 7, 2013 at 15:52
  • @joeqwerty Fully agree, if the data center is held responsible for uptime to the box however, I would fully accept their need to mandate the use of 2 straight cables instead. That wasn't the case for me, and in a shared cabinet scenario the Y-cable was a much better choice. Jan 7, 2013 at 16:13
  • I think the "PSU" in point 3 should also be "PDU", resulting in the points meaning: you consider the risk of cable or PDU failure to be much smaller than the risk of PSU failure which sounds about right to me. I've seen many more PSU failures than cable failures. I've not done much rack-based work, so I can't really comment on PDU failure rates. Jan 7, 2013 at 22:21
  • @joeqwerty To me, and from my experience, it sounds like sometime in the past someone had a Y cable hooked up to a dual-PSU server (maybe even the datacenter tech did it), and then a circuit died, and a pointy-haired-boss let loose on the DC that the "redundant" power they paid for (in both the server and dual circuits in their rack) didn't work, they were in violation of their SLA, etc etc. Easier for the DC to just ban the cables than to worry about whether people understood the outcome of how they were being used, especially if it was done by their own in-house techs.
    – gregmac
    Jan 8, 2013 at 0:02
3

When a server boots up, both PSU start up at full power to do a system check. This draws double the normal operating power. Now that power is more of an issue, the circuits coming into your cabinet are very specific. They don't want you to trip a breaker should you have to reboot your server.

Also, some office building don't let you do that because of fire code. You are doubling up the power that could be consumed from one outlet.

8
  • 1
    Do you have any documentation that shows that powering on a server with two 750W PSUs will cause 1500W of draw?
    – MDMarra
    Jan 7, 2013 at 15:52
  • 2
    Your effect is right (or close to it) but I think your cause is wrong: Servers draw more power at startup because they have "startup stuff" to do: Spinning up disks, power on fan check, charging up capacitors, etc. - I don't know any PSU that does alias test. Doing so would require a load to test with (big honkin resistor) & switching hardware which I've never seen in a PSU. If you have seen such a beast I'd like to hear about it.
    – voretaq7
    Jan 7, 2013 at 16:05
  • 1
    @voretaq7 But even so, this additional load should fall well below the rating of a single PSU in the system. Sure there's increased consumption, but it's not an unsafe increase.
    – MDMarra
    Jan 7, 2013 at 16:14
  • @MDMarra That depends on how many disks you have to spin and how intelligent the controller is -- I saw a box that produced a 2-3 Amp spike (on 208V power - so ~450-500W plus the normal 150ish it ran at) because its disk spin instructions were "all 16 disks RIGHT FREAKIN NOW. (That configuration didn't last long.) Generally though you're right - we're not talking "Full rated draw on both power supplies" (at least I've never seen it be THAT bad)
    – voretaq7
    Jan 7, 2013 at 16:43
  • @MDMarra I know from personal experience that most HP Proliant servers will briefly (3 to 8 seconds) draw full load on both PSU's on startup. Some models Dell Poweredge do it too.
    – Tonny
    Jan 7, 2013 at 16:44
3

A colo which is offering two redundant power circuits into your cabinet may be interested in having the two circuits balanced.

This may be why they don't like you connecting servers with dual PSUs to a single circuit, regardless if this is done with a Y-cable or with two cables plugged into the same PDU.

2
  • 4
    Maybe, but then again if I'm paying for the circuits then it should be my choice how I choose to use them, load balanced or not. As long as I don't exceed the allowed capacity of either circuit the colo/datacenter shouldn't have anything to say about how I use them.
    – joeqwerty
    Jan 7, 2013 at 16:09
  • If they feel the need to keep those two circuits balanced, I'd be worried that they are not as redundant as they were supposed to. After all, at any given point if one goes down, the other is supposed to be ready to take the full load instantly.
    – kasperd
    Mar 20, 2015 at 7:41
2

Being in the colo and hosting business for quite a while, I can give you the three very best reasons why a hosting company would not allow it.

Money, money and money

Think about how much a single 15 or 20 Amp circuit cost you, the hosting companies would much rather you purchase a second one for failover and charge you the big bucks twice a month instead.

4
  • Well, I'm not sure that covers it, honestly - you can plug two cables into the same circuit just as well as you can plug in one Y-cable. Jan 7, 2013 at 16:13
  • Sure you can, but you will eventually run out of space for your plugs. That means a second PDU. Believe me, that business is all about two things 1) Space 2) Power. Whatever will make you consume more of one or the other they will try it.
    – Alex
    Jan 7, 2013 at 16:17
  • Alright, but running your PDU out of plugs doesn't necessitate the lease of a second circuit, so I'm not sure how this is related to profit-taking by the colo/host. Jan 7, 2013 at 16:23
  • But it does make you use 1 more U if not 2. For customers renting a 1/4 or even 1/2 cabinet, that could be enough to have them upgraded to 1/2 or a full cabinet.
    – Alex
    Jan 7, 2013 at 16:26
-2

I'm really surprised nobody on here did not realize you can have two 20amp PDUs running to different power then take a y cable plug into two different servers and plug into the first PDU then take another y cable plug into the second power supply of the two servers and run it to the second PDU. You now have redundancy and less power cord mess.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .