Server Room and Three Phase Power for Systems Administrators

There doesn't seem to be much educational material about server room power that is comprehensible to systems administrators.  I don't think there is a "typical" sysadmin type out there but I'm guessing that most have had little to no formal training about server room power.  Three phase power may seem like black magic and lots of incorrect assumptions are made, thus I decided to write this post.  Hopefully this will be useful to some sysadmins out there.

Assumptions/Disclaimers

• An electrician was kind enough to review this and state that everything seemed accurate.  That said...
• I am not an electrician, nor do I play one on TV. I am some random guy on the internet. Consult with an electrician so you don't accidentally fry yourself, blow up servers, burn down a building, or shave a few "9"s off your uptime.
• This article is intended for IT people trying to understand server room power, not for electricians.  I try to use terms a sysadmin would understand instead of 100% correct terminology an electrician would use.  That's easy since I'm not an electrician.  (In return, I'll forgive electricians if they abuse words like "memory" when referring to disk space or stating that a computer is "thinking" when pulling data off of a disk)
• This article deals with power in U.S. server rooms. It should be mostly universal, but YMMV elsewhere in the world.
• You know at least a little about how electrical power works.
• You know the names and functions of basic server room equipment (PDU, UPS, power supply, etc).
• This article only discusses AC power and focuses mostly on three phase.

Definitions

• 3-wire = 3 hot wires + ground
• 4-wire = 3 hot wires + neutral + ground
• delta connections (aka "Δ") = three phase connections where single phase is provided by tapping line-to-line (aka phase-to-phase)
• wye connections (aka "Y") = three phase connections where single phase is provided by tapping line-to-neutral (aka phase-to-neutral)
• transformer = device that converts between voltages
• Ln (e.g. L2) = phase n (e.g. phase 2)
• PSU = power supply unit
• ampacity (aka current rating) = the amount of current, in amps, a conductor can carry before immediate or progressive damage begins. Exceeding this rating is a fire hazard.
• P=IV: power = current * voltage  (units:  watts = amps * volts).  Examples: 2080 W = 10 A * 208 V.  1200 W = 10 A * 120 V.
• VA (or volt-amps) is described below in the section VA versus W.

Cables / Connectors

 

Always take note of the amperage rating of a cable.  Just because a connector should be rated for something doesn't mean the entire cable is rated the same way (e.g. 20A connector with the cable only rated for 10A).  This is not a common issue but is something to be aware of, especially when buying really cheap cables.

NEMA connectors. Examples L6-30R, 5-20R.  The naming scheme is fairly simple: an "L" prefix means it is a locking connector.  The next letter is usually a 5 or 6. 5 is rated for 125 V max and 6 is rated for 250 V max. This is followed by a hyphen and a number.  The number designates the current rating.  This is sometimes followed by a "P" for plug or "R" for receptacle.

Example: L6-30P is locking, 250V max, 30A max, plug.  5-15R (commonly seen around houses and offices) is not locking, 125V max, receptacle.

IEC 60320 connectors (formerly IEC 320).  Examples: C13, C14, C19, C20. Females are odd (e.g. C13, C19).  C13/C14 (rated for 10A) is used for lower current requirements than C19/C20 (rated for 16A).  Desktops almost always have a C14 and servers have a C14 or C20.

IEC 60309 connectors (formerly IEC 309) for high current three phase.  I sometimes hear these referred to as "60 amp connectors".  One of our server rooms uses these for 60A three phase 208V feeds to PDUs.  These often have 4 wires (aka "3-wire") for 208V and 5 wires (aka "4-wire") for 415V.  Blue connectors are typically 208V and red connectors for 415V.  There are other variants.

These are pretty straightforward but talk to an electrician for help.  These also have a naming and numbering scheme to show the number of pins: (P=power, N=neutral, E=earth/ground, PE=powered earth).  208V is usually 3P+E (three hot wires plus ground) and 415V is 3P+N+E (3 hot wires + neutral + ground).

Can you convert between different connector types?

Yes, as long as the weakest link in the chain can support the maximum current draw.  For example, you should not connect a blade enclosure's C20 (16A) to a C13 (10A) outlet on a PDU because the blade enclosure may draw more than 10A and light something on fire (or just trip a breaker on the PDU).  However, you can safely connect a server's C14 (10A) to a UPS's L6-30R (30A) since the UPS's outlet rating is higher than the maximum draw of the server.

UPS

A quick note about UPSs.  Since a UPS has batteries in it, the output power has to be converted to something to be usable as AC.  You can get a UPS with almost any standard input voltage and specify almost any standard output voltage, including three phase.  Shop around and ask a UPS vendor for more information if you need a specific input/output configuration.

Be sure to keep track of input and output voltages and plug types, kVA ratings, estimated run time at the anticipated load (usually a graph on the vendor's website), recharge time, etc.  Sometimes you can ask the vendor to add non-standard plugs or receptacles to a UPS.

Three Phase Power

 

What is three phase power?

Wikipedia has a decent answer.  Basically, there are three conductors that carry power that is 120 degrees out of phase from each of the other conductors, thus "three phase".  There are typically 4 wires (aka 3-wire) or 5 wires (aka 4-wire) in the cable.  The ground wire is not included in the wire count in the cable.

Why use three phase power?

The short answer is that it's more efficient.  Less copper is needed (thus it's cheaper) and there are other efficiencies that are introduced.  Besides, eventually it's plugging into three phase anyway, probably at the building transformer.  The long answer is found elsewhere on the Internet and in text books...

Delta vs. Wye Configuration for Three Phase to Single Phase

You may hear some people say that calculating the power draw per phase on 208V is hard and 415V is easy.  The reason why is that the 208V is being pulled in a delta configuration and the 415V as 240V in a wye configuration.

• delta configuration: connect the load to two different phases
• wye configuration: connect the load to one phase and neutral

(Left: Δ,  Right:  Y.  N represents a phase/wire and R is a load such as a server)


When you use three phase power, there are two ways to wire things up so that you get single phase power.  Remember that the wiring that carries the three phases has four different wires (sometimes there is an extra for neutral): phase 1, phase 2, phase 3, and ground.

Let's say you have 208V three phase.  If you measure the voltage by connecting one phase to another, you get 208V.  This is a "delta" (or "Δ") configuration.

If, instead, you measure from a phase to ground, you'll measure 120V (208V/sqrt(3) = 120V).  This is called "wye" (or "Y") configuration.  If you ever see a three phase 208V PDU report that a phase is doing 120V, it is correct because that is the phase's voltage with respect to ground.  The outlets are typically pulling from two phases (e.g. "L1-L2" or "L3-L1"), thus they are 208V.

Another way of saying this is that the phase-to-phase voltage for 208V three phases is 208V and the phase-to-ground voltage is 120V.

415V three phase is done in a wye configuration, meaning that single phase 240V is used by connecting phase to neutral (415V/sqrt(3) = 240V).

Determining PDU capacity for each phase pair in 208V three phase

To start off with, always balance your load across phases as best as possible.  It's inefficient to do otherwise and you may exceed wire ratings faster so it's a bad idea and should be avoided.  Besides, the math to calculate the load gets very complex (and imaginary!) when phases are not balanced.

That said, determining capacity is easy when you balance across phases.  You just need to know simple math and the square root of 3 (~1.73).  When you look at a PDU rating, you will typically see a wattage rating and and amperage rating.  The wattage rating is for the entire PDU.  The amperage rating is per phase (aka line), *not* per bank, per outlet, or for the whole PDU unless otherwise indicated.

When you take a three phase 208V PDU and pull power off of single phase outlets (typically named "L1-L2", etc), you pull power from two separate phases.  The math isn't what you may think.  Since we're assuming a balanced load across all phases, all you need to do to calculate the capacity of each phase pair (L2-L3, etc) is divide the amperage rating (e.g. 48A) by the square root of three (e.g. 48A / sqrt(3) = 27.71A).

The capacity equation for each phase pair (assuming balanced loads):
phase rating in amps / sqrt(3) = rating of L1-L2 (and L2-L3 and L3-L1)

Example: A PDU specifies a phase rating of 30 amps (derated to 24A). What load can you put on L2-L3, assuming L1-L2 and L3-L1 will each have the same load added?  24A / sqrt(3) = 13.8A.


Basically, the PDU vendor is telling you about the capability of the wiring but you probably care more about "how much power can I pull off of each phase pair?".  Assuming you balance the load across the phases, 48A / sqrt(3) is all you need to do to calculate the capacity of each of L1-L2, L2-L3, and L3-L1 (about 83.13A total).  In our example of a 208V three phase PDU that is derated to 48A, the PDU capacity is 3*(48A/sqrt(3))*208V = 17.3kW.  You may notice that the result lines up quite nicely with the vendor's advertised 17.3 kW rating for the whole PDU.

(A side note, you may notice that a "60 amp" PDU is "derated" to 48 amps.  I'm not sure what the exact reason is for this so I can only assume that damage will happen close to 60A but for safety and regulatory reasons it is derated to 80%.  Some "60 amp" PDUs are also derated to 45 amps due to using smaller wiring.)

It is worth repeating the difference between a phase and a pair of phases.  When you plug into a single phase outlet that is labeled similar to "L1-L2", you are plugging into two phases.  That means that the outlet draws power from both phases.  The capability of that pair of phases (when loaded with an equal load to L2-L3 and L3-L1) is the maximum current rating per phase divided by the square root of three.  Also keep in mind any other limitations of the PDU such as maximum current per bank of outlets.

There is a great online calculator for three phase power and more information about three phase power (including unbalanced phases) on Raritan's blog (archive.org link in case the original ever disappears).

Keeping things balanced between phases

Keep the load balanced between phases!  An imbalance in current will result in current flowing through the neutral wire.  The neutral wire must be rated to handle the increased load.

There are several strategies for doing this.  If you can afford it, always get a managed PDU with metering.  You can easily view how much power is being drawn per phase and per pair of phases (assuming the PDU is worth anything).  If you make any mistakes you can check the current load and figure out how to adjust it.

Let's say you have a rack with two three-phase PDUs and your servers have a similar purpose and usually pull about the same amount of power per server.  Let's also assume dual PSUs in an active-passive configuration.  An easy way to balance load and maintain redundancy is to cable half of the active PSUs to one PDU and half to the other.  The passive PSUs plug into the PDU that the server's active PSU is not plugged into.  As you do this, you'll also want to keep the amount of active PSUs balanced between the three phase pairs.

A simple example configuration is the following.  The letters A-F represent servers and the number is the PSU number (assuming 1 is active).  Column 1 is wired to PDU 1 and column 2 is wired to PDU 2.

L1-L2
-----
A1 A2
B2 B1

L2-L3
-----
C1 C2
D2 D1

L3-L1
-----
E1 E2
F2 F1


What happens if a PDU fails?  The other PDU takes over and the phases remain balanced.  If a single PSU in a server dies, there will be an imbalance.  If a single server is shut off, there will also be an imbalance.  There isn't much you can do to completely avoid it but the imbalance isn't something that should cause an issue.  Though the decrease in load per phase is not linear and difficult to calculate, it is still a decrease in load and shouldn't have any detrimental effects.

VA versus W (volt-amps versus watts)

Now we're starting to get into the realm of electricians and EEs.  The first thing to understand is that not all power loads are created equally.

Typically you assume that watts = volts * amps.  However, it is not that simple.  Some loads have a reactance that causes the power factor (real power / apparent power) to drop below its ideal 1.0.  To quote Wikipedia, "reactance is the opposition of a circuit element to a change of electric current or voltage, due to that element's inductance or capacitance."  Things that have motors tend to increase the reactance (motors, hard disks, etc).

Calculations of watts should actually include power factor.  Thus the equation should be:
watts = volts * amps * power factor

Watts are actually a measure of real power instead of the apparent power that volt-amps measures.  Volt-amps do not take into account power factor thus vendors often specify VA instead of W while others specify both.
VA = volts * amps

watts = real power
volt-amps = apparent power

Your power factor will probably never be 1.0, thus the difference between VA and W can become an issue.  You'll likely have to work with your facilities people to obtain the power factor of your equipment.  The power factor is a unitless number of up to 1.0 with 1.0 being ideal.  It can also be viewed as a measure of power efficiency.

In my opinion, the easiest way to account for power factor is to look at the capacity of a power system and multiply it by the power factor.  Thus a 3kVA rating (apparent power) becomes 2.7 kW (real power) when a .9 power factor is applied.  You must stay below both the real and apparent power ratings (watts and VA).

watts / power factor = VA
watts = VA * power factor 
power factor = watts / VA = real power / apparent power

A good power factor for a server room is typically >= 0.9.  We generally assume .9 in our calculations because that is what we have measured in our server rooms.  Some devices can measure the power factor while others can't.  Hopefully someone from your facilities group can tell you the number if your devices don't report it.

Why Higher Voltages Can Be Better

This answer is mostly a simple one that results in cost savings: P=IV, or power = current * voltage.  A server needs a certain amount of power (specified in watts) and its power supplies can accept a range of voltages (e.g. 100-250V).  To determine the size of the wiring needed to supply that number of watts (let's say 1400 W) you must look at the required number of amps.  For example, a C13/C14 cable supports 10A.  A C19/C20 cable supports 16A.

In this example with 1400 watts, the number of amps required with 208V is: 1400 W / 208 V = 6.8 A
In this example with 1400 watts, the number of amps required with 120V is: 1400 W / 120 V = 11.7 A

This means you need a minimum size of C13/C14 for 208V and C19/C20 for 120V.  The C19/C20 cable has more copper in it to allow for the higher amperage requirement.  (Side note: In practice this PSU will probably be rated for 100V-250V.  It has to support the worst case of 100V and would thus have a C20 connector.  It may also require the use of multiple PSUs instead of having big PSUs. This was for illustrative purposes only.)

In the example above, the difference may seem negligible.  Now let's say you have 1000 servers each pulling 500 W.  You'll need roughly twice the amount of copper if you do 120V instead of 208V.  What about getting lots of power into a rack?  Do you have enough rackspace for the PDUs you need?  If you design for higher voltages you should be able to handle more power without needing more PDUs.

One voltage that is very intriguing but mostly unfamiliar in the U.S. is 415V three-phase.  415V has the interesting feature of providing 240V when wired as single-phase (415V / sqrt(3) = 240V) without the requirement of a transformer.  Most PSUs support up to 240V or 250V so almost anything that runs on 208V will also run on 240V.  415V is also wired differently (wye) and much easier to do power calculations with.

One really nice feature of 415V is that it is about twice the voltage of 208V.  You can get almost twice the power (watts) using the same sized wiring.  P=IV (power = current * voltage), so double the volts and you double the power while holding the current constant.  That doesn't magically mean that your building will get more power from the power company, but it does mean that thinner, cheaper wiring can be used to get power from the power company to your servers.  It also means you can effectively double the wattage going into a rack versus 208V by using the same (physically) sized PDUs and wiring.  More watts flowing over the same sized wiring (at a higher voltage) means you spend less money on copper.

You might ask yourself "but who sells PDUs that do 415V?".  Pretty much everyone.  "But my power supplies only handle 208V inputs!"  I would double check on that.  Look for a sticker on the power supplies that says "100-250V", "200-250V", "200-240V", or similar for the voltage ratings.  If everything specifies an upper limit of 240V or higher, you're good.  "But my wiring probably doesn't support anything higher than 208V."  That's a real concern but many electrical busways are designed to handle up to 480V or 600V.  Check the labeling and definitely consult with an electrician to determine if it's possible.

Convincing your facilities group to do something different won't be easy and it usually won't be worth redoing an existing server room.  If you're starting from scratch, find a good datacenter consulting firm and see what they think about running 415V.  It's a shame that most PSUs don't support 277V because that's the single-phase output from 480V three-phase, a voltage commonly used in large appliances like server room air-conditioning units.  Your building probably has 480V running through it already and anything else requires extra transformers to be used.  At least 415V is common for PDUs and I believe is also common in Europe.  If I were to design a datacenter from scratch, I would almost certainly want 415V three phase.

Other options include doing 480V three phase to the row and using an in-row step-down transformer to go to 208V at the row.  That gives you the benefit of smaller, cheaper wiring per watt all the way to the row but has the added cost of additional in-row transformers.

Other random note

It is quite likely that the path from the power company to a server room PDU or UPS goes something like this:
high voltage line (12kV or some other large number) -> building transformer (12kV:480V) -> transformer (480V:208V) -> PDU or UPS

Did I miss anything?

Please comment if there's something else I should add.  Corrections are also appreciated.

Extra Reading

For extra reading about electricity and electronics in general (though not three phase), I highly recommend Darren Ashby's excellent book, Electrical Engineering 101, Third Edition: Everything You Should Have Learned in School... But Probably Didn't.  He does a great job of explaining things at a level that was easy to understand but with enough information to learn the material.