Micro PUE. The Key to Data Center Energy Savings.


Published on

In this paper, we propose a new metric to support the goal of reducing Macro PUE.
This new data is logically constructed along the same formula lines as Macro PUE but is
based on sets of individual IT and cooling energy usage data and, we therefore refer to
it as Micro PUE.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Micro PUE. The Key to Data Center Energy Savings.

  1. 1. Micro PUE. The Key to Data Center Energy Savings. A White Paper By Robert Hunter and Chet SandbergRevision 1
  2. 2. OverviewMost in the data center world have heard of the metric Power Usage Effectiveness, orPUE. PUE was one of the first metrics developed to give data center managers an ideaof the energy efficiency of their operations as compared to others. This metric wasdeveloped by The Green Grid and has since been adopted by the EnvironmentalProtection Agency as the standard for determining Energy Star achievements for datacenters in the United States.PUE is defined as follows:As a brief description, PUE is the average amount of energy used to process and cooleach 1 kWh of IT load throughout a data center. The numerator in the formula includesthe IT load, the cooling load and, to a lesser extent, electrical equipment efficiencylosses. IT and cooling loads comprise the vast majority of power usage in this equation.It is important to understand that the PUE calculation produces a single, macro figurefor the entire data center. While PUE is a key to understanding the overall energyefficiency of a computer room, a single summary number does little to help manageindividual components in order to increase energy usage effectiveness. Therefore theneed to supplement Macro PUE with a more granular metric becomes evident.In this paper, we propose a new metric to support the goal of reducing Macro PUE.This new data is logically constructed along the same formula lines as Macro PUE but isbased on sets of individual IT and cooling energy usage data and, we therefore refer toit as Micro PUE.How PUE is Driven by Cooling EfficiencyIn order to understand how to lower PUE, be it either Macro or Micro, it’s important tounderstand the math behind the formula. Recall that PUE is defined as follows:If we break that formula into its components, we get the following:Because IT Energy appears in both the numerator and the denominator, a change inthat value will have much less effect on the outcome of PUE than other components of 1
  3. 3. the formula. If you couple this with the fact that cooling energy is typically 3 to 5 timesthe combined total of electrical equipment efficiency losses, it becomes clear thatcooling energy is the primary component that influences the outcome of the PUEcalculation. (Lighting, for the purposes of this discussion is ignored as it accounts forless than 1% of the equation.)To illustrate this, consider the following example of a data center with a PUE of 2.00.Three cases are presented each which reduces one component by 10%. The resultsare rather startling. As can be seen, a reduction of 10% in cooling energy has asignificantly higher effect on lowering Macro PUE than the lowering of either IT load orincreasing electrical efficiency. In fact, lowering the IT load actually raises PUE.Base Case. Macro PUE = 2.00Case 1. Lower IT load by 10% from base case (efficiency loss from UPS and PDUdrops same percentage of kW). PUE actually rises by 4.7%.Case 2. Lower Electrical Efficiency Loss by 10% from base case. PUE drops by lessthan 1%. 2
  4. 4. Case 3. Lower Cooling load by 10%. PUE drops by over 4.2%.While the example in Case 1 may seem improbable (IT load decreases but cooling loadstays the same) it is not as uncommon as one might think. In sites where virtualizationis employed, IT load reductions are common but, remaining hot spots and newlycreated mismatches between IT heat and cooling resources often stifle expectedreductions in cooling energy usage. Thus, simply reducing IT loads without the ability tomanage cooling energy is actually counterproductive in the attempt to lower PUE.On the other hand, Case 3 shows how reducing cooling loads vis-à-vis IT loading canlower PUE by a significant amount. In this case, a 10% reduction in cooling energyusage leads to a reduction in PUE of slightly more than 4.2%. While these examplesuse a hypothetical starting point of a Macro PUE of 2.00, the same effects could beseen from other PUE values as well. In fact, the higher the starting PUE position, themore effect will be seen by changing cooling energy. These figures make it clear thatPUE is really a measure of the effectiveness of cooling energy usage as compared to ITloading.Given those organizations such as the Uptime Institute show average PUE values of 2.5or morei, it’s clear that data center managers must begin to focus anew on loweringcooling energy in relation to IT load in order to reduce PUE. Lowering PUE will requireaccess to cooling energy data and the ability to manage cooling loads in relation to ITenergy use. The principal of Management by Information (MBI) seems very appropriateto offer assistance in the goal of managing cooling loads and PUE. MBI states that:“You can only manage what you measure and that you can only manage well what youmeasure accurately.” It stands to reason, then, that both IT energy and cooling energyusage must be measured accurately and continuously.It is clear that the ability to measure the IT circuits and cooling loads form a keycomponent to managing PUE. The EPA has already codified PUE as the principalcalculation for their data center Energy Star program but, their initial meteringrequirements include only metering for the whole data center and for the IT load at theUPS output. However, in releasing its new standards, the EPA has made the followingstatement: “meters located at the PDU output, or on the servers directly, may be extremely valuable for yourorganization. These allow for a more advanced calculation of PUE which can help you measure and improveEffectiveness of power distribution at your facility. Therefore EPA encourages you to install this type of meter at ii The use of branch circuit meters at the PDU (power distribution unit) andyour facility. “metered power strips at server level are increasing in popularity and this impetus by the 3
  5. 5. EPA is likely to accelerate this trend. The crux of their statement is that such submetering allows for “a more advanced calculation of PUE” that can improve both themeasurement and management of data center energy efficiency. Micro PUE is formedon the basis of sub metering data for both IT loads and cooling loads.Sub-metering, and more specifically, branch circuit sub metering can provide significantvalue for managing energy and billing energy usage. While many are focusing onbranch circuit metering for IT circuits within PDU’s, sub metering can also be applied toindividual cooling units and to the cooling system as a whole. It is logical that measuringand managing energy data for cooling loads would be highly advantageous as well andcould well be the missing piece of data that so many have been looking for. It also isintuitive that, if reasonably small gains from cooling savings can be achieved, the cost ofbranch circuit metering of these circuits could have a relatively short payback time andprovide significant long term benefits.It can be seen that measuring and managing cooling energy at the micro, or individualunit level, is necessary to reduce PUE. With that in mind, this paper will examine thespecifics of a metric with that goal in mind: Micro PUE. The following section gives abackground on the philosophy of Micro PUE and discusses how it is calculated.Measuring Micro PUEWe have seen that changes in cooling energy vis-à-vis IT load are the key in movingPUE as a number. Unfortunately, as a single number for all data center energy usage,PUE can indicate the direction of energy effectiveness but, it cannot isolate the causesof inefficiencies. If one is to manage and reduce PUE, it will be necessary to view thecooling energy in relation to IT energy at the individual device level, that is, theindividual cooling unit. That will require the knowledge of the energy used by theindividual cooling units and the knowledge of the appropriate share of IT energy that isremoved by each cooling unit.On the surface, that appears to be a daunting challenge. However, sub meteringtechnologies now exist that would allow users to complete the first task, themeasurement of energy use of each air conditioning unit. The ability to allocate the pro-rata share of IT load to each cooling unit requires a bit of math and physics. But, we willdemonstrate that the amount of IT load directly attributable to each cooling unit can beaccurately measured. These two pieces of data will enable us to create a measurementof Micro PUE.Sub-metering individual cooling unitsMeasuring the energy use of cooling units requires the ability to measure the twoprimary types of cooling systems that are common to data centers; Computer Room Air 4
  6. 6. Conditioning (CRAC) systems and Computer Room Air Handler (CRAH) systems. In aCRAC system, each data center cooling unit has its own compressor for refrigerant andeach unit releases its heat via a remote heat rejection unit such as an outdoorcondenser or dry cooler. In a CRAH system, each air handler unit in the data centerrelies on a central chiller to provide cool liquid and that chiller, in turn, uses a heatrejection unit such as a cooling tower.Sub-metering, and more specifically, branch circuit sub metering can provide significantvalue for managing energy and billing energy usage. While many are focusing onbranch circuit metering for IT circuits within PDU’s, sub metering can also be applied toindividual cooling units and to the cooling system as a whole. It is logical that measuringand managing energy data for cooling loads would be highly advantageous.Energy consumed by CRAC units and their heat rejection units as well as CRAH unitsand their chillers can be measured at the source or at the central panel level. If coolingcircuits are clearly marked and accessible within electrical panels, this is the easiest andmost economical method of measuring cooling circuits with branch circuit metering. Ifpanels are not centrally located then some of the cooling circuits may need to bemonitored at the individual unit itself.Measuring the pro-rata share of IT load associated with each cooling unitTo get to a point where we can measure the Micro PUE of each air conditioner unit, wemust measure the pro-rata share of IT energy and electrical infrastructure energy thateach unit removes. Fortunately, we can take advantage of some physics related toenergy to provide this data. The key fact that comes into play here is that the energyconsumed by each IT device is equivalent to the heat that it releases. Physics teachesus that energy is neither created nor destroyed but simply changes states. A greatexample of this can be seen in a data center where the electrical energy consumed asmeasured in kilowatt hours is the same as the heat energy released in kilowatt hours. 5
  7. 7. These two, kilowatt hours of electrical energy consumed and kilowatt hours of heatradiated are identical, they are simply two sides of the same coin. If we can measurethe amount of heat being removed by each cooling unit, we would then know the exactamount of energy being consumed by the IT equipment and electrical infrastructurewhose heat is being removed by that cooling unit.As the figure on the below shows, electrical energy is used and then radiated first by theUPS, then the PDU and finally the IT equipment. The total amount of electrical energyconsumed is equal to the amount of heat energy released.The removal of heat energy is traditionally measured in British Thermal Units, or BTU’s.The BTU’s of heat removed can be calculated by measuring the difference in supply airor liquid temperature vs. return air or liquid temperature times the flow in cubic feet perminute for each cooling unit. The formula for calculating BTU’s is as follows: ( ) Where k is a constant associated with the coolant (e.g. air, water, etc.)BTU’s removed by each air conditioner unit can then be converted into watt hours ofheat removed through the mathematical relationship between the two:With the data now converted into kilowatt hours of heat removed by each CRAC orCRAH unit, one can now know the exact amount of wattage of energy being removedby the unit. With this information, Micro PUE can now be calculated.Micro PUE is simply the PUE of each individual cooling unit and is defined as follows: 6
  8. 8. Because all heat loads, whether from IT or electrical efficiency losses, must pass-through and be removed by individual cooling units, the wattage of heat removedautomatically includes each of the IT loads and their electrical losses. One needs onlyallocate the respective portion of IT vs. efficiency loss by using the procedure describedin the above paragraph to yield a correct answer for Micro PUE.The total kilowatt hours of heat removed is calculated through measuring BTU’s. Thetotal energy used by the cooling units is measured by a branch circuit meter. Lastly, theIT portion of the wattage of heat being removed is simply: kWh of Heat Removed by Cooling Unit N * (1- Efficiency loss of UPS and PDU)These efficiency losses are relatively fixed on a day-to-day basis, although they dochange somewhat over time. In general, one can look at the kilowatts in vs. kilowattsout on a UPS to get electrical efficiency loss. The loss of a PDU would beapproximately 4% for a standard efficiency transformer and 1.5% for a high efficiencytransformer, as discussed by Mazzetti iii, in a thorough analysis of UPS and PDUefficiency losses. We have added 1% in the examples in this paper to this to allow forline losses inherent in breakers, junction boxes and wire length losses bringing anestimate of standard PDU losses to 5%.As a brief description, Micro PUE is the actual amount of energy used to process andcool 1 kWh of IT load through a given cooling unit. Thus, Micro PUE looks at PUEthough each IT/cooling segment of a data center. As a granular measurement, itfocuses on discovering the relative efficiencies or inefficiencies of each data centersegment, providing the visibility to users necessary to effectively manage their PUE.It should be noted that, the measurement for the heat removed by all of the CRAC andCRAH units will likely be slightly different than total IT load and efficiency lossesbecause of shell loads from outside heat or cooling. However, studies by the EPA haveshown that, regardless of climate, shell loads add or subtract only a small percentageenergy compared to the total IT load and electrical efficiency losses. Such shell loadsare assumed but never quantified in Macro PUE but can, for the first time, be quantifiedby Micro PUE.Practical uses of Micro PUE for lowering Macro PUEWith Micro PUE data, the user will now have the ability to see their cooling units rankedin order from most efficient to least efficient in terms of energy usage effectiveness. Inorder to reduce Macro PUE, changes may be made to the dynamics of the hardware ofthe air conditioning unit, to the air flow patterns of the heat being removed, or both. 7
  9. 9. The ability to see Micro PUE also provides a view of the actual movement of heat fromthe IT cabinet source to the cooling unit that removes the heat. This can be seen byaddition, subtraction or movement of IT heat load vis-à-vis the corresponding addition orsubtraction to the heat removed by individual or combinations of cooling units. Whilecalculations such as Computational Fluid Dynamics provide estimates of heatmovements, Micro PUE offers the user the ability to see the actual track of heat flows inreal-time and historical trends.Micro PUE can expose the actual causes of cooling inefficiencies. Some of the leadingcauses of cooling inefficiencies have been identified by organizations that have studiedthese problems and include the following:  Chiller/Compressor Cycling. This condition is analogous to driving a vehicle in stop-and-go city traffic as opposed to a constant speed on the freeway. In stop- and-go traffic, the vehicle is constantly cycling up-to or through its optimum energy savings band (measured in Miles per Gallon or MPG) and never achieves its optimum MPG range. A chiller or compressor operates in a similar manner. Each unit has an optimum zone of maximum energy efficiency (measured in BTU’s or tons – 12,000 BTU’s – removed per kilowatt hour). By cycling above and below this zone, the user wastes an enormous amount of energy. The ability to see chiller or compressor energy usage in relation to BTU’s removed can allow users to tune each CRAC or CRAH unit and chiller for maximum efficiency.  Fighting CRAC or CRAH units. This condition is analogous to a server’s use of two power supplies at the same time as opposed to using one supply with the second in standby. Two or more air units are said to be fighting one another when they compete to remove the same heat load of a nearby heat source such as a data cabinet. In this situation, one or more units are always being underutilized while another is either over-utilized. This creates a problem for the underutilized units similar to that of chiller/compressor cycling. That is, one or more units are always operating well below the optimum efficiency band and significant amounts of energy are wasted. By seeing the Micro PUE of each unit, especially units that are in close proximity, users can detect a condition of fighting. This can be alleviated by the use of Variable Frequency Drive fans to coordinate heat removal and/or by the use of a master system controller unit.  Low Delta T. Because Delta T is an inherent component of BTU measurement to compute Micro PUE, it is suggested that it and all components of Micro PUE be graphed and stored as individual pieces of data. This allows conditions such as low Delta T to be quickly identified as a cause of high Micro PUE. Low Delta T can be alleviated in a number of ways including the use of hot isle/cold isle 8
  10. 10. technologies, direct venting of IT heat load to return air, lowering the supply air temperature and others.SummaryThis paper has shown that PUE is essentially a measurement of the cooling efficiencyof a data center in relation to its IT load. The use of Macro PUE to manage data centerenergy effectiveness, without the use of its micro-components, has been demonstratedto be a difficult if not impossible task.The ability to measure, trend and manage by Micro PUE has been introduced. MicroPUE was discussed with its ability to reduce or even cure many of the common reasonsfor data center cooling efficiency losses. As such, we propose that Micro PUE is avaluable and perhaps vital tool for data center managers in achieving energyeffectiveness goals set with Macro PUE.i Power Usage Effectiveness, Mark Fontecchio, ManageDataCenter.com, May 6, 2008ii Energy Star Rating for Data Centers, Frequently Asked Questions, July 2010iii William Mazzetti, Where Did My Effectiveness Go? William Mazzetti, Data Center Guru, May 10, 2009 9