February 2010   Thermal Design andManagement of Servers
Thermal Design and Management of Servers | February 2010ContentsAbstract	                                                 ...
Thermal Design and Management of Servers | February 2010IntroductionAny computer or device providing services can technica...
Thermal Design and Management of Servers | February 2010must plan to manage over 20kW in the next five years. The trendtow...
Thermal Design and Management of Servers | February 2010Thermal management should be carried out for a server in orderto i...
Thermal Design and Management of Servers | February 2010Thermal Challenges in ServersThe rise in power densities, performa...
Thermal Design and Management of Servers | February 2010    Sl.     Technology                     Cooling                ...
Thermal Design and Management of Servers | February 2010    Sl.     Technology                     Cooling                ...
Thermal Design and Management of Servers | February 2010                                           Fig. 5: Sublevel NodeTh...
10Thermal Design and Management of Servers | February 2010Innovative Thermal Solutions• Multi-Core Processors Cooling  – D...
11Thermal Design and Management of Servers | February 2010Thermal Management of Telecommunication ServerA typical telecomm...
12Thermal Design and Management of Servers | February 2010• HDD air flow channel  – HDDs were cooled with a flow duct, whi...
13Thermal Design and Management of Servers | February 2010Acronyms1U		                          One Rack Height (Equals 1....
14Thermal Design and Management of Servers | February 2010ABOUT HCLHCL TechnologiesHCL Technologies is a leading global IT...
Upcoming SlideShare
Loading in …5
×

HCLT Whitepaper: Thermal Design and Management of Servers

474 views

Published on

In today’s digital age of rapid knowledge development, an enormous amount of information is being generated every day across the world. This data needs to be stored, processed and secured so the user can access this data quickly. Servers play a major role in this type of data-intensive business applications. The advancements in
hardware, software and miniaturization technologies, along with the information evolution, has led to a vast increase in servers power
densities and computing power. To improve the reliability and to enhance performance, thermal management needs to be performed
in servers by removing the heat generated by the devices. This paper focuses on the role of thermal management of servers in data centers and green data centers. It also investigates the challenges
faced in thermal design and management of servers. The emerging cooling technologies which have evolved over the years in the server
industry will be discussed. Case studies on thermal management of servers will be presented

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
474
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

HCLT Whitepaper: Thermal Design and Management of Servers

  1. 1. February 2010 Thermal Design andManagement of Servers
  2. 2. Thermal Design and Management of Servers | February 2010ContentsAbstract 2Introduction 3Role of Servers in Data Centers 5Thermal Challenges in Servers 6Innovative Cooling Solutions 6HCL Case Studies 8Conclusion 12References 12Acronyms 13Authors 13ABOUT HCL 14AbstractIn today’s digital age of rapid knowledge development, an enormousamount of information is being generated every day across theworld. This data needs to be stored, processed and secured so theuser can access this data quickly. Servers play a major role in thistype of data-intensive business applications. The advancements inhardware, software and miniaturization technologies, along withthe information evolution, has led to a vast increase in servers powerdensities and computing power. To improve the reliability and toenhance performance, thermal management needs to be performedin servers by removing the heat generated by the devices.This paper focuses on the role of thermal management of servers indata centers and green data centers. It also investigates the challengesfaced in thermal design and management of servers. The emergingcooling technologies which have evolved over the years in the serverindustry will be discussed. Case studies on thermal management ofservers will be presented.© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  3. 3. Thermal Design and Management of Servers | February 2010IntroductionAny computer or device providing services can technically be calleda server. In the hardware sense, server means a computer modelintended for running software applications under the heavy demandof a network environment. With the evolution of the internet, theamount of information exchange with the server is vast. In worldwide web applications, servers are playing a major role in helpingthe data reach the user in fractions of seconds. A typical serverconsists of multi-core CPUs, DIMMs, hard drives, power supplyunits, network connections, etc. The servers were classified basedon their applications and are shown in Fig. 1. Servers General Application based based Personal Enterprise Communication Modem Computer application server Server Database DNS Server Game Server Web Server server Print Server Fig. 1: Servers classificationAs information increased every day, server capabilities increasedin line with demanding business applications. IDC[3] estimatesthat server system density has increased by 15% annually over thelast 10 years as organizations have moved from pedestal servers torack-optimized systems, and now to extensive implementation ofblade servers (Fig. 2). Fig. 2: Worldwide server installed base by form factor, 1996-2010 (Source: IDC, 2006)This shift toward smaller form factors has increased the demandson power and cooling management at the rack level. While theaverage power consumption per rack in the year 2000 was 1kW,datacenter managers today must account for 6.8kW per rack, and© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  4. 4. Thermal Design and Management of Servers | February 2010must plan to manage over 20kW in the next five years. The trendtoward high density has resulted in hot spots within the datacenterthat are subject to failures and reliability concerns.As servers became more powerful and compact, their numbersincreased with the information evolution. Figure 3[3] shows theworldwide server installed base, new server spending and power,and cooling expenses. One interesting fact can be observed fromFig. 3, i.e. power and cooling expenses were approaching thenew server spending. This means the power and cooling expenseswere going to outweigh the cost of the hardware and software inthe future. Fig. 3: Worldwide server installed base, new server spending , and power and cooling expense (Source: IDC, 2006)Fig. 4 shows the worldwide thermal management market trenddone by BCC Research, USA[11]. It says that technology spendingincreased to an estimated $6.8 billion by the end of 2008 and shouldreach $11 billion by 2013. This means investment in thermalmanagement is growing strongly. Fig. 4: World thermal management market trend from the year 2007 to 2013 (Source: BCC research, USA)© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  5. 5. Thermal Design and Management of Servers | February 2010Thermal management should be carried out for a server in orderto increase the performance and reliability, as it’s consuming morepower and dissipating more heat. Computational Fluid Dynamics(CFD) simulation software and advances in thermal managementtechniques evolved over the years to meet the cooling demands inever growing servers. With the help of CFD software and emergingtechnologies in the industry, thermal engineers could providethermal solutions to ever-growing, power-hungry servers.Role of Servers in Data CentersA data center is a collection of computer servers usually maintainedby an enterprise to accomplish server needs far beyond the capabilityof one machine. These centers run enormously scaled softwareapplications with millions of users. As data centers increasinglybecome the nerve centers of business and society, the demand forbigger and better ones increased. There is a growing need to producethe most computing power per square foot at the lowest possiblecost in energy and resources, all of which is bringing a new level ofattention and challenges.The growth in the number of servers and the Internet is driving towardmore energy consumption. As servers become more powerful, morekilowatts are needed to run and cool them. As data centers growto unprecedented scales, attention has shifted to making serversless energy intensive. Uptime’s[12] Brill notes that while it once took30 to 50 years for electricity costs to match the cost of the serveritself, the electricity on a low-end server will now exceed the servercost itself in less than four years. The huge power draws have spurredinnovation through computational fluid-dynamics modeling in thethermal management of servers, from the component level to racklevel and to data center level.Servers for Green Data CentersA green data center is fundamentally a repository for thestorage, management, and dissemination of data. In this center,the mechanical, lighting, electrical and computer systems weredeveloped to optimize energy efficiency and environmentalimpact. The operational scope of a green data center design for theIT industry includes the following aspects:• Minimizing the carbon footprints of buildings• Use of alternative energy technologiesA green data center focuses on two primary objectives, namelyenergy conservation and environmental safety. This is achievedthrough optimizing the performance of cooling systems usingreal-time sensing technology. To resolve this issue, green datacenters perform sensor-based optimal cooling.A green data center optimizes power consumption and coolingright from the chip to the chiller. Thus, green data centers withreal-time smart cooling[4] functionality can provide up to 35-40%savings in terms of power consumption.© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  6. 6. Thermal Design and Management of Servers | February 2010Thermal Challenges in ServersThe rise in power densities, performance, and reliability constraintswill produce major hurdles for the thermal management ofservers. These thermal challenges will vary with different productsand requirements. The following were the challenges faced inthe servers:• Very high power dissipation• High ambient temperature• Stringent thermal requirements (HDDs, PSUs, Processor, etc.)• Harsh environment• Meeting strict standards and compliances• Miniaturization• Product design cycle time reduction• Minimizing the thermal cost• Feasibility solution using available resourcesConsidering all the above challenges for servers, the electronicscooling industry has come up with path breaking innovativesolutions time after time.Innovative Cooling SolutionsThe solutions for thermal challenges have led to innovation inthermal management. The latest technologies in the thermalmanagement arena function in and around the basic heat transfermodes. The development has reached a stage where the technologiesoverlap the basic functional industrial domains. The developmentof technologies is moving from single-phase heat transfer to multi-phase heat transfer, which has led to the design of advanced coolingsolutions. Table 1 describes the innovative cooling solutions whichcan be used in servers. Sl. Technology Cooling Description No. Type Technique This is an important cooling technique where more heat will be Conduction 1 Conduction transferred to surrounding ambient Cooling through the conduction heat transfer. This is an extruded surface, where Heat Sink extended area will help to increase the heat transfer coefficient. When heat sink solution is not sufficient, a fan will be used Fan to increase the heat transfer General Cooling 2 coefficient. This solution is used Technologies extensively. Combination of fan and heat sink will enhance the chances of Fan and Heat providing a solution in less time. Sink Both options will increase the heat transfer rate.© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  7. 7. Thermal Design and Management of Servers | February 2010 Sl. Technology Cooling Description No. Type Technique It has applications in high power Cryogenics electronics and high speed high Cooling performance computing. It will be used when sub-cooled Refrigerant temperatures are required in a Cooling system. Direct The components/system will be Immersion directly immersed in a liquid bath. Cooling More than one coolant will be used. Hybrid Cooling will be achieved through Cooling more than two different modes of Advanced heat transfer. 3 Cooling Liquid will flow across micro Technologies channels drilled in a chip and Micro boiling heat transfer will occur. Channel Heat transfer rate is high in this technique. Liquid jet will be sprayed across a Spray chip and heat will be removed from Cooling the chip through liquid vaporization. High conductive material with the flow passes will be in contact with Cold Plate the power dissipated components. Technologies Low melting point liquid will flow through the passes and remove heat from the chip. Electro Heat will be removed from the chip Wetting with electron movement. In this process, wherever hot spots are present, a cold fluid will be blown across the spot, and Spot Cooling cooling will be achieved. This can be achieved directly or indirectly through a fan. In this technique, heat sink base, which is in contact with the chip, will be filled with a low melting liquid. As Vapor heat is transferred from the chip to Smart Cooling Chamber 4 the base, it will be dissipated to the Technologies Cooling ambient through an evaporation and condensation process at the vapor chamber. These are hollow tubes filled with a low melting point liquid, with the tube wall and liquid separated by a Heat Pipes/ wick. This is a very high conductive Heat Super material which will help disburse the Conductors heat with low resistance. This is a very effective, noise-free technique which will be used to remove heat from the processor chips.© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  8. 8. Thermal Design and Management of Servers | February 2010 Sl. Technology Cooling Description No. Type Technique With the miniaturization in the place, Compact heat will be removed from the Heat system to ambient through compact Exchangers heat exchangers. This will be used in transient cooling in conjunction with heat sink, where either heat sink base or complete heat sink will be filled Phase with this material. As it absorbs the Change heat, material phase will change Material from solid to liquid, and there it will maintain constant for a while. Later, it reverts to solid phase. In the future, this can be used in server’s components cooling. Smart Cooling 4 This works on the Peltier effect. Technologies Micro TECs These have applications in the sub Cooling ambient cooling. This has application in today’s multi-core processors where some of the cores may be very hot and eTECs the temperature must be brought (Embedded down to allowable limit. TECs will be TECs) inscribed at a certain portion of the die, and these will take care of the die temperature. A very high velocity jet will be Jet pumped across a chip and heat Impingement will be blown away with the liquid. Cooling Rapid cooling takes place. Heat transfer rate is high. Table 1: Innovative Cooling SolutionsHCL Case StudiesHCL has successfully provided innovative thermal managementsolutions for various servers, from the concept phase to the fullproduct life cycle phase of the product. The following two casestudies illustrate HCL’s capability in thermal management solutionsfor servers.Thermal Management of High End ServerA high end server consists of as many as 13 sublevel nodes packedin a compact chassis in two rows. Sublevel node is shown in theFig. 5. Each sublevel node consists of six IO cards, 32 DIMM cards,four multi-core processors, one voltage transformation module andtwo DCA channels. The total power dissipation of each node is2.4kW and the total power dissipation of full high end server is31.2kW. Providing a thermally feasible optimal solution to thishigh end server is very difficult and challenging.© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  9. 9. Thermal Design and Management of Servers | February 2010 Fig. 5: Sublevel NodeThermal Challenges• Multi-Core Processors Cooling – Total power dissipation = 920W – Less space availability and also preheated air will flow across the processors, making the design more challenging – Need to use advanced cooling technologies from the industry to cool these next generation processors• DIMMs Cooling – Total power dissipation = 736W – Space constraint between DIMMs – Less flow rate available• Voltage Transformation Module (VTM) Channel Cooling – Total power dissipation = 370W – It has one of the critical flow path designs in the server – VTMs are placed in a narrowed flow channel through which fluid enters the modules – this is one of the geometric constraints for the thermal design – Second constraint is available flow rate to cool the modules• System Pressure Drop Optimization – As it is a very high power dissipating system, there is a need to optimize the system in such a way that maximizes the heat transfer and minimizes the pressure drop© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  10. 10. 10Thermal Design and Management of Servers | February 2010Innovative Thermal Solutions• Multi-Core Processors Cooling – Designed and optimized a customized heat sink which will fulfill the thermal requirements – Heat sink has been designed in such a way that it minimizes the weight and reduces the cost – Base and center fins of the heat sink were designed as copper, and extreme fins were made up of aluminum – Heat pipes were used to connect through the aluminum fins and the base of the heat sink – This combination of different heat sink materials and heat pipes is providing allowable temperature limit for processor – This innovative design added value to the customer• DIMMs Cooling – Nova chip which controls the DRAMs cooled with custom made optimized heat sink – Nova chip heat sink was designed with a constraint on the height of the heat sink (distance between any two respective DIMMs was less) and the less available flow rate across the DIMMs – The innovative design of Nova chip heat sinks across all DIMMs made thermal cooling possible for high dissipating DIMMs• Voltage Transformation Module Channel Cooling – Two different innovative concepts were proposed for VTM channel cooling – One is a generic solution with innovative optimized individual heat sinks for each VTM within a narrow flow channel. These new heat sinks were fully custom made – The other is a novel concept in which we achieved a thermal solution with the available flow rate within a narrow flow channel. This is achieved fully with conduction cooling and partly through convection cooling – Optimal pressure drop is achieved to maximize the flow rate through the channel• System Pressure Drop Optimization – From the start of the thermal solution, emphasis has been on the pressure drop for each and every module across the system – Pressure drop optimization helped to reduce the acoustic related problems with fully utilizing the available flow rate across the system© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  11. 11. 11Thermal Design and Management of Servers | February 2010Thermal Management of Telecommunication ServerA typical telecommunication server consists of a power supply unit,hard disk, IO board, motherboard, network switching board, and apower distribution board. This server is a transformation from 1Urack to 2U rack. The total power dissipation of the server is 640W.A thermal solution to this 2U rack server at sea level, as well as highaltitude conditions, must be provided. Figure 6 shows the overviewof telecommunication rack. Fig. 6: Telecommunication RackThermal Challenges• High ambient temperature• Cooling HDDs• Fan locations were fixed, most of the flow is taking the path of least resistance• Providing a solution at high altitude conditions• Meeting the strict compliances and standards of the productSmart Cooling Solutions• Flow deflectors/ducts were used to utilize the available flow across the unit• Four dedicated flow channel were designed to control the flow behavior inside the system. – HDDs air flow channel – Network switch board and PDB flow channel – CPU flow channel – IO Board flow channel© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  12. 12. 12Thermal Design and Management of Servers | February 2010• HDD air flow channel – HDDs were cooled with a flow duct, which will direct the flow from the fan to HDD without any preheat in the air• Network switch board and PDB flow channel – Custom made heat sinks were designed to cool the components on these two boards• CPU flow channel – Processor was cooled with a custom made heat sink with a heat pipe and heat sink advanced cooling optionConclusionThis paper gave a brief description of the evolution of servers andtheir power consumption. As servers are very important in datacenters, thermal management of servers in data centers and greendata centers was also highlighted with descriptions of the innovativecooling technologies which have evolved in the industry. Presentthermal challenges faced by servers and the innovative solutionswhich emerged from the industry were discussed. Two case studies(high end server and communication server) illustrating HCL’scapability in the complete thermal management of servers werediscussed. These case studies exemplify the application of advancedtechnologies from the electronics cooling industry to achieveoptimal feasible workable thermal solutions.References1. http://en.wikipedia.org/wiki/Server_(computing)2. Tom Vanderbilt, ‘Data Center Overload’, (http://www.nytimes. com/2009/06/14/magazine/14search-t.html?_r=1)3. Jed Scaramella, ‘Enabling Technologies for Power and Cooling’, Sept 2006, IDC4. ‘Data Center Cooling Strategies’, Technology Brief, www.hp.com5. Qpedia, April 2009, Vol. III, Issue. II, Advanced Thermal Solutions, Inc., www.qats.com/qpedia6. Qpedia, July 2009, Vol. III, Issue. VI, Advanced Thermal Solutions, Inc., www.qats.com/qpedia7. Hossam Metwally, ‘Methods for Evaluating Advanced Electronics Cooling Systems’, Fluent Inc, WP-1038. ‘Power and Cooling in the Data Center’, www.amd.com9. Christian L. Belady, P.E, ‘In the data center, power and cooling costs more than the IT equipment it supports’ (http://electronics-cooling. com/articles/2007/feb/a3/)10. Lisa Stapleton, ‘Getting smart about data center cooling’, November 2006, (http://www.hpl.hp.com/news/2006/oct-dec/power.html)11. BCC Research http://www.bccresearch.com/report/SMC024E.html12. Uptime Institute (http://www.uptimeinstitute.org/)© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  13. 13. 13Thermal Design and Management of Servers | February 2010Acronyms1U One Rack Height (Equals 1.75 in or 44.45mm)2U Two Rack Height (Equals 3.5 in or 88.9mm)CPU Central Processing UnitCFD Computational Fluid DynamicsDCA Distributed Converter AssemblyDIMMs Dual In-line Memory ModuleDRAM Dynamic Random Access MemoryHDDs Hard Disk DrivesIO Card Input Output CardPDB Power Distribution BoardPSU Power Supply UnitTEC Thermo Electric CoolerAuthorsJagadish Thammanna is a Manager and Heads the CFD andThermal team at HCL Technologies. He has 15 years of experiencein Thermal management in all the niche domains and variouscross-application industries. His areas of interest include ComputationalFluid Dynamics, heat transfer and scientific programming. In hisvast experience, he has presented and published many national andinternational papers at technical symposiums.Benarji Nalamala received his MS degree specialized in HeatTransfer from the Indian Institute of Technology Madras in2006. He is a Thermal Analyst at HCL Technologies Ltd. He hasover 4 years of experience in thermal design and management ofelectronic equipment in various domains. He has provided novelcooling solutions for various electronic devices with emergingcooling technologies from the industry. His areas of interest include‘Computational Fluid Dynamics and Heat Transfer’.© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
  14. 14. 14Thermal Design and Management of Servers | February 2010ABOUT HCLHCL TechnologiesHCL Technologies is a leading global IT services company, workingwith clients in the areas that impact and redefine the core of theirbusinesses. Since its inception into the global landscape after itsIPO in 1999, HCL focuses on ‘transformational outsourcing’,underlined by innovation and value creation, and offers integratedportfolio of services including software-led IT solutions, remoteinfrastructure management, engineering and RD services andBPO. HCL leverages its extensive global offshore infrastructureand network of offices in 26 countries to provide holistic, multi-service delivery in key industry verticals including FinancialServices, Manufacturing, Consumer Services, Public Servicesand Healthcare. HCL takes pride in its philosophy of ‘EmployeeFirst’ which empowers our 54,443 transformers to create a realvalue for the customers. HCL Technologies, along with itssubsidiaries, had consolidated revenues of US$ 2.3 billion (Rs.11,270 crores), as on 30th September 2009. For more information,please visit www.hcltech.comAbout HCL EnterpriseHCL is a $5 billion leading global Technology and IT Enterprisethat comprises two companies listed in India - HCL Technologies HCL Infosystems. Founded in 1976, HCL is one of India’soriginal IT garage start-ups, a pioneer of modern computing, anda global transformational enterprise today. Its range of offeringsspans Product Engineering, Custom Package Applications,BPO, IT Infrastructure Services, IT Hardware, SystemsIntegration, and distribution of ICT products across a wide rangeof focused industry verticals. The HCL team comprises over62,000 professionals of diverse nationalities, who operate from26 countries including over 500 points of presence in India. HCLhas global partnerships with several leading Fortune 1000 firms,including leading IT and Technology firms. For more information,please visit www.hcl.in© 2010, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.

×