Data Center Lessons Learned


Published on

Data Center Lessons Learned at an Intel data center. Innovations in cost and energy savings in high-density data centers including: air economizer, retrofit of factory builiding, high efficiency air-cooled cabinets, and a container data center proof-of-concept.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Data Center Lessons Learned

  1. 1. Data Center Lessons Learned CIO Council Meeting Friday, March 27th 2009 Human Services Department 729 St. Michaels Drive Santa Fe, NM
  2. 2. Bio: Tom Greenbaum, CDCP <ul><li>Intel Information Technology employee for 8 ½ years, worked on the Intel Rio Rancho campus all that time </li></ul><ul><li>Data Center Operations Manager. Responsible for all Information Technology data center rooms on the Rio Rancho site </li></ul><ul><li>Room owner for the Encanto Supercomputer </li></ul><ul><li>Owner of Intel’s first Container Data Center </li></ul>Tom at the back of the Container Data Center <ul><li>Speaking Events: </li></ul><ul><ul><li>Data Center Lessons Learned. New Mexico Technology Council (NMTC), Santa Fe, NM, March 13, 2009. </li></ul></ul><ul><ul><li>Spherical Metaphor (Spheriphor) For Geoscope Multi-Dimensional Data Visualization . The 5th International Symposium on Digital Earth (ISDE5), UC Berkeley, June 5, 2007. </li></ul></ul><ul><ul><li>Full-Dome Immersive Business Visualization (BizViz) . Intel Information Services and Technology Group, Technical Community Conference, Granlibakken Resort, Lake Tahoe, Oct. 24-26, 2005. </li></ul></ul><ul><ul><li>Application Analysis for Factory Network . Computer Security Institute (CSI) 31st Annual Conference, Washington D.C., November 8, 2004. </li></ul></ul><ul><li>Interests: </li></ul><ul><ul><li>The Art of Innovation, Process Improvement, CMMI, Information Visualization, 3D Animation, Raytracing, Haptics, Arcitectural Design, Light Electric Vehicles, Sustainable Design, Living Off-The-Grid, Digital Fine Art, Virtual Globes, Spherical and Geodesic Geometry, R. Buckminster Fuller, Synergetics, Dome Construction, Nanotechnology, Tensegrity, Stained Glass, Woodworking, Xeriscaping, Science Fiction </li></ul></ul>Tom is President of the local Albuquerque AFCOM Chapter Tom’s personal website:
  3. 3. New Mexico Data Centers Energy Saving Innovations, Factory Reuse, High-Density Air-Cooled Cabinets, Air Economizer, Container Data Center, Supercomputer <ul><li>Support the full range of IT customers including: </li></ul><ul><ul><li>Design Engineering (chip design validation) </li></ul></ul><ul><ul><li>Office users </li></ul></ul><ul><ul><li>Manufacturing </li></ul></ul><ul><ul><li>Enterprise systems </li></ul></ul><ul><li>Customer systems include: </li></ul><ul><ul><li>Storage arrays </li></ul></ul><ul><ul><li>Tape backup systems </li></ul></ul><ul><ul><li>Virtual tape libraries </li></ul></ul><ul><ul><li>Virtual server environments </li></ul></ul><ul><ul><li>High memory servers </li></ul></ul><ul><ul><li>Large batch computing pools </li></ul></ul><ul><li>In total, the IT data centers on the Rio Rancho site contain close to 7,000 servers </li></ul><ul><li>One of Intel's global Hub data centers </li></ul><ul><li>Supports Intel’s data center consolidation strategy </li></ul><ul><li>Several data center rooms with different missions and infrastructure </li></ul>Encanto Supercompter 3D Rendering by Tom Greenbaum
  4. 4. Solar Power Proof-of-Concept <ul><li>Does NOT supply power to data centers: proof-of-concept for larger arrays </li></ul><ul><li>64 Sharp solar panels photovoltaic (PV) array </li></ul><ul><li>Peak yield of 11.2kW of Direct Current power </li></ul><ul><li>Intel Rio Rancho’s first solar energy system </li></ul>Over a 25 year lifespan, the Intel Rio Rancho solar system, will offset an estimated: 907,000 lbs of CO2, the leading greenhouse gas Tom Greenbaum demonstrating his Electric Bicycle at the Solar Panel Ribbon Cutting Ceremony
  5. 5. Factory Reuse <ul><li>Retrofit Existing Factory Building </li></ul><ul><li>Cost Avoidance Of $ Millions </li></ul><ul><li>3 Story Factory Conversion </li></ul><ul><ul><li>1st Floor UPS/Battery Electrical Room </li></ul></ul><ul><ul><li>2nd Floor Data Center </li></ul></ul><ul><ul><li>3rd Floor Air Handling And Cooling </li></ul></ul><ul><li>6 Megawatts Of IT Power </li></ul><ul><li>Innovative Air-cool Design </li></ul><ul><ul><li>No Computer Room AC (CRAC) </li></ul></ul><ul><ul><li>No Raised Metal Floor (RMF) </li></ul></ul><ul><ul><li>Chimney Racks </li></ul></ul><ul><li>Strong Load-bearing Floor Enabled High-density Computing </li></ul><ul><ul><li>2,220 lbs Per Rack </li></ul></ul><ul><ul><li>30 kW Per Rack </li></ul></ul><ul><li>Managed Cable Overhead </li></ul>Schematic Building Section by Tom Greenbaum Proof-of-Concept for mobile data center, non-production, validate cost savings, JIT compute, DR, energy savings. 480 Container DC Home to Encanto Supercomputer, future home of IT GHDC Phase 3. 55,000 Encanto SC Trailer data center. Subject of Air Economizer POC, whitepaper and video. EOL - completely decommissioned 2008 Q04 1000 Trailer DC RMF, CRAC units, redundant cooling, generator power, mission critical, storage, manufacturing, Office/Enterprise. 1713 Mission Critical 2 RMF, CRAC units, redundant cooling, generator power, mission critical, storage, tape backup, VTL, EC and Office/Enterprise. 2047 Mission Critical 1 High-density high-performance, over 6,000 EC blade servers, non-production virtual servers. 6775 HPDC Manufacturing support systems, moved to Mission Critical 2. EOL - completely decommissioned 2008 Q04 1261 MFG Support Notes Area (Sq Ft) Data Center
  6. 6. High-Density Air Cooled Cabinets <ul><li>Optimized Air Management </li></ul><ul><ul><li>Chimney cabinets closely ducted to air handling units </li></ul></ul><ul><ul><li>Reduce leakage to 1% </li></ul></ul><ul><ul><li>Reduce obstructions to air flow at rear of servers </li></ul></ul><ul><ul><li>Increase pressure differential to draw more air thru cabinets. </li></ul></ul><ul><ul><li>Result = maximum kW per cabinet </li></ul></ul><ul><li>Heated air is contained and isolated from the cooling air </li></ul><ul><li>Equalize temperature of available air at both top and bottom of cabinet </li></ul><ul><li>Data Center room at a comfortable 68 o F </li></ul><ul><li>Heat transfer from hot air to air handling unit (AHU) cooling coils is maximized </li></ul>Achieve Maximum Cooling Capacity At Lowest Total Operational Cost With Air-Cooled “Chimney” Cabinets Row of High-Density Blade Server Chimney Cabinets in HPDC
  7. 7. Air Economizer Proof Of Concept <ul><li>Used an air economizer to cool 900 high-performance production servers </li></ul><ul><li>Cooled with 100% outside air at temperatures up to 90 o F </li></ul><ul><li>Data showed that nearly all DC cooling could be supplied with an air economizer, greatly reducing power consumption </li></ul><ul><li>Potential savings for a 10-MW DC: </li></ul><ul><ul><li>Reduce annual operating costs by up to $2,870,000 </li></ul></ul><ul><ul><li>Save 76,000,000 gallons of water annually </li></ul></ul><ul><li>In 2008 the PoC won the prestigious Intel Innovation Award </li></ul><ul><li>Received a lot of industry praise as leading the way towards reduced power consumption and Green Computing </li></ul><ul><li>We cooled servers with 100 percent outside air at temperatures up to 90 degree F </li></ul>Challenged the current industry assumption that the usefulness of air economizers is limited by the need to supply cooling air at a relatively low temperature Trailer Data Center 3D rendering by Tom Greenbaum
  8. 8. Container Data Center (CDC) Proof Of Concept <ul><li>Evaluation of containerized technology and capabilities </li></ul><ul><li>40’ ISO standard roll-on, roll-off shipping container containing: </li></ul><ul><ul><li>Computer racks </li></ul></ul><ul><ul><li>Electrical distribution </li></ul></ul><ul><ul><li>Cooling coils </li></ul></ul><ul><ul><li>High density blade servers </li></ul></ul><ul><ul><li>3,520 compute nodes </li></ul></ul><ul><li>Costs savings </li></ul><ul><ul><li>Construction </li></ul></ul><ul><ul><li>Energy </li></ul></ul><ul><ul><li>Operational </li></ul></ul><ul><li>Just-In-Time compute </li></ul><ul><li>Disaster recovery </li></ul><ul><li>Closely matching cooling capacity + IT power consumption can achieve improved energy efficiency </li></ul>
  9. 9. Encanto Supercomputer <ul><li>Debuted as the 3rd fastest in the world (now ranked 12). </li></ul><ul><li>Entire installation of the supercomputer took only four days! </li></ul><ul><li>Savings cost avoidance of building a separate data center </li></ul><ul><li>Reuse of factory power, cooling and architectural infrastructure </li></ul><ul><ul><li>1 megawatt </li></ul></ul><ul><ul><li>600 GPM chilled water system </li></ul></ul><ul><ul><li>Load bearing floor </li></ul></ul><ul><li>3 Exemplar systems (2.1 TFLOPS peak) </li></ul><ul><ul><li>UNMHPC at UNM </li></ul></ul><ul><ul><li>NMSU </li></ul></ul><ul><ul><li>NMIT </li></ul></ul><ul><li>172 TFLOPS Altix ICE 8200 Cluster (SGI) </li></ul><ul><ul><li>1792 Nodes </li></ul></ul><ul><ul><li>3,584 Sockets </li></ul></ul><ul><ul><li>14,336 Cores </li></ul></ul><ul><ul><li>28 Racks </li></ul></ul><ul><ul><li>28.7 TB memory </li></ul></ul><ul><ul><li>172 TB parallel file system (10GB/sec) </li></ul></ul><ul><ul><li>20 TB NFS Backup Storage system </li></ul></ul>Governor Richardson shaking hands with Tom Greenbaum at ribbon cutting
  10. 10. Encanto Supercomputer (continued) Governor Bill Richardson Announces Partnerships & New Mexico’s Supercomputer to Create New High-Tech Jobs