This document discusses the Project GreenLight, which aims to minimize the energy consumption of cyberinfrastructure through instrumentation and measurement. It focuses on enabling more efficient computing options for key user communities like metagenomics and digital media. The project constructs an energy instrumented modular data center to house specialized computing clusters and measure their energy usage. It also explores direct DC power from solar/fuel cells and virtualization techniques to improve efficiency and reduce energy costs of computational science. The goals are to better understand optimization of work per watt and develop middleware to automate optimal power strategies.

1. Project GreenLight Presentation for the … … Dr. Gregory Hidley California Institute for Telecommunications and Information Technology, UCSD

2. ICT is a Key Sector in the Fight Against Climate Change Applications of ICT could enable emissions reductions of 7.8 Gt CO2e in 2020, or 15% of business as usual emissions. But it must keep its own growing footprint in check and overcome a number of hurdles if it expects to deliver on this potential. www.smart2020.org

3. ICT Industry is Already Actingto Reduce Carbon Footprint

4. Application of ICT Can Lead to a 5-Fold GreaterDecrease in GHGs Than its Own Carbon Footprint “While the sector plans to significantly step up the energy efficiency of its products and services, ICT’s largest influence will be by enabling energy efficiencies in other sectors, an opportunity that could deliver carbon savings five times larger than the total emissions from the entire ICT sector in 2020.” --Smart 2020 Report Major Opportunities for the United States* Smart Electrical Grids Smart Transportation Systems Smart Buildings Virtual Meetings * Smart 2020 United States Report Addendum www.smart2020.org

5. GreenLight Motivation: The CyberInfrastructure (CI) Problem Compute energy/rack : 2 kW (2000) to 30kW in 2011 Cooling and power issues now a major factor in CI design IT industry is “greening” huge data centers … but today every $1 spent on local IT equipment will cost $2 more in power and overhead Academic CI is often space constained at departmental scale Energy use of growing departmental facilities is creating campus crises of space, power, and cooling Unfortunately, little is known about how to make shared virtual clusters energy efficient, since there has been no campus financial motivation to do so Challenge: how to make data available on energy efficient deployments of rack scale hardware and components?

6. The NSF-Funded GreenLight ProjectGiving Users Greener Compute and Storage Options PI is Dr. Thomas A. DeFanti $2.6M over 3 Years to construct GreenLight Instrument Start with Sun Modular Data Center Sun Has Shown up to 40% Reduction in Energy Measures Temperature at 5 Levels in 8 Racks Measures power Utilization in Each of the 8 Racks Chilled Water Cooling input and output temperatures Add additional power monitoring at every receptacle Add web and VR interfaces to access measurement data Populate with a variety of computing clusters and architectures Traditional compute and storage servers GP GPU arrays and specialized FPGA based coprocessor systems DC powered servers SSD equipped systems Turn over to investigators in various disciplines Measure, Monitor and Collect Energy Usage data With the goal of maximizing work/watt

7. The GreenLight Project: Instrumenting the Energy Cost of Computational Science Focus on 5 Communities with At-Scale Computing Needs: Metagenomics Ocean Observing Microscopy Bioinformatics Digital Media Measure, Monitor, & Web Publish Real-Time Environmental Sensor Output Via Service-oriented Architectures Allow Researchers Anywhere To Study Computing Energy Cost Enable Scientists To Explore Tactics For Maximizing Work/Watt Develop Middleware that Automates Optimal Choice of Compute/RAM Power Strategies for Desired Greenness Partnering With Minority-Serving Institutions Cyberinfrastructure Empowerment Coalition Source: Tom DeFanti, Calit2; GreenLight PI

9. NLR Cavewave

10. Cinegrid

11. Cisco

13. DWDM or Gray OpticsOn-Demand Physical Connections Your Lab Here Microarray Source:Phil Papadopoulos, SDSC/Calit2

14. GreenLight Goals: More Work/Watt Build a full-scale virtualized device, the GreenLight Instrument Measure then minimize energy consumption Develop middleware to automate optimal choice of compute/RAM power strategies Discover better power efficiency configurations and architectures Teach future engineers who must scale from an education in Computer Science to a deeper understanding in engineering physics Measure, monitor, and make publicly available, via service-oriented architectures, real-time sensor outputs Focus on 5 communities: metagenomics, ocean observing, microscopy, bioinformatics, and digital media Allow researchers anywhere to study the energy cost of at-scale scientific computing

15. GreenLight Research activitiesLeading to Greener CI Deployments Computer Architecture – FPGA, GP GPU systems Rajesh Gupta/CSE Software Architecture – Virtualization, memory management, networking and modeling Amin Vahdat, Ingolf Kruger/CSE CineGrid Exchange – mixed media storage, streaming, and management Tom DeFanti/Calit2 Visualization – Using 2D and 3D modeling on display walls and CAVEs Falko Kuster/Structural Engineering, Jurgen Schulze/Calit2 Power and Thermal Management Tajana Rosing/CSE DC Power Distribution Greg Hidley/Calit2 http://greenlight.calit2.net

16. Calit2/UCSD [http://greenlight.calit2.net] Monitoring, Modeling and Management GLIMPSE Decision Support System http://glimpse.calit2.net

17. Situational Awareness Calit2/UCSD [http://greenlight.calit2.net] 12 Dashboard interface “Tap” for details Power utilization Enterprise reach Multiple perspectives

18. Datacenter vitals 2010.08.20 Calit2/UCSD [http://greenlight.calit2.net] 13 Input/Output sampling Live/average Fan speeds Live/Average data Live Temperature Heat Exchangers Environmentals

19. Domain specific views 2010.08.20 Calit2/UCSD [http://greenlight.calit2.net] 14 Control elements Real-time heatmap Realistic models

20. Airflow dynamics 2010.08.20 Calit2/UCSD [http://greenlight.calit2.net] 15 Live fan speeds Airflow dynamics

21. Heat distribution 2010.08.20 Calit2/UCSD [http://greenlight.calit2.net] 16 Combined heat + fans Realistic correlation

22. Heat Trends Calit2/UCSD [http://greenlight.calit2.net] 17 Heat exchangers Hotspot identification Trends over past 24h

23. Past changes in airflow Calit2/UCSD [http://greenlight.calit2.net] 18 Fan slices rpm Heat distribution changes Potential for failures Trends over past 24h

24. Power spikes Calit2/UCSD [http://greenlight.calit2.net] 19 IT assets Computation zone Unused asset Average load Peak computation 1 minute resolution

25. Zoom-in Analysis Calit2/UCSD [http://greenlight.calit2.net] 20 History over several days. Zoom on desired time range. Hint on each sample point. Automatic average area. Multiple sensors per asset with up to 1 min sampling resolution.

26. The GreenLight Project Focuses on Minimizing Energy for Key User Communities Microbial Metagenomics Ocean Observing Microscopy Bioinformatics Digital Media—CineGrid Project Calit2 will Host TBs of Media Assets in GreenLightCineGrid Exchange to Measure and Propose Reductions in the “Carbon Footprint” Generated by: File Transfers and Computational Tasks Required for Digital Cinema and Other High Quality Digital Media Applications

27. GreenLight Project: Putting Machines To Sleep Transparently 22 Rajesh Gupta, UCSD CSE; Calit2 Laptop Network interface Peripheral Low power domain Secondary processor Network interface Management software Main processor, RAM, etc SomniloquyEnables Servers to Enter and Exit Sleep While Maintaining Their Network and Application Level Presence

28. Improve Mass Spectrometry’s Green Efficiency By Matching Algorithms to Specialized Processors Inspect Implements the Very Computationally Intense MS-Alignment Algorithm for Discovery of Unanticipated Rare or Uncharacterized Post-Translational Modifications Solution: Hardware Acceleration with a FPGA-Based Co-Processor Identification and Characterization of Key Kernel for MS-Alignment Algorithm Hardware Implementation of Kernel on Novel FPGA-based Co-Processor (Convey Architecture) Results: 300x Speedup & Increased Computational Efficiency Large Savings in Energy Per Application Task

29. Virtualization at Cluster Level for Consolidation and Energy Efficiency Source: Amin Vadhat, CSE, UCSD Fault Isolation and Software Heterogeneity, Need to Provision for Peak Leads to: Severe Under-Utilization Inflexible Configuration High Energy Utilization Usher / DieCast enable: Consolidation onto Smaller Footprint of Physical Machines Factor of 10+ Reduction in Machine Resources and Energy Consumption Original Service Usher Virtualized Service

30. DC Power: UCSD is Installing Zero Carbon EmissionSolar and Fuel Cell DC Electricity Generators UCSD 2.8 Megawatt Fuel Cell Power Plant Uses Methane Available Late 2011 San Diego’s Point Loma Wastewater Treatment Plant Produces Waste Methane 2 Megawatts of Solar Power Cells at UCSD, 1 MW to be Installed off campus

31. Zero Carbon GreenLight Experiment:DC-Powered Modular Data Center Concept—Avoid DC to AC to DC Conversion Losses Computers Use DC Power Internally Solar and Fuel Cells Produce DC Both Plug into the AC Power Grid Can We Use DC Directly (With or Without the AC Grid)? DC Generation Can Be Intermittent Depends on Source Solar, Wind, Fuel Cell, Hydro Can Use Sensors to Shut Down or Sleep Computers Can Use Virtualization to Halt/Shift Jobs Experiment Now Underway Collaboration with Sun, EPRI, DPTI and LBNL NSF GreenLight Year 2 and Year 3 Funds Sun Box <200kWatt Source: Tom DeFanti, Calit2; GreenLight PI

32. DC Power Distribution Today's data center AC based distribution model: 480-V AC power from the grid is converted to DC to charge a battery-based UPS DC stream from the UPS is converted to AC and transformed to 208-VAC for distribution The above AC is then rectified back to 380-VDC in each server's power supply DC distribution offers a comparatively simpler scheme: a single rectifier turns 480-V AC into 380-V DC that both charges the UPS and supplies the servers. Energy Power Research Institute (EPRI) and Duke Energy Corp. measured a 15 percent reduction in power consumption in a test of 380-V DC distribution at the utility's Charlotte, N.C., data center. Net energy savings could be twice that, they claim, once the cooler-running equipment's reduced air conditioning burden is factored in.

33. UCSD Two Rack DC Experiment

34. Questions?