State Of The Market Mission Critical Facilities


Published on

Research Report on Data Center Facilities for Jacobs Enginnering 2008

  • Be the first to comment

  • Be the first to like this

State Of The Market Mission Critical Facilities

  1. 1. Analysis of the Market for Mission Critical Facilities (Focus on Data Centers) Conducted for Jacobs Global Buildings North America April – July, 2008 This summary represents a synthesis of online reports, presentations, articles, survey research, white papers, interviews and Webinars from dozens of industry experts, groups, associations, market analysts, media, service providers and government agencies and programs. Specific sources are cited throughout the report and summarized in Appendix A
  2. 2. Table of Contents Executive Summary i Recommendations iii Current Situation Market Size / Scope 2 Demand Drivers / Challenges 3 Breakdown of Data Center Costs 4 Breakdown of Data Center Energy Costs 5 Impacts of Efficiency Improvements 8 Energy Usage Growth Predictions 9 How the Industry is Responding 10 Consolidation Facilities 11 Systems (virtualization) 12 Evolving Data Center Designs 15 New Tools / Technology 16 Containers 18 Subterranean Facilities 19 Floating Data Centers 20 Locating to Secondary Markets 21 Colocation / Outsourcing 25 DOE / EPA Energy Management Programs / Initiatives DOE / EPA Organizational Context 29 Goals 31 Take-up of Green Initiatives 33 Barriers to Change 34
  3. 3. Table of Contents National Data Center Energy Efficiency Information Program 35 Metrics & Benchmarking 36 Database of State Incentives 39 Convergence of Facilities & IT 40 Appendix A (Information Sources) 41 Appendix B (Overall IT Spending) 44 Appendix C (Infrastructure Characteristics by Space Type – EPA) 45 Appendix D (Vendor Container Products) 46 Appendix E Reference Materials (see separate attachment) Appendix F Snapshot of Providers & Scope of Services (see separate attachment) Appendix G More Detailed Snapshot of Providers and Services (separate attachment)
  4. 4. Executive Summary • Many of today’s data centers have reached their limits on capacity, efficiency and ability to support power and cooling needs. While IT spending is tightening, companies are making major investments in IT infrastructure: - “More than 70% of the world’s Global 1,000 organizations will be unable to host next generation, high density equipment.” Source: Gartner Oct. 2007 Trendwatch - 70% of AFCOM’s 3000 member organizations intend to move, build or expand their facilities within the next 5 to 10 years • Rapid advances in technology, unprecedented growth of digital data and the need to control costs -especially for power and cooling- are creating a paradigm shift in the way data centers are designed, operated and managed. Organizations that do not adapt may risk their ability to survive • While two years ago, energy efficiency was low on the list of data center managers’ concerns, “power and cooling” is now the number 1 issue – going green is not a choice, but an economic imperative. - As directed by law, the DOE and EPA are directed to guide how data centers are operated and designed - The agencies have initiated a joint National Data Center Energy Efficiency Information Program to coordinate a number of activities around metrics and benchmarking, tools and training, certification of experts, equipment performance specifications, recognizing best-in-class data centers - One of the program’s first endeavors is a Data Collection Initiative to Develop and Energy Star Rating for Data Centers. To date, over 215 data centers from more than 60 organizations have signed up to participate in the data collection process including JCPenney Co., ADP, Inc., Digital Realty Trust, Inc. UPS, The Green Grid and Data Center Operations Council. • The landscape of products, services and solution providers is more diverse, complex and fragmented than ever. Distinctions between technology, hardware, software, architecture, engineering and outsourcing and facilities providers are blurring. • Consequently, users have a wider range of options for meeting increased demand and improving efficiency. - States are offering economic incentives for companies and service providers to move to secondary locations including Utah, Alabama, Ohio, Indiana, Nebraska, Iowa, Virginia and Texas. In fact, the Invest in Iceland Agency recently cited a PricewaterhouseCoopers Belgium study that describes Iceland as an attractive option for the growing data center market - A number of utilities are also offering economic incentives for data centers specifically - Data center containers, floating and subterranean facilities are options, which deliver benefits including flexibility, lower - i -
  5. 5. Executive Summary cost, security and energy advantages - The number of collocation / outsourcing providers is growing as is the demand; not just for space and infrastructure, but for technology, applications, talent and a wide range of managed services • According to experts, next generation data centers will take advantage of modular, repeatable design, scalable capacity planning, predictable operations and increased use of specialized data center services. Data centers are becoming more organic and hence, flexibility is the name of the game. • The roles of IT managers and facilities managers, though typically separate and distinct, will need to become integrated. Most data center managers don’t see the utility bill – often, the data center’s utility usage isn’t broken out at all • Cloud computing will continue to gain momentum. More capabilities and services will move to the network - ii -
  6. 6. Recommendations • Consider the whole range of issues and options in today’s mission critical facility marketplace • In particular, engage the Energy and Power group - determine opportunities to leverage energy-specific expertise and to collaborate • Have discussions with all Jacobs Building NA market leaders about opportunities for their clients • Review the “Snapshot of Providers & Scope of Services” (Appendix F) to get a broad brush view of the current competitive landscape and to identify potential alliances / teaming partners • Assess our client’s situation - Fact gathering to understand their unique situation. Use this report as a basis for discussion to confirm our understanding of the marketplace and where they are along the “Belief to Action” continuum. - Understand their level of awareness, attitudes and usage of specific providers; and specifically how Jacobs ranks • Be prepared to educate clients about EPA initiatives and help them sort through, assess and leverage the best mix of alternatives for them (regardless of whether EPA initiatives stick in the long-run, we should understand them to convey marketplace leadership) • Define potential scenarios, corresponding value propositions/offerings and points of competitive differentiation to test with clients - Energy assessment? - IT/Facilities integration? - 2-year capacity program? • Ensure that a robust mix of facilities and technical expertise is integrated in our offerings – perhaps create an interim CIFO (Chief Information Facilities Officer) role. • Become recognized as a leader in the marketplace by - iii -
  7. 7. - Actively participate in select groups and initiatives, such as Kevin Dickens role on the Mission Critical’s editorial board - Leverage opportunities, such as the one offered by Deal Architect Inc., to submit points of view, information and insights a • Understand the status of our own mission critical data center facilities and the plans we have for our own organization - iv -
  8. 8. Data center design is one of the most complex and important undertakings for any company. Ten years ago, the process was about finding a room with enough power and air conditioning. Today, data center pros are expected to have expertise in computing, networking, electrical and mechanical engineering, thermodynamics and disaster mitigation -- the list could go on. Source: 08 Feb 2007 | Page 1 of 49
  9. 9. CURRENT SITUATION "Despite the challenging market conditions, companies are making major investments in IT infrastructure, reflecting the critical nature of these assets to today’s corporations,” says Michael F. Foust, CEO of Digital Realty Trust. 2007 was "The Year of the Data Center," becoming the focal point for fundamental changes in computing, involving massive shifts toward cloud computing platforms and a more energy- efficient, virtualized infrastructure. As predicted, these trends continue to drive infrastructure investment, both in the form of huge data centers and massive spending on hardware and software to save money and Market Size / Scope Gartner Trendwatch, Oct. 2007 More than 70% of the world’s Global 1,000 organizations will be unable to host next generation, high-density equipment In 5 years, power requirements will outpace current requirements and CIOs will have to refurbish, build or seek alternatives, such as hosting Tom Rosato CEO Fortress Int’l Market for MCFs in the U.S. is about a $51B and is expected to reach $100B by 2010 or 2011 John Slve, Input analyst Computerworld 2-08 Government spending on outsourcing application mgnt and services, including data centers, will grow from $6B to $8B in 5 years (Total Govn’t IT spending is about $70 billion) Len Eckhaus AFCOM founder 70% of AFCOM’s 3000 member organizations say they are moving, building or expanding their facilities within the next five to 10 years Oct-Nov 2007 survey 80% are building, renovating or expanding their data centers in 2007 or 2008. 20% remain….. Page 2 of 49
  10. 10. energy as IT operations expand their scale. Data Center Knowledge Digital Realty Trust Inc. (DRT) May 2007 study Over the next 24 months, more than 80% of U.S. and European companies surveyed said they were or had plans to expand their data centers Refer to Appendix B for a Summary of Overall IT Spending Factors Driving Data Center Demand Top Data Center Challenges Page 3 of 49
  11. 11. • Globalisation (follow-the-sun and for flexibility) • Broader deployment of applications that increase demand for enterprise storage such as − Enterprise software − Data warehousing − Data recovery − Data security operations • IT displacing traditional distribution channels • Unprecedented growth of digital data - use of multimedia and converged communications (bandwidth) • Ability to transfer data in real-time • Legal requirements − Compliance (Sarbanes-Oxley, HIPPA) − Data retention − Discovery requests • Security − External threats − Protecting data Source: IDC INSIGHT “Building, Planning and Operating the Next Generation Data Center” Michelle Bailey IDC OPINION (n=197) • IT organizations are under ever increasing pressure to provide 24/7 availability, greater functionality and higher levels of security at the lowest possible cost. For years, data centers met growing demands by installing more and more servers, which led to inefficiencies. The problem is that most data center computers run at 15% or less of capacity, though consuming electricity all the while. Vernon Turner, an analyst for IDC Page 4 of 49
  12. 12. • According to a survey conducted by DatacenterDynamics, the key issues identified for 2008, center on the hub areas of Energy Efficiency together with the convergence, consolidation and automation of facilities and IT management. Note that in a survey conducted in 2006, “power and cooling” was cited, but ranked last on the list of top challenges One Estimated Breakdown of Data Center Costs Page 5 of 49
  13. 13. Sample There is a wide variance in estimated breakdowns of data center and energy costs Page 6 of 49
  14. 14. Typical Data Center Energy End Use Data Center Server Load 51% Other 13% Data Center CRAC Units 25% Office Space Conditioning 1%Lighting 2% Electrical Room Cooling 4% Cooling Tower Plant 4% Source: “High-Performance Data Centers: A Research Roadmap.” Lawrence Berkeley National Laboratory, March 30, 2004. Page 7 of 49
  15. 15. Data center power use consists of information technology loads and infrastructure loads. - IT loads include: servers, disk drives, and network equipment. - Infrastructure loads: cooling, fans, pumps, lighting and UPS. On average about half the energy coming into the data center goes to the IT equipment, the bulk of the remaining energy goes to cooling the center (Koomey, 2007) 60 percent of the data center electrical load is used to power IT equipment, with approximately - 56 percent of that being used to power servers, - 27 percent for storage, and - 19 percent for network equipment. October 11, 2007 survey conducted by the Data Center Users’ Group (DCUG), formed by Emerson Network Power Page 8 of 49
  16. 16. Breakdown of Data Center Facility Costs Breakdown of Power Costs Page 9 of 49
  17. 17. 9% 7% 2% 82% Land Core & Shell Architecture Mechanical / Electrical Source Mike Manos, Microsoft 10% 50% 12% 25% 3% Electricity / Transformer / UPS IT Equipment Air Movement Cooling Lighting Source: EYP Mission Critical Systems Page 10 of 49
  18. 18. Page 11 of 49
  19. 19. How Data Centers Use Power Today 34% 33% 18% 9% 1% 3% 1% 1% IT Equipment Coolers UPS Computer Room Air Condition Power Distribution Unit Humidifier Lighting Main Switchgear Generator Page 12 of 49
  20. 20. Source: APC Page 13 of 49
  21. 21. Impacts of Virtualization While virtualization has improved efficiencies, it has pushed power and cooling requirements to the point that data centers built only a few years ago are unable to support new high-density 1U and Blade Servers. Source 2008 NAAT, Inc. • The average power consumed by an enclosure in a data center is about 1.7 kW, but the maximum power that can be obtained by filling a rack with available high density servers, such as blade servers, is over 20 kW. • Such loads greatly exceed the power and cooling design capabilities of the typical data center, including many Tier IV centers • Data center operators have very little experience with enclosures drawing over 10 kW, but recent trends suggest that many will be confronted with the need to install and provision power and cooling for high density racks either singly or in groups. Despite efficiency gains, “an increasing number of data centers (many built in the 1980s or 1990s) in both the private and public sectors are hitting their limits in terms of space, power and cooling capacity,” Lowell Sachs, senior manager of federal government affairs at Sun Microsystems. Page 14 of 49
  22. 22. Many industry leaders also site Jevons’ Paradox, which is the observation that greater energy efficiency, while in the short- run produce energy savings, may in the long-run result in higher energy use. It was first noted by the British economist W. Stanley Jevons, in his book The Coal Question published in 1865, where he argued that “it is a confusion of ideas to suppose that the economical use of fuel is equivalent to diminished consumption. The very contrary is the truth." Page 15 of 49
  23. 23. Energy Usage Predictions ASHRAE March 1, 2008 • “Estimates show that the total energy usage by data centers approaches 2% of the U.S. electricity usage - equivalent to about eight 1000-MW power plants,” said Roger Schmidt, chair of ASHRAE’s technical committee on mission critical facilities, technology spaces and electronic equipment. “ Gartner predicts data center power and cooling crisis June 14 2007 • By next year, about half the world's data centers will be functionally obsolete due to insufficient power and cooling capacity to meet the demands of high-density equipment. • In a poll of 125 users at the Gartner data center conference in November, 68% said either power or cooling is their primary data center's greatest facility problem • In a survey this year of 374 North American data center managers, 43% said either power or cooling was the factor most limiting data center growth Yankee Group - Jan 2008 • In 2000, power and cooling was approximately 20% of new server spend. Today it's slightly over 50% and will be over 80% by 2010 if nothing changes EPA EPA Report to Congress on Server and Data Center Energy Efficiency • In ‘06 U.S. data center energy costs were $4.5B. Worldwide costs were over $26B. According to the latest EPA report, typical data centers are consuming 40-100W per SF • If current trends continue, this demand would rise to 12 GW by 2011, and require 10 power plants. • Among the different types of data centers, more than one-third (38 percent) of electricity use is attributable to the nation’s largest (i.e., enterprise-class) and most rapidly growing data centers. Page 16 of 49
  24. 24. The Uptime Institute Rte of computational increase (Moore’s Law) is greater than the rate of increase of power efficiency - 2007 • According to a survey of 311 participants who manage or work within a data center 41.7 percent indicated that their data center would run out of power capacity in 12-24 months. • This finding supports previous Uptime Institute research which indicates that data center energy consumption may make or break enterprise computing profitability in the near future. More Data in the Data Center Tim Phillips, VP Int’l Rectifier • Cost to operate equipment over its lifetime is now about 3 Xs larger than the original hardware purchase “In the Data Center, the cost of Power and Cooling is greater than IT Equipment it Supports,” C. Belady, Electronics Cooling Magazine (Feb, 2007) Page 17 of 49
  25. 25. HOW THE INDUSTRY IS RESPONDING Capacity Security Service Addressing Competing Demands Energy Spending Overall Costs Consolidation • Of both equipment and facilities (higher density computing) being used to improve efficiency, increase utilization and to meet growing computing demands within constrained budgets Data Center Design • Containers, modular systems, subterranean facilities Secondary Markets • Becoming strategic locations for data centers – availability of land, access to power & tax incentives Colocation / Outsourcing • Demand for skills and capacity along with increasing data center complexity • Part of a broader desire to reduce capex IT spending and buy more as an opex service - SaaS, HaaS, PaaS Energy Initiatives • Higher density also results in higher energy usage and costs, which is counter to the EPA’s call for data centers to lower energy use -- EPA Report to Congress on Server and Data Center Energy Efficiency“IT hybrids” • Increased demand for professionals with both business acumen and technical know-how Sources Include - Network World “Security dominates 2008 IT agenda” January 30, 2008 - Deal Architect Inc., founded by Vinnie Mirchandani, former technology industry analyst (with Gartner), outsourcing exec (with PwC, now part of IBM) and founder of sourcing advisory firm, Jetstream Group - Data Center Knowledge “2007 was "The Year of the Data Center" - November 2007 “Data center construction is booming” IDC believes that a true datacenter transformation is taking place and that IT organizations will continue to consolidate the number of sites and number of supported systems and will increasingly place greater emphasis on energy efficiency. This will Page 18 of 49
  26. 26. consequently have a great impact on the operations and economics for the datacenter of the future. Source: IDC INSIGHT “Building, Planning and Operating the Next Generation Data Center” Michelle Bailey IDC OPINION (n=197) Page 19 of 49
  27. 27. DATA CENTER CONSOLIDATION Facilities October 31, 2007 Intel's Data Center Consolidation 133 data centers worldwide into just eight high-density facilities of about 300K sq. ft. apiece October 29, 2007 Marines Plan Huge Data Center Consolidation will reduce IT infrastructure from 300 data centers worldwide to30 facilities. October 16, 2007 Data Center Consolidation for Defense Unit The Defense Contract Management Agency (DCMA) recently reduced its data center network from 18 facilities to just two. August 22, 2007 State CIOs Keen on Consolidations - National Association of CIOs 2007 Survey of State Consolidation Strategies All but 2 of 29 completed or are planning a consolidation project June 18, 2007 Solar Host AISO.Net Consolidates Servers AISO.Net has further reduced power and cooling costs with a data center consolidation using IBM Cool Blue technology May 14, 2007 Verizon Consolidates to Save on Power data center consolidation from 10 to 3 and simplification, saving ~. $20 m in build-out costs May 09, 2007 Data Center Consolidation for First Data will reduce its U.S. network from 12 facilities to just three. Expected to take more than two years and cost approximately $225 million, according toPayments News. April 24, 2007 Data Center Consolidation for ABN, Barclays if the deal is finalized, according to, which says the companies will seek cost savings of 1.65 billion Euros (about $2.24 billion US) by 2010 May 17, 2006 HP Details Plans to Consolidate 85 data centers worldwide into six state of the art centers, two each in Atlanta, Houston and Austin, TX Others Include: ADP consolidated from 20 data centers to two Allstate consolidated from four data centers to two Page 20 of 49
  28. 28. Consolidation has moved from a focus on just reducing the number of installed systems to reducing the number of actual datacenters. IDC estimates there are 7,000 enterprise-class datacenters in the United States but 2.9 million server rooms and closets. The consolidation we are seeing involves closing smaller sites and moving assets to either existing or to new and larger enterprise-class datacenters. Source: IDC INSIGHT “Building, Planning and Operating the Next Generation Data Center” Michelle Bailey IDC Opinion Page 21 of 49
  29. 29. Systems Consolidation Virtualization • A technique for hiding the physical characteristics of computing resources from the way in which other systems, applications, or end users interact with those resources. Source: AIC Systems, IT strategy consulting, outsourcing, and IT support firm • Can make a single physical resource (server, operating system, application, or storage device) appear to function as multiple logical resources; or make multiple physical resources (such as storage devices or servers) appear as a single logical resource." Hardware Virtualization Network and Storage Virtualization Partitioning the computer's memory into separate and isolated "virtual machines" simulates multiple machines within one physical computer. It enables multiple copies of the same or different operating systems to run in the computer and prevents apps from interfering with each other. In a network, virtualization consolidates multiple devices into a logical view so they can be managed from a single console (see network virtualization). Virtualization also enables multiple storage devices to be accessed the same way no matter what their type or location (see storage virtualization). Application Virtualization Security Virtualization (the next frontier) Computer people love the word "virtualization," and vendors use the term for virtually anything. Numerous technologies fall under the umbrella of "application virtualization," some of which have been around for decades, while others are at the forefront. See application virtualization. It is hard to manage static security devices next to a pool of dynamic virtual servers and the security often gets in the way of virtualization benefits. Encapsulation and portability of virtual machines free your virtual servers to move from server to server. But if the security context can’t follow the servers, you’re forced to keep them stuck in place. Page 22 of 49
  30. 30. Server virtualization enables multiple operating systems to run on a single physical machine. With server virtualization workloads of underutilized server machines can be consolidated onto a smaller number of fully utilized machines. Source: Microsoft Page 23 of 49
  31. 31. Server Virtualization Adoption Source: IDC server virtualization multiclient study, 2007 n=410 • Virtualization is now considered a mainstream management tool for the datacenter • Move than 40% of he respondents reported that virtualization is now the default build for new server hardware unless a case can be made for standalone, unvirtualized server hardware • IDC expects that by 2011, more than 17% of all new servers will be actively virtualized. • Customers report average ratios of virtual machines to physical machines of 4:1 - Many customers are reporting 10:1 ratios - 40:1 ratios common for leading-edge adopters • By leveraging mobility tools that allow movement of virtual servers across different physical servers, customers are using virtualization as a means to increase availability and reduce planned and unplanned downtime. Page 24 of 49
  32. 32. • Using these same tools, customers are already architecting solutions that allow the movement of virtual servers across datacenters, and this is creating a new paradigm for disaster recovery. Page 25 of 49
  33. 33. Power Density Increasing by Approximately 15% annually Refer to Appendix C for EPA Overview of IT Equipment and Site Infrastructure System Characteristics, by Space Type Page 26 of 49
  34. 34. EVOLVING DATA CENTER DESIGNS Modular, Scalable, Predictable Modular, repeatable design Customers will reuse blueprints not only to speed time to deployment but also create a sense of predictability in the operation of the datacenter Scalable capacity planning Leading-edge customers are already taking a new approach to long-range planning for datacenter space. Rather than conditioning 50,000 sq ft or 100,000 sq ft all at once, IT organizations are purchasing or leasing the entire forecast space but only conditioning what they immediately need Predictable operations One of the greatest benefits of a building-block approach is improved understanding of how the datacenter operates which can be leveraged across the globe so that remote offices and datacenters can be managed by a central team with a specific understanding of the environment. A repeatable design also opens the door to the creation of "datacenter simulators," to test outages and impacts of changes without altering the physical production environment Increased use of specialized datacenter services Ultimately, customers will look externally for services that can help them across the entire life cycle of the datacenter. Design, construction, and consolidation services are already in high demand, and emerging services such as thermal assessments are gaining traction. Collocation and outsourcing services are also an option for customers that decide that day-today operations are best served by an external organization. Source: IDC INSIGHT “Building, Planning and Operating the Next Generation Data Center” Michelle Bailey IDC OPINION Page 27 of 49
  35. 35. Adoption of New Technology, Tools & Equipment According to a 2007 industry survey conducted by the Data Center Users’ Group (DCUG), formed by Emerson Network Power and completed in coordination with the U.S. EPA and Lawrence Berkeley National Labs to support the EPA’s recently released “Report to Congress on Server and Data Center Energy Efficiency,” the majority of respondents have made operational improvements to increase energy efficiency. - 77% have their data center arranged in a hot aisle/cold aisle configuration - 65% use blanking panels to minimize recirculation of hot air - 56% have sealed the floor to prevent cooling losses - 25% have conducted a computational fluid dynamics (CFD) analysis to identify hotspots and optimize airflow within the facility Traditional Raised Floor Cooling In-Rack and In-Row Cooling • Mainframes only averaged 25-50 watt/ per SF. • Originally everything was rows facing the same way -- a relatively efficient cooling method as all cold air went directly into the equipment cabinets - did not mix with warm air. • With rack mounted servers power levels began to rise to 35-75w/sf and cabinets all facing the same way become a problem - hot air exited out the back of one row into the front of the next • “Hot Aisle – Cold Aisle” began in the mid-to-late 90’s. CRAC units were still located mainly at the perimeter of the center, but floor tiles had vents (or were By 2011, in-rack & in-row cooling will be the predominant cooling strategy for high-density equipment. Gartner Non-Raised Floor Cooling • Some newer cooling systems do not require a raised floor and have placed the cooling system in close proximity to the racks. This improves cooling performance and efficiency. New Technology - Close Coupled Cooling • Cooling manufacturers have developed systems that shorten the distance air has to travel from racks to the cooling unit. - Some systems are “in-row” and others are “overhead”. Page 28 of 49
  36. 36. Adoption of New Technology, Tools & Equipment perforated) in the cold aisles. • Once past a certain power level there are multiple drawbacks, since it takes much more energy to deliver enough cold air into a single 2’x 2’ perforated tile to support a 30KW rack. • Perforated floor tiles have been replaced by floor “Grates.” Unfortunately 3.5KW per rack has been exceeded many times over with the advent of the “1U” and Blade Server. • Amount of power used to cool high- density server “farms” has actually exceeded the power used by the servers themselves. • In some cases for every $1.00 spent to power servers, $2.00 or more is spent for cooling. Blade Server: a server chassis housing multiple thin, modular electronic circuit boards, known as server blades. Each is a server in its own right. Blade servers allow more processing power in less rack space, simplifying cabling and reducing power consumption. Sometimes referred to as a high-density server - typically used in a clustering of servers dedicated to a single task - can also be managed to include load balancing and failover capabilities. June 2008 - Each increases ability to cool up to 20KW per rack. - With right infrastructure should lower cooling costs. Liquid Cooled Servers • Today all servers use air to transfer heat out of the server. • Manufacturers are exploring building or modifying servers to use “fluid based cooling”. This technology is still in its infancy - only in the testing and development stage. Because liquids can be a problem if leaked into electronic systems, this may cause a major hurdle to acceptance. This is different than using liquid (chilled water or glycol based) as a heat removal medium for the CRAC In-Row Cooling has a horizontal airflow pattern designed for hot aisle/cold aisle configurations. Closely coupling the air conditioner with the heat source produces a more efficient direct return air path to the system. 2008 NAAT, Inc. Page 29 of 49
  37. 37. Adoption of New Technology, Tools & Equipment Subsystem solutions System solutions Low-voltage processors and power throttling Semiconductor vendors are now producing low-voltage processors and processors with power throttling capabilities. These lower thermal designs are helping reduce the power draw and heat load on servers. The trade-off for customers comes in a higher price tag. Power management Vendors are also offering power management software that enables customers to cycle off their servers when not in use. System resources As server virtualization increases, so does development of, more richly configured systems - processors, memory, disk, and I/O. Rack solutions Room solutions Blanking panels Fit the empty spaces within a rack and improve air flow – plates are very inexpensive and are quick to install. Modular power and cooling infrastructure Rack-level UPSs, cabinet enclosures, and aisle enclosures offer additional power and cooling equipment as needed. Modular cooling equipment (versus large CRAC units) helps stop hot air from escaping at the source Network sensors Wired or wireless sensors can be placed on a rack to monitor and provide alerts for changes in temperature, humidity, brownouts /blackouts, HVAC loss, and so forth at the rack level. Hot/cold aisles IDC’s 2007 power and cooling trends study revealed that less then 20% of datacenters were using hot and cold aisle configurations, which underscores the lack of planning that is often seen in datacenter environments. Reducing cabling/cleaning out floor plenum It’s common to find many layers of cabling under the datacenter floor impeding air flow and cooling. While this can be risky to clean out, customers in mature datacenters are finding this is an effective solution for increasing the longevity of the site. Page 30 of 49
  38. 38. Data Center Containers Data centers in a box: Really? It could be the largest appliance available for the data center to date” Sandra Gittlen Computerworld Benefits Claimed - Lower upfront capital expenditure - Able to add capacity quickly, easily - Ease of mobility - Built in power and cooling efficiency - Built in redundancy - Security • The container-based model, which is still in its infancy, allows companies of all sizes to locate their data centers anywhere there is power, cooling and a network connection. • "These are the same rack systems and servers you would use in a data center. The only difference is that they've deployed in a shipping container instead of a building," says George Hamilton, director of enterprise infrastructure at Yankee Group Research Inc. • "This gives you the ability to add a big chunk of processing power and storage without having to figure out how to design a data center from the ground up," says Andreas Antonopoulos, a senior partner at consultancy Nemertes Research. • "It's best to think about these as a large version of a network appliance. You assume they've configured everything from the cable and cooling optimally so you can just plug it in," Antonopoulos says. See Appendix D for Vendor-Specific Offerings Page 31 of 49
  39. 39. (Verari Systems, Sun, IBM, HP, Microsoft, Google, Emerson) Page 32 of 49
  40. 40. Subterranean Facilities • Site selection for Mission Critical Real Estate (MCRE) is trending towards subterranean facilities that can greatly reduce risk and vulnerability to unpredictable elements Organizations turning to subterranean MCRE include: - In early ‘08, Wikia Search was launched via servers housed in the U.S. Secure Hosting Center (USSHC) ultra-secure underground hosting facility - Fall ‘07 Sun Microsystems announced a JV that will place computing facilities in a dormant Japanese coal mine - PHNS, which serves hospitals and healthcare customers, developed a subterranean data center 85 feet underground in a disaster-proof solid limestone cave near Springfield, MO - Cavern Technologies, a 200,000 square foot facility near Kansas City that is 125 feet underground. - United States Secure Hosting Center USSHC - StrataSpace, a 500 K s.f. underground data center outside Louisville, Kentucky. - The SpringNet Underground, 56 K s.f data center 85 feet underground in a limestone cave near Springfield, Missouri. - The Mountain Complex, built into side of a dolomite mountain in the Ozarks near Branson, MO. recently won a deal to house backup data for thousands of financial institutions. - The InfoBunker, a 65,000 square foot ultra-secure underground data center in Iowa, built in a decommissioned Air Force bunker. - The Montgomery Westland (The Westlin Bunker) provides 40 K s.f of underground data center and office space in Lake Conroe, Texas, recently announced plans to add 100K, s.f. - Iron Mountain has a data center within its huge underground records storage facility near Pittsburgh, previously known as the National Underground. - Mountains West Exploration Inc. (MXWI) entered the data center business in November with the acquisition of Secured Digital Storage, and plans to develop former military ammunition bunkers as ultra-secure storage. Key Motivators Power and Cooling • A key motivation for Sun’s Japanese Security • Reduce physical points of Page 33 of 49
  41. 41. underground site is that the site temperature will be a constant 59 degrees Fahrenheit • Appealing to orgs that require high number of cooling days. vulnerability. Extremely difficult to find the location, or the entry point. • Facilitate TS/SCI (Top Secret/Sensitive Compartmented Info). Green Standards • Offers a way to satisfy existing LEED priorities while also carving a path to meeting future green data center standards. Protection from Natural Disasters • Facilities increasingly built with blast- hardened limestone, making it weather proof, natural disaster proof and safeguarded from nearly any conventional weapon, Sources: • Data Center Site Selection: How Deep Can You Go? by Lee Kirby, VP & GM with Lee Technologies, June 2008 • Floating Data Centers 11 January 2008 Datacentres take to the high seas By Chris Mellor IDS stands for International Data Security and is run by CEO Ken Choi and president Richard Naughton, an ex-US Navy admiral. • US start-up IDS plans to open the first of 50 ship-borne floating datacentres at Pier 50 in San Francisco in April. • Floating data centres are said to be much more environmentally friendly than land-based datacentres. Other green datacentre locations have previously included Siberia, Iceland, Smartbunker, a UK NATO military site, and a Japanese coalmine. • The company aims to have 22 container ships housing datacentres around the USA coastline and 28 elsewhere around the globe. • The datacentres will be constructed in the ship's cargo space and also housed in shipping containers stacked on the deck, using products such as Sun's Blackbox and Page 34 of 49
  42. 42. A somewhat basic datasheet has come to light. It is not known if IDS is going to be (ahem) floated. Find out more from SilverBack Migration Solutions. Rackable's ICE Cube • It is said that each ship will have a minimum of 200,000 square feet of potential datacentre space. • The Pier 50 ship already has its 'anchor' tenants. The full fleet of 50 decommissioned cargo ships is said to have already been purchased. • Power demands will be supplemented by on-ship generators running on the ship's bio-diesel supply, allowing sustained power outages of up to one month. • Sea water will be used to cool the air-conditioning towers with a 30-40 percent power reduction expected. • Waste heat from the datacentres will be re-used to heat the ship's accommodation • These marine data centres are being targeted at the disaster recovery market initially • IDS says that a ship-borne datacentre can be commissioned in just a few months whereas building a land-based datacentre can take a year or more and be hindered by real estate constraints. Page 35 of 49
  43. 43. SECONDARY MARKETS Public and Private Entities are Flocking to Secondary Markets to Leverage Availability of Land, Access to Lower Cost Resources, Close Proximity to Power and an Array of Incentives from States and Utilities The Boyd Company Inc. Best Places to Build a Data Center by Estimated Annual Operating Cost Cheapest places to build Most expensive places to build City Cost Sioux Falls, S.D. $16,131,793 Ft. Walton/Destin, Fla. $16,386,387 Pensacola, Fla. $16,562,083 Jacksonville, Fla. $16,879,315 Lee County, Fla. $17,022,961 San Antonio, Texas $17,054,703 Ft. Wayne, Ind. $17,062,231 Orlando $17,076,394 Sarasota/Bradenton, Fla. $17,136,032 Birmingham, Ala. $17,185,474 City Cost New York City $22,542,097 San Francisco $21,889,593 Newark, N.J. $21,227,240 Boston $21,031,699 Stamford, Conn. $20,857,417 Chicago $20,433,269 Los Angeles $20,418,852 Hartford, Conn. $20,011,311 Philadelphia $19,953,246 San Diego $19,456,506 Page 36 of 49
  44. 44. Page 37 of 49
  45. 45. State Relocation Incentives Utah • Offered a $15 million-plus incentive to get business software giant Oracle Corp. to expand its presence in Utah. Iowa • Microsoft Corp. confirmed plans to build a data center in the Des Moines area, a move that comes about four months after the state approved a package of tax breaks designed to lure the company Ohio • Cincinnati Bell Technology Solutions, located in Lebanon, received a 40 percent credit for a five-year term to expand its current facility with a data center and IT managed services. Indiana • An Indiana company has announced plans to build an 80,000 square foot data center near Columbus, Indiana, with the state providing more than $300,000 in incentives to get the project off the ground. The data center will be built and operated by Data Cave Inc., a new company spun off from Analytical Engineering, Inc. (AEI) Alabama • BCBS plans to buy 25 acres in Jefferson Metropolitan Park Lakeshore, a deal that was authorized by the Jefferson County Economic and Industrial Development Authority board of directors. The company plans to build a 55,000-square-foot center in Birmingham's Oxmoor Valley, with plans to occupy the building in 2010. • Birmingham is one of the most affordable markets in the U.S> to operate a data center, according to a recent study by The Boyd Company, Nebraska • Yahoo's interest in Omaha was first noted by TechHermit, and has been confirmed by our sources, who say the Internet portal has looked at development opportunities in several Midwestern states. Nebraska, which has stepped up its efforts to attract data center projects in recent months, has emerged as the front runner. • Omaha was recently named one of the most affordable U.S. cities to operate a data center by The Boyd Company, with an annual operating expense of $12.9 million, less than half the cost of Page 38 of 49
  46. 46. operating a similar facility in New York. • Nebraska has also been developing incentive programs to compete with its neighbor Iowa, EDAs with Booths at AFCOM Data Center World • Virginia Economic Development Partnership: let companies know that there's still plenty of room for data centers in the state, especially in southern Virginia. • Nebraska Advantage: Omaha was named one of the best U.S. cities to locate a data center by The Boyd Company, and is developing incentive programs to compete with its neighbor Iowa. The Nebraska Advantage booth also had representatives of the Nebraska Public Power District on hand to discuss power issues. • Temple (Texas) Economic Development Corp.: Temple is a city between Dallas and Austin that hopes to attract data center tenants to a business park controlled by the EDA. • North Dakota Department of Commerce: Reps of North Dakota say they've had site visits from some of the largest data center providers, attracted by the state's cost and infrastructure. A state- funded telecommunications network, STAGEnet, provides broadband connections to 192 North Dakota communities. • The town of Cedar Falls, Iowa distributed packages of economic development information to media at Data Center World. The city's EDA has developed an FAQ page addressing the region's benefits for data center site location. Iceland Attracts Data Center Market • April 1, 2008 -- (WEB HOST INDUSTRY REVIEW) -- Iceland has recently been trying to put itself on the map as a competitive location for the operation of data centers. • The Invest in Iceland Agency ( recently cited a PricewaterhouseCoopers Belgium study that found that Iceland has become one of the most attractive options, if not the most attractive, for the growing offshore data center market Page 39 of 49
  47. 47. Federal Moves Federal agencies, including the FBI, are relocating operations to the I-81 corridor – Virginia Far enough from the capital to escape the fallout of a nuclear explosion -- a distance often estimated at 50 miles -- but close enough so employees can get to the District relatively easily. U.S. Government Security Exposures Page 40 of 49
  48. 48. In a January 2008 Report to the Acting Commissioner of Internal Revenue, the U.S. Government Accountability Office cited a variety of physical security issues at IRS data centers, mostly centering around access by personnel who didn't have a need to be there. In an earlier GAO report last summer about information security issues within the government as a whole (download PDF), the GAO Almost all of the 24 major federal agencies had weaknesses in one or more areas of information security controls. • IRS made limited progress toward correcting previously reported information security weaknesses. It has corrected or mitigated 29 of the 98 information security weaknesses that we reported as unresolved at the time of our last review. • IRS has not consistently implemented effective controls to prevent, limit, or detect unauthorized access to computing resources from within its internal network. For example, IRS did not always 1. Enforce strong password management for properly identifying and authenticating users 2. Authorize user access to permit only the access needed to perform job functions 3. Encrypt sensitive data 4. Effectively monitor changes on its mainframe 5. Physically protect its computer resources 1. Agriculture 2. Commerce 3. Defense 4. Education 5. Energy 6. Health and Human Services 7. Homeland Security 8. HUD 9. Interior 10. Justice 11. Labor 12. State 13. Transportat ion 14. Treasury 15. Veterans Affairs 16. EPA 17. GSA 18. NASA 19. NSF 20. NRC 21. Office of Personnel Mgnt 22. SBA 23. Social Security Admin 24. U.S. Agency for International Development Page 41 of 49
  49. 49. Colocation Providers Leveraging Secondary Markets Power Loft LLC Building a 200K SF center in Manassas, VA Terremark Building new centers in VA and Silicon Valley Equinix Acquired Northern VA campus for $53 million and bought one of its flagship Silicon Valley facilities in San Jose for $65 million. Construction continues on new IBX centers in Chicago and Secaucus, N.J. Peak 10: Entered Atlanta, Richmond and Cincinnati in ‘07- added capacity in Jacksonville, FL.; Nashville, Raleigh, and Charlotte - has facilities in Louisville, Ky. and Tampa, Fla. ViaWest Acquisition of Dataside added 7 centers in Dallas, Austin and Las Vegas, to seven facilities between Portland, OR Salt Lake City, and Denver Colospace 4 centers in MA, 2 in NH - Plans in CT, RI, ME, VT. Hosted Solutions 2 facilities apiece in Charlotte and Raleigh - Acquired Boston Datacenters in 2006 to enter New England SAVVIS Constructing four new data centers Carpathia Hosting June 25, 2008 -- Carpathia Hosting, announced addition of over 20,000 SF, Tier III data center in VA. Microsoft Company’s major center in Quincy, WA, attracted projects from Yahoo, Sabey Corp., Intuit, , Base Partners $550M SF data center project in the Westover Hills area of San Antonio, attracted projects by National Security Agency, Stream Realty Partners, Christus Health Systems, Power Loft LLC Page 42 of 49
  50. 50. Other providers with 1-3 facilities in secondary markets, include CDW Berbee, Team Companies, MarquisNet, FiberCloud et al Oklahoma is becoming home to companies including Google, EDS, Perimeter Technology and Titan Private Security and others Page 43 of 49
  51. 51. GROWING DEMAND FOR OUTSOURCING / MANAGED SERVICES Colocation / Outsourcing Trends Industry Views Demand Drivers • Over the next three or more years, midsize and large users will increasingly rely on hosting services, effecting one of the most important changes to the U.S. data center landscape, according to Gartner Research VP Rakesh Kumar who said companies should move quickly to avoid high prices. • “We are seeing a definite trend in colocation across multiple industries as a result of the increased need for availability of systems as well as the adoption of denser technologies” notes Blake McClean, VP of Business Development for CyrusOne. • Farah Saeed, Senior Consultant for the Energy and Power Group for the research consultant Frost & Sullivan agrees that they are seeing steady growth in the colocation sector that is likely attributable to several factors that include operational cost savings and personnel. • Scott Morrison, research director at Gartner, stated "The increase in demand for data center services is being driven primarily by internal enterprise requirements, including growth in content-rich applications, storage needs and virtualization." • Economic downturn and restricted access to capital also influencing companies' approach to data center space. Even financial companies that traditionally build their own facilities are considering leasing data center space instead. • Shrinking talent pool: - Per AFCOM, by 2015, the talent pool of qualified senior- level technical and management data center professionals will shrink by 45%. - Per Bill Clifford, CEO of Aperture and the Aperture Research Institute, the data center industry is and will continue to see a problem with filling the role of the data center manager. Interactive Map Page 44 of 49
  52. 52. A hosting company in Denmark has launched Data Center Map - a visual guide to colocation facilities in major markets around the world. Currently there are 848 co-location data centers from 45 countries in the index, of which 715 are carrier neutral and 133 are not. Furthermore there are 193 tenants listed under the various data centers. Click: Map / Index / Search Move to Managed Services Increasingly, providers are shifting from basic colocation to managed services, moving up the value chain to pursue higher profit margins. The move to Software as a Service (SaaS) and Hardware as a Service (HaaS) allows companies to consider buying more IT as an operating expense (OpEx) on a variable capacity basis rather than using capital expenditures (CapEx) to build capacity, notes Vinnie Mirchindani, founder of Deal Architect, Inc. Data Center Real Estate There is broad agreement among the industry's leading executives, who say demand for data center space and services remains strong, and is likely to continue - Digital Realty Trust (DLR), Equinix (EQIX), DuPont Fabros Technology (DFT), Switch and Data (SDXC), Terremark (TMRK), Savvis (SVVS), 365 Main and i/o Data Centers. Source 1. DuPont Fabros: A Bet on Internet Growth 2. Digital Realty Trust • Raised more than $640M with its October IPO • Bullish on the data center real estate space," and expect fundamentals to remain strong. "This is a real estate bet on internet growth,” Hossein Fateh, Pres. and CEO of (DFT), • Products are actually power, cooling and the security • Feb ‘08 - acquired 365 South Randolphville Road, in NJ • Dec ‘07 - acquired Units 1, 2 and 3 of the Foxboro Business Park, in suburban London, England • Dec ‘07 - acquired Naritaweg 52, located in Amsterdam • Dec ‘07 - acquired Cressex 1, in suburban London, Page 45 of 49
  53. 53. • Over 60% of leasing revenue from Microsoft and Yahoo, but customer base diversifying: social networking sites & online video and delivery of medical records Long-Term View In past 3 years, everything we’ve built has leased 100 %." 4 centers planned in next 2 years Could build up to $2 B of construction without new money Property Types Internet gateways Data centers - hubs for Internet and data communications Corporate data centers Technology manufacturing properties, which contain specialized manufacturing environments Regional or national offices of technology companies Equinix (EQIX) Terremark • Equinix (EQIX) appointed former co- chair and chief investment officer for Prologis, Irving F. Lyons, III, to its board • Company’s aggressive building plan is fully-funded • "We definitely heard from customers that in this economic environment that they are more prone to outsource their colocation needs as opposed to build from a CapEx perspective," said Margie Backaus, the chief business officer at Equinix (EQIX). • In June Terremark Worldwide (TMRK) arranged for $250M in financing to fund its expansion projects • Terremark CEO Manny Medina says that new business has shifted towards managed hosting services. Customers have come to the Infinistructure platform, to reduce their total cost of ownership and at the same time increase their level of services," said Medina. CRG West Holding of The Carlyle Group Savvis Inc. (SVVS) Raised $200 million through asset sales i/o Data Centers Backed by i/o Capital Cloud Computing Page 46 of 49
  54. 54. April 07, 2008 • Cloud computing encompasses any subscription-based or pay-per- use service that, in real time over the Internet, extends IT's existing capabilities. • Performing enormous scientific calculations in hundreds or thousands of idle machines within an enterprise or worldwide. See grid computing. See also utility computing. • Running apps in or from network servers. May refer to a company's own network - often refers to the Internet and use of Web browser-based or rich client applications. Implies trend toward thin client computing, embodied in the "network computer" of the late 90s (network computer). Based on discussions with vendors, analysts, and IT customers, following is a rough breakdown of what cloud computing is all about SaaS Delivers a single application through the browser to thousands of customers using a multitenant architecture Platform as a service A SaaS variation, this form of cloud computing delivers development environments as a service Web services in the cloud Closely related to SaaS, Web service providers offer APIs for developers to exploit Internet functionality, rather than delivering full-blown apps Service commerce platforms Hybrid of SaaS and MSP, this cloud computing service offers a service hub that users interact with. Think of it as an automated service bureau Utility computing Not a new idea, but this form of cloud computing is getting new life from, Sun, IBM, et al who now offer storage and virtual servers that IT can access on demand. MSP (managed service providers) One of the oldest forms, basically an application exposed to IT not end-users, like a virus scanning svce for e-mail or an application monitoring svce Internet integration The integration of cloud-based services is in its early days, but cloud computing aggregators & integrators are emerging Cloud computing might be better described as "sky computing," w many isolated clouds of services. Note As virtualization and SOA permeate, loosely coupled services running on agile, scalable infrastructure will make every enterprise a node in the cloud. Among big meta-trends, cloud computing is the hardest one to argue with. Page 47 of 49
  55. 55. As TechEd 2008, Microsoft Chairman Bill Gates described Microsoft's "mega data centers" as providing an advantage in the current cloud-building arms race, saying that "Microsoft and only a few others" will be able to build these facilities. "Today in our data centers we have literally hundreds of thousands of servers. In the future, we'll have many millions of those servers." iTricity Collaborates With IBM to Open a New Cloud Computing Hosting Center June 18, 2008 Based on IBM "Blue Cloud" technology - corporate data centers operate more like the Internet by computing across a distributed, globally accessible fabric of resources, rather than local machines. INDUSTRY-WIDE ENERGY MANAGEMENT PROGRAMS & INITIATIVES DOE EPA Enterprise Server and Data Center Energy Efficiency Initiatives The DOE and EPA are directed by law to guide how we design and operate data centers. Page 48 of 49
  56. 56. They have initiated a joint national data center energy efficiency information program, which coordinates a wide variety of activities from the DOE Industrial Technologies Program Save Energy Now initiative, the DOE Federal Energy Management Program (FEMP), and the EPA ENERGY STAR program. EPA Report to Congress on Server and Data Center Energy Efficiency (helps define a vision for achieving energy efficiency in U.S. data centers) Page 49 of 49
  57. 57. Data Centers in the Mix The Industrial Technologies Program (ITP) leads national efforts to improve industrial energy efficiency and environmental performance. We are part of the U.S. Department of Energy's Office of Energy Efficiency and Renewable Energy Save Energy Now is a national initiative of the Industrial Technologies Program (ITP) to drive a 25% reduction in industrial energy intensity in 10 years. Industrial companies can participate in no- cost energy assessments and utilize ITP resources to reduce energy use while increasing profits. The Industrial Technologies Program (ITP), through Save Energy Now, is working with U.S. computer data centers to reduce their energy consumption. This Web page DOE Save Energy Now in Your Data Center describes the planned activities and partnerships. Sign up now to participate in the DOE effort to reduce data center energy use! Major Data Center Program Elements Vision for Energy Efficiency in Data Centers • Develop and test “DC Pro” Software using pilot energy assessments • Create consensus metrics • Create and publicize Save Energy Now case studies based on pilot energy assessments • Create best practice information and a training curriculum • Develop Qualified Specialists program for Data Centers • Establish metrics for overall data center energy intensity - IT and infrastructure - Energy cost ($), source energy (Btu), and carbon emissions (M tons) - Specified Best-in-Class targets for various types of data centers • Create technologies, tools and guidelines to drive continuous improvement • Support third-party certification process to validate energy intensity improvement and Best- Page 50 of 49
  58. 58. • Create guidelines for “Best-in- Class” data centers and validate with Technology Demonstrations • Create and implement a collaborative research program with industry in-Class • Provide recognition for data centers that achieve a certain level of energy savings Page 51 of 49
  59. 59. Goals By the beginning of 2011: • 3,000 data centers will have completed awareness training through classes or webcasts via our partners • 1,500 mid-tier and enterprise- class data centers will have applied the Assessment Protocols and Tools to improve data center energy efficiency by 25% (on average) • 200 enterprise-class data centers will have improved their energy efficiency by 50% (on average) via such aggressive measures as accelerated virtualization, high-efficiency servers, high-efficiency power systems (e.g., fuel cells), optimized cooling, and combined heat and power systems • 200 Qualified Specialists will be certified to assist data centers Page 52 of 49
  60. 60. Source: EPA Report to Congress on Server and Data Center Energy Efficiency Page 53 of 49
  61. 61. EPA report shows that with Efficiency Improvements DC power footprint will go down, but this assumes that demand doesn’t change Page 54 of 49
  62. 62. Industry Take-Up of Green Initiatives • Europe has organizations working on IT and data center energy efficiency, including the UK’s Green Technology Initiative • Many multi national corporations are applying their corporate green strategies to their data center portfolios • While Allstate is seeking LEED certification for a new center scheduled to open in the spring of '09, but recognizes that LEED is not designed for data centers Page 55 of 49 Green Programs, Initiatives and Trends “To continue delivering uninterrupted uptime in enterprise computing, it is necessary to recognize that profitability, efficiency, and greenness are one and the same. The Uptime Institute The Uptime Institute’s final conclusion noted: • Greater attention to energy consumption and green initiatives is required. • The threat of running out of cooling and power capacity a growing issue • The C-suite is not adequately involved in or committed to the reduction of energy. Whereas, in a survey conducted late last year on Data Center Journal 30% indicated that they have no plans for a green initiative - another 33% indicated that they are planning and discussing it only.
  63. 63. • California Energy Commission’s Recommended Program for Data Center LEED Certification - expected to be ready sometime in 2008 The challenge has not been getting data centers to “go green.” Instead, what organizations lack are universally recognized industry standards for the green data center. In fact, a recent Digital Realty Trust survey finds that 82 percent of companies said that no clear industry standard for green data centers existed. 3 Barriers of Particular Importance to Data Centers Page 56 of 49
  64. 64. According to the EPA, barriers that prevent data centers from adopting changes are typically not technological but organizational. 1. Lack of efficiency definitions: Data center operators need standard definitions of productivity in order to purchase energy-efficient equipment, operate it in an optimal way, and design and operate the buildings to house it. (Recognized standards) 2. Split incentives: In many data centers, those responsible for purchasing and operating the IT equipment are not the same people that are responsible for the power and cooling infrastructure, who in turn typically pay the utility bills. 3. Risk aversion: data center operators are particularly averse to making changes that might increase the risk of down time. Page 57 of 49
  65. 65. National Data Center Energy Efficiency Information Program Metrics and benchmarking • DOE and EPA are working with industry to develop consistent measurement protocols and metrics to define energy performance in a data center facility Energy saving tools and training • DOE Save Energy Now and EPA ENERGY STAR are developing tools to assist data center operators in characterizing their energy use and identifying opportunities for improvement. Two tools form the cornerstone of these efforts: 1. The DOE Save Energy Now DC Pro tool suite 2. The EPA ENERGY STAR Portfolio Manager tool (will house EPA’s energy performance rating for data center infrastructure) Certification of data center energy efficiency experts • DOE Save Energy Now is developing a certification program for data center energy efficiency experts. Save Energy Now Qualified Specialists will be able to assist data center operators in identifying and implementing energy saving projects Equipment performance specification and labeling • EPA is developing an ENERGY STAR specification for enterprise servers which will allow this equipment to earn the ENERGY STAR. • EPA will consider future opportunities to develop ENERGY STAR specifications for additional IT equipment ENERGY STAR Enterprise Server Specification Development Process Recognition of Best- in-Class data centers • DOE Save Energy Now will recognize data centers which have demonstrated a defined level of energy savings using the DC Pro tool to validate energy saved. • EPA ENERGY STAR label given to data centers that achieve a high level of energy performance. Page 58 of 49
  66. 66. • DOE Save Energy Now and FEMP will develop Best-in-Class guidelines with industry partners to initially guide the specification and design of newly constructed Federal data centers Designation of a Data Center Energy Efficiency Organization • Section 453 of the Energy Independence and Security Act of 2007 requires DOE and EPA to designate an organization to coordinate the National Data Center Energy Efficiency Information Program activities • DOE and EPA are in the process of determining the role of this organization and selection criteria. DOE and will issue a Federal Register notice announcing the plan to select an organization Page 59 of 49
  67. 67. Data Collection Initiative to Develop an ENERGY STAR Rating for Data Centers To develop the rating system, EPA is collecting data on energy use and operating characteristics from a large number of existing data centers, both stand-alone facilities and data centers located in office and other types of buildings. Goals for the ENERGY STAR Data Center Rating What is the ENERGY STAR Rating for Data Center Infrastructure? • Build on existing ENERGY STAR methods and platforms. Methodology similar to existing ENERGY STAR ratings (1-100 scale). • Usable for both stand-alone data centers, as well as data centers housed within office or other buildings. • Assess performance at the building level to explain howa building performs, not why it performs a certain way. • Provide users with information and links to resources to aid in their efforts to determine next steps after receiving an energy performance rating for their building. • Offer the ENERGY STAR label to data centers with a rating of 75 or higher (performance in the top quartile). Unit of Analysis IT Energy/Total Energy What Measure of infrastructure efficiency - Captures impact of cooling and support systems - Does not capture IT efficiency Why Best available whole building measure - Industry still developing ways to understand and measure IT output and efficiency How Express ratio (IT/Total) as an ENERGY STAR 1-to-100 rating - Percentile of performance - Ratio value adjusted for: climate, tier level, other factors Rating Development Process Overview Timeline • Consult with various industry stakeholders to develop consensus on the use February Industry stakeholder meetings to discuss data collection needs March Initiate data collection with data center operators Page 60 of 49
  68. 68. Data Collection Initiative to Develop an ENERGY STAR Rating for Data Centers of a ratio of ITEnergy/Total Energy as the metric for the rating • Develop data collection template, instructions, and support materials. • Gather data on actual energy use from existing data centers. Data required from at least 100data centers. • Analyze data to develop rating models. • Launch ENERGY STAR Data Center Infrastructure Rating in Portfolio Manager. May ‘09 Conclude data collection Summer/ Fall 2009 Review and analyze data and develop energy performance rating system Fall 2009 Share analytical results with industry January 2010 Launch data center infrastructure energy performance rating in Portfolio Manager; ENERGY STAR label available for stand-alone data centers Over 215 data centers from more than 60 organizations have signed up to participate in the Data Collection Process Data Collection Initiative to Develop an ENERGY STAR Rating for Data Centers - 365 Main Inc. - 7x24 Exchange - ADP Inc. - Comp utershare - Data Center Operations Council (Corp Exec. Board) - Digital - EMC2 - Hill Environmental Operations - Hostin - JCPen ney Co, Inc. - Junipe r Networks, Inc. - Racks pace - Rarita n Inc. - The Green Grid - The Uptime Institute and Site Uptime Network - United Parcel Service - Unum - VISI Page 61 of 49
  69. 69. Data Collection Initiative to Develop an ENERGY STAR Rating for Data Centers Realty Trust, Inc. Some Measurement Tools Currently in Place Power Usage Effectiveness (PUE) and Data center Infrastructure Efficiency (DCiE) • First paper published in Feb, ‘07, "Green Grid Metrics: Describing Data Center Power Efficiency". • Initially, The Green Grid proposed the use of Power Usage Effectiveness (PUE) and its reciprocal, Data Center Efficiency (DCE) metrics, which enable data center operators to quickly estimate the energy efficiency of their data centers, compare the results against other data centers, and determine if any energy efficiency improvements need to be made. • The Green Grid has since refined definitions in a new paper is called Data Center Infrastructure Efficiency (DCiE) Note that different metrics are being proposed by other organizations as well Association of IT professionals seeking to dramatically raise the energy efficiency of data centers through a series of short-term and long-term proposals In his comments at the Silicon Valley Leadership Group's Data Center Energy Summit in June, Andrew Fanara of the EPA's Energy Star program addressed concerns that the government may eventually Page 62 of 49
  70. 70. regulate data center energy usage in some form. "I am not aware of any one planning any regulation of the data center industry. There may be climate legislation at some point, but that's a broader issue." The Data Center Energy Profiler “DC Pro” Software Tool, Release 1.1.1 Now Available Page 63 of 49
  71. 71. • DC Pro is an online software tool to help industries worldwide “diagnose” how energy is used in data centers and how to save energy and money. • DOE is collaborating with the U.S. EPA to ensure integration between DC Pro and EPA’s ENERGY STAR® Portfolio Manager tool. • DOE is also working with industry groups such as the Green Grid Association to develop the tool suite. • Read the workshop presentation materials to learn more Test the beta version and submit feedback Read the DC Pro FAQs Run the DC Pro Software 1 September 2008: DC Pro tool version 1.0 release Page 64 of 49
  72. 72. Page 65 of 49
  73. 73. Incentives Database of State Incentives for Renewables & Efficiency (DSIRE) • A comprehensive source of information on state, local, utility, and federal incentives that promote renewable energy and energy efficiency. It maintains databases broken down by state and individual utility, offering details on each program along with a link to the utility's web site. • The utilities that are developing efficiency programs for data centers have formed a committee to exchange ideas and coordinate their programs, • The Consortium for Energy Efficiency is beginning an initiative to address energy use in data centers – working closely with the EPA's ENERGY STAR program • PG&E became the first utility to offer rebates to business customers that implement virtualization and server consolidation projects. • Avista Utilities, serving Spokane, Wash., offers rebates up to $5,000 per rack for implementing a chip-level liquid cooling solution from SprayCool. • Xcel Energy, operations in eight Western states offers a custom project program in which business customers (including data centers) can get rebates of up to $200 per kilowatt of demand savings, as well as up to $15,000 towards an efficiency study to identify savings • For permanently reducing electric demand by at least 20 kilowatts in Con Edison territory, NYSERDA (New York State Energy Research and Development Authority) will help pay for capital improvement costs with an incentive of $600 a kilowatt • Austin Energy, is planning to introduce a data center program that will include rebates on a variety of efficiency measures, including server virtualization and efficient cooling practices Page 66 of 49
  74. 74. Convergence of IT and Facilities Now more than ever Facility and IT must work together to manage what has been a disconnect between the two. From capacity planning and right sizing your datacenter the need for integrated approaches to designing, building and operating facilities are driving new demands for Econometrics in design and planning. • In general, IT decisions are based on computing demands, while data facilities management makes decisions based on the cost and availability of energy. • CIOs know green technology is important, but they may not see it as urgent, because IT typically doesn't deal with the rising electricity bills at the facility. • Often, data center electricity costs are not broken out • Mechanical and electrical engineers with experience in data center design, air-flow modeling and power systems management are in demand. “If you have those skills, there are jobs waiting,” says Phil Calabrese, a mechanical engineer and director of I.B.M.’s real estate engineering and construction unit. Page 67 of 49 ITFacilities Organization
  75. 75. APPENDIX A Sources Industry Analysts Gartner • Oct. 2007 Trendwatch • “Global IT spending growth stable” By Chris Kanaracus, IDG News Service April 03, 2008 • Presentation to Oregon Association of Government Information Technology ManagementJanuary 2008 Conference • Gartner Data Center Conference November 27-30, 2007 Post Event Summary • Rakesh Kumar, VP Gartner Research IDC • IDC INSIGHT “Building, Planning and Operating the Next Generation Data Center” Michelle Bailey IDC OPINION • Vernon Turner, Analyst • Presentation “Analyze the Future” • IDC server virtualization multiclient study, 2007 • Outlook 2008: U.S. IT Spending by Region and Vertical IDC - 2/15/2008 • The Impact of Power and Cooling on Data Center Infrastructure Forrester • Forrester Cuts Tech Spending Prediction Reuters: 2/11/2008 • April 15, 2008 The Forrester Wave™: Data Center Automation, Q2 2008 by Evelyn Hubbert for IT Infrastructure & Operations Professionals Deal Architect Inc. • founded by Vinnie Mirchandani, former technology industry analyst (with Gartner), outsourcing exec (with PwC, now part of IBM) and founder of sourcing advisory firm, Jetstream Group Page 68 of 49
  76. 76. Yankee Group Research, Inc. • George Hamilton, director of enterprise infrastructure Frost & Sullivan • Farah Saeed, Senior Consultant, Energy and Power Group Input • John Slve, analyst The Boyd Company Inc. • Best Places to Build a Data Center by Estimated Annual Operating Cost Publications • 08 Feb 2007 | • Oct-Nov 2007 survey • “Gartner predicts data center power and cooling crisis” June 14 2007 • Uptime Institute expands data center tier rating system, By Mark Fontecchio, News Writer 19 Mar 2008 • November 2007 “Data center construction is booming” The Data Center Journal • Data Center Site Selection: How Deep Can You Go? Written by Lee Kirby, Wednesday, 25 June 2008 • Data Center Efficiency Experts In Demand Written by don dashefsky, Friday, 13 June 2008 • IT Projects Undeterred by Slowing Economy, Experts in Data Center Efficiency in Demand May 23, 2008 • 2007 was "The Year of the Data Center," • “2007 was "The Year of the Data Center" DataCenterDynamics • DatacenterDynamics, the key issues identified for 2008 Page 69 of 49
  77. 77. Information Age • Data centre building boom “Companies are building and expanding data centres despite economic gloom” Tuesday, 17th June 2008 Network World • “Security dominates 2008 IT agenda” January 30, 2008 • 02/19/2007 “New Data Center role could put you on the path to the executive suite” New York Times • June 17, 2008 “ Demand for Data Puts Engineers in Spotlight”, By Steve Lohr • Down on the server farm May 22nd 2008 | from The Economist print edition “The real-world implications of the rise of internet computing” TechWorld • 11 January 2008 “Datacentres take to the high seas” TechWeb Network Computerworld InfoWorld Governmental Entities EPA • EPA Report to Congress on Server and Data Center Energy Efficiency • ENERGY STAR Enterprise Server Specification Development Process • National Data Center Energy Efficiency Information Program • Data Collection Initiative to Lawrence Berkeley National Laboratory, • “High-Performance Data Centers: A Research Roadmap.” March 30, 2004 • “Data Center Energy Use, Metrics and Rating Systems” Steve Greenberg, Energy Management Engineer Environmental Energy Technologies Division October 31, 2007 GAO Page 70 of 49
  78. 78. Develop an ENERGY STAR Rating for Data Centers • Read the DC Pro FAQs • DOE Save Energy Now in Your Data Center • January 2008 Report to the Acting Commissioner of Internal Revenue • In an earlier GAO report last summer about information security issues within the government as a whole (download) Database of State Incentives for Renewables & Efficiency Independent Groups / Organizations Service Providers Page 71 of 49
  79. 79. National Association of State CIOs • Enterprise Data Center Consolidation In The States: Strategies & Business Justification August 2007 Data Center Institute • Five Bold Predictions For The Data Center Industry For The Data Center Industry That Will Change That Will Change Youryourfuture The Uptime Institute • Rte of computational increase (Moore’s Law) is greater than the rate of increase of power efficiency – 2007 • White Paper, “Tier Classifications Define Site Infrastructure Performance” • Symposium “Revolutionizing Data Center Efficiency” McKinsey & Company AFCOM ASHRAE The Green Grid • Power Usage Effectiveness (PUE) and Data center Infrastructure Efficiency (DCiE) ITIL® the IT Infrastructure Library ® Computer Economics • Survey: 2008 IT spending to be 'anemic' Jim Carr November 13, 2007 • Verari Systems • Sun Microsystems • Rackable Systems • IBM • Microsoft • Google • Emerson Network Power • Consortium for Energy Efficiency • Computer Room Services Corporation • Digital Realty Trust • EYP Mission Critical Systems • Fortress Int’l • APC • NAAT, Inc. • Sun Microsystems. Page 72 of 49
  80. 80. Consortium for Energy Efficiency • Government Insights Report: April 18, 2008 • Data Centers and Servers Initiative, June 2007 CDW IT Monitor • HP • Aperture Research InstituteTM (ARI) • Et al APPENDIX B Overall IT Spending Page 73 of 49
  81. 81. Analysts reports mixed, but many point to slowing U.S. IT spending in 2008, at least in some sectors Outlook 2008: U.S. IT Spending by Region and Vertical IDC - 2/15/2008 Computer Economics survey: 2008 IT spending to be 'anemic' Jim Carr November 13, 2007 • Per Monika Kumar, program manager of IDC's U.S. Vertical Markets research, "Going to market in 2008 will be challenging. • The nature of IT projects will change with pockets of opportunity in certain industries and geographic regions. • Financial services and retail will curtail overall IT spending, healthcare providers in states such as CA, TX, and FL will face a surge in demand to support their aging population • An uncertain economy is likely to negatively impact IT spending next year, especially within large enterprises. • The report, "2008 IT Spending Outlook: Anemic Growth," predicts that 16% of companies with annual revenues of $750 + million expect IT budget cuts • 33% of large organizations indicated that 2007 IT spending will be less than budgeted –per the report's author, Computer Economics President Frank Scavo. CDW survey says businesses, Feds slowing IT spending Forrester Cuts Tech Spending Prediction Reuters: 2/11/2008 • The overall CDW IT Monitor index dropped one point to 72, compared to February's reading of 73. • Based on an online survey of at least 1000 IT decision makers from businesses of all sizes and all sectors of government. Data breakdowns available at • Concerns over recession affecting tech spending in the U.S. • Forrester has cut its global tech spending forecast from 9 percent in 2008 to 6 percent, according to this Reuters article. • With the U.S. accounting for two- thirds of global IT spending, a growth of only 2.8 percent in 2008 is sure to have an effect. Government Insights Report: April 18, 2008 Gartner: Global IT spending growth stable By Chris Kanaracus, IDG News Service April 03, 2008 Page 74 of 49
  82. 82. • “We predict accelerated investment in government IT spending in 2008, but the five-year overall U.S. IT government spending picture will remain relatively flat,” said Teresa Bozzelli, chief operating officer and managing director of Government Insights. • Bozzelli estimated that worldwide government IT spending would reach $161 billion in 2008 - U.S. government IT spending would account for about $73 billion or 46% • Bozzelli estimated government IT spending in the U.S. and Europe was growing at roughly a 6% annual rate between 2008 and 2009. • Study finds IT spending has been largely unaffected, despite ongoing signs of weakness in U.S. & global economies. • On a global basis, the projected IT budget growth rate for 2008 is 3.3 percent, unchanged from a previous Gartner survey. • In the U.S., budgets are still growing, at 2.3 percent, but that represents a drop from 3.1 percent in the last study. • Responses from 1,011 CIOs between Feb. 12 and Mar. 12. • 62% said their 2008 IT budgets did not change, 23% reported a drop and 15% said their budgets grew. APPENDIX C Table 2-1. Typical IT Equipment and Site Infrastructure System Characteristics, by Space Type Space type Typical size Typical IT equipment characteristics Typical site infrastructure system characteristics Server closet <200 ft2 1-2 servers No external storage Typically conditioned through an office HVAC system. To support VOIP and wireless applications, UPS and DC power systems are sometimes included in server closets. Environmental conditions not as tightly maintained as other data center types. HVAC energy efficiency associated with server closets similar to efficiency of office HVAC systems Server room <500 ft2 A few to dozens of Typically conditioned through an office HVAC system, with additional cooling capacity, Page 75 of 49
  83. 83. servers No external storage probably in the form of a split system specifically designed to condition the room. The cooling system and UPS equipment are typically of average or low efficiency because there is no economy of scale to make efficient systems more first-cost competitive Localized data center <1,000 ft2 Dozens to hundreds of servers Moderate external storage Typically use under-floor or overhead air distribution systems and a few in-room CRAC units. CRAC units in localized data centers are more likely to be air cooled and have constant- speed fans and are thus relatively low efficiency. Operational staff is likely to be minimal, which makes it likely that equipment orientation and airflow management are not optimized. Air temperature and humidity are tightly monitored. However, power and cooling redundancy reduce overall system efficiency Mid-tier data center <5,000 ft2 Hundreds of servers Extensive external storage Typically use under-floor air distribution and in-room CRAC units. The larger size of the center increases the probability that efficient cooling, e.g., a central chilled water plant and central air handling units with variable speed fans, is used. Staff at this size data center may be aware of equipment orientation and airflow management best practices. However, power and cooling redundancy may reduce overall system efficiency Enterprise- class data center 5,000+ ft2 Hundreds to thousands of servers Extensive external storage The most efficient equipment is expected to be found in these large data centers. Along with efficient cooling, these data centers may have energy management systems. Equipment orientation and airflow management best practices are most likely implemented. However, enterprise-class data centers are designed with maximum redundancy, which can reduce the benefits gained from the operational and technological efficiency measures Source: EPA Report to Congress Derived from Bailey et al. (2007) Page 76 of 49
  84. 84. APPENDIX D Verari • March 4, 2008 - Verari Systems announces FOREST container solution (Flexible, Open, Reliable, Energy efficient, Scalable and Transportable) - High Density Storage and Compute Blades - Up to 11.5 Petabytes of Storage Capacity or Up to 1400 Servers - Self Contained Cooling or Chilled Water Cooling - Able to Exploit External Temperatures to Reduce Operating Costs - Local and Remote Management and Warning Interface • By combining ultra-efficient solutions at the blade level with the industry leading patented Vertical Cooling Technology the Verari FOREST Container can reduce total energy costs by improving data center efficiency by up to 110% • A market leader in blade storage and energy efficient platforms. Customers include Wachovia, Akamai, Microsoft, Qualcomm, ESPN, CGGVeritas, Harris, Lockheed Martin, Northrop Grumman, Sony Imageworks, and top universities and research institutions Sun Microsystems (JAVA) Project Blackbox (now the Sun MD S20) • Introduced in 2006 • First effort at a “data center in a box” incorporating a high-density computing environment into a 20-foot shipping container • The containers can travel on trains, ships or trucks • Some of the benefits that Sun touts: Page 77 of 49
  85. 85. - 10X faster to deploy vs. traditional datacenter - Reduced capital expenses with incremental expansion - 4X higher density per rack vs. typical datacenter - 40% lower cooling costs in 1/8th the space. Page 78 of 49
  86. 86. Rackable Systems' ICE Cube™ • Self-contained, fully portable, extreme server and storage density with highly efficient cooling and easy serviceability • This newest generation of the Integrated Concentro Environment (ICE) ™ features nearly 20% greater density, broader choice of container sizes and configurability, as well as more rapid production times • Designed to augment or replace traditional brick-and-mortar data centers of any size • Well suited for a broad range of deployment scenarios — from military applications to disaster recovery scenarios to basic data center expansion IBM's Scalable Modular Data Center • The Scalable Modular Data Center service product is available Page 79 of 49
  87. 87. from IBM's Site and Facilities Services business unit. Services include: - Data center strategy and design - Server and storage integration - Relocation planning - Project management - Infrastructure equipment sourcing - Installation services and management - Testing and start-up management • Entire solution can be implemented in 8 to 12 weeks and is 15% less expensive than traditional data center builds. • Uses APC InfraStruXure design, which integrates power, cooling, rack, management, services and security, allowing for selection of standardized components to create a solution through modular configurations. IBM's data center design and build experience and services expertise are utilized to bring the solution to clients Page 80 of 49
  88. 88. Microsoft’s Customized data center containers, called CBlox "Our mission is to reinvent infrastructure for the industry." Chrapaty • Microsoft is embracing containers as key to building scalable, energy-efficient cloud computing platforms • Company predicts growth of 20,000 servers per month in the next several years • New Chicago Data Center represents first container data center - $500 million, 500,000 SF facility in Northlake, Illinois - Able to pack up to 300,000 servers into the data center "container farm" on the first floor - Will fill entire first floor with up to 220 shipping containers packed with servers - Part of company’s fleet of next-generation data centers being built to support its Live suite of "software plus services" online applications - Will move from about 400 to 450 watts per square foot to 1,200 watts a square foot • Coverage of Microsoft (MSFT) and its data center projects and innovations: - Microsoft: 300,000 Servers in Container Farm - Microsoft Embraces Data Center Containers - 3D Look at Microsoft's Container Data Center (May 1) - Microsoft's 198 Megawatts of Motivation - A Look at Microsoft's Data Center Construction (May 23) New wind-powered container-based data lab in Boulder, Colorado Page 81 of 49
  89. 89. - Microsoft To Release Best Practices on Energy - Microsoft Unveils Wind-Powered Containers - Apr 18, 2008 - Red Dog: Microsoft's Developer Cloud Platform - Apr 09, 2008 Page 82 of 49
  90. 90. Google • October 2007 Google received a patent from the USPTO for the concept of a "mobile datacenter" stored in a standard shipping container and equipped with multiple racks of high-powered servers with its own internal cooling system • Originally filed in Dec. 2003, by Google employees William H. Whitted and Gerald Aigner • Sun’s Blackbox consists of a single mobile data center, Google's idea is for "at least one modular computing module" with "an interconnecting module to interconnect a plurality of the modular computing modules" • According to Robert Cringely, the idea wasn't new and wasn't even Google's, backing up his claim with a link to an Internet-Archive-in-a-Shipping- Container presentation (PDF, dated 11-8-2003) that was reportedly pitched to Larry Page • Patent covers a method for deployment that includes building one module within a ready-to-ship container, shipment via a transport infrastructure, several computing systems mounting within the container and transporting the container to a new site via the infrastructure Emerson Launches Container for Telcos • Unveiled its Containerized Computing Solution, a modular approach designed to help telecom companies deploy wireless and broadband services faster and cheaper. Page 83 of 49
  91. 91. • Positioned as a "central office in a box," a slight variation on the recent "data center in a box" container-based solutions introduced by major vendors such as Sun, Rackable and IBM. • Components include embedded computing and DC power conversion equipment, precision cooling units, power transfer switches, server racks and enclosures. • The containers feature Emerson's ATCA blade servers housed in a Knurr server cabinet, with power and cooling infrastructure from Liebert. • The new products are designed to help wireless companies expand their networks or quickly replace network infrastructure damaged in a disaster such as a tornado or hurricane. • The FCC recently began requiring telecom and wireless companies to provide backup power for cell sites and remote telecom facilities. The new measures were prompted by an FCC review of telecom outages in the aftermath of Hurricane Katrina. Video Page 84 of 49