The document discusses the economics of green high-performance computing (HPC). It separates marketing buzzwords from real actions organizations can take to improve sustainability. Sustainability has costs and returns, so initiatives should provide a return on investment. Key drivers of total cost of ownership for data centers are discussed, including capital expenditures, space/footprint costs, infrastructure costs, energy costs, and maintenance costs. Methods to improve sustainability through energy efficiency, space efficiency, and reliability are outlined. Comparisons of investments and annualized total costs of ownership are provided for data centers with different power usage effectiveness values.

1. The economics of green HPC
Giovanbattista Mattiussi
Marketing HPC, Eurotech

2. Green computing
• Green computing is a much
discussed subject nowadays
• It is good to separate the
buzzwords from the action we, as
organizations, can really afford to
undertake to realize long term
sustainability strategies
• In other words, which is the overall
incentive to top the Green500 list
other than marketing returns
• Sustainability has a cost, has a
return and hence should have a
ROI

3. Data center TCO drivers
Driver Cost components
IT CAPEX Initial SW and HW capital expenditures
Space occupancy
(footprint)
Cost of the occupied space and auxiliary infrastructure: rent,
opportunity cost, civil, structural and engineering, permits and
taxes
Data center infrastructure Electrical (UPS, generator, cables…)
Cooling (Chillers, AHUs, heat exchangers, pumps…)
Facilities (fire prevention, plants, security, building mng systems)
Installation Delivery, installation, test and tuning of IT, electrical and cooling
equipment
Energy Cost of energy: IT, cooling, lighting and waste
Maintenance and
additional operation
costs
Warranty extensions, support, software licenses, IT maintenance,
electrical and cooling maintenance, facilities maintenance, costs
of outages, heating, security
Other: disposal, green… Costs of end of life, carbon footprint (missed) incentives, fines…

4. Main Areas of impact on TCO
Links to sustainability
Energy savings
Lower cost due to
less energy
consumed
Space savings
Savings in real
estate, racks,
electrical, cooling
and network
Reliability
Savings in downtime
indirect cost and
maintenance
Sustainability impact
High
Sustainability impact
High
Sustainability impact
Medium

5. Energy efficiency - methods
IT equipment
Maximize Flops / Watt
Data Center
Facility PUE
Data Center or
ecosystem
Reuse thermal energy
1 IT equipment
Maximize efficiency
2
3
Increased work per watt
Eliminate fans
Component level heat
exchange
Newest processors are
more efficient
Liquid cooling
Energy aware design
Optimize air cooling
Free cooling
Liquid cooling
Direct liquid cooling
Optimization of power
conversion
Direct liquid cooling
Maximize outlet
temperature
Holistic view of data
center planning

6. PUEs in various data centers
Source: Intel
Global bank’s best data center (of more than 100) 2.25 Air
EPA Energy Star Average 1.91 Air/Liquid
Intel average >1.80 Air
ORNL 1.25 Liquid
Google 1.16 Liquid coils,
evaporative tower,
hot aisle
containment
Leibniz Supercomputing Centre (LRZ) 1.15 Direct liquid
National Center for Atmospheric Research (NCAR) 1.10 Liquid
Yahoo Lockport *(PUE declared in project) 1.08 Free air cooling +
evaporative cooling
Facebook Prineville 1.07 Free cooling,
evaporative
National Renewable Energy Laboratory (NREL) 1.06 Direct Liquid +
evaporative tower

7. Accelerators
First conclusion
• Any reduction of PUE (facility side) costs efforts (especially economic
efforts!)
• We know how to do point 2 getting successfully to PUEs of 1.x value.
Let’s focus on the «1» part of the PUE
• One way to do so this through Energy Aware design, a technique that
Eurotech has used for many years to date
Optimized design
No unused components
No fans
Soldered components
Dense architecture (with
integrated interconnect)
Optimized power
conversion chain
Direct liquid cooling

8. Less civil, structural
and engineering
costs
Less hardware
(racks)
Less maintenance
(fewer components)
Less cooling
(ventilation)
Less cabling
Density - methods
MORE
Flops/m2
Direct liquid cooling
1
High density design
Small form factors
Soldered memory
Compact blades
High performance
processors
2
3
Accelerators

9. Reliability impacts TCO in 2 ways:
- Direct costs, associated with spare parts, extended warranties, support
personnel
- Indirect costs, related to the business cost associated to an outage.
The direct costs depend on the number of components and their estimated
FIT (failure in time) rate, as demonstrated in the MTBF equation where 𝜇
is the failure rate of the single component and N the number of components
𝜇 is related to quality,
operating conditions,
monitoring and preventive
maintenance of components
MTBF =
1
𝜇 𝑖
𝑖=𝑁
𝑖=1
The indirect impact depends on type of organisation and could range from
thousands to millions € per hour of outage.
So the impact of low reliability on the business could offset any saving reached
during purchase and installation of IT solutions!
Reliability – resilience - methods

10. Impact of downtime

11. Comparison - investment
Investment (K US dollars) Datacenter A Datacenter B Datacenter C
Servers $6,200 $6,200 $6,200
Network and other IT $440 $440 $440
Building $1,260 $540 $360
Racks $280 $120 $60
Cooling $2,670 $3,060 $1660
Electrical $3,570 $3,570 $2,420
TOTAL INVESTMENT $14,420 $13,930 $11,140
Data center A – PUE 2.2
Data center B – PUE 1.6
Data center C – PUE 1.05
Medium density (20 kW per rack) – air cooled
High density (50 kW per rack) – optimized air
cooling, rear door liquid cooling
High density (87 kW per rack) – direct hot liquid
cooling, floating Tamb

12. Comparison – annualized TCO
Annual cost (K US dollars) Datacenter A Datacenter B Datacenter C
Cost of energy $1,970 $1,430 $640
Retuning and additional CFD $6 $3 $0
Total outage cost $270 $270 $230
Preventive maintenance $150 $150 $150
Annual facility and infrastructure
maintenance. $310 $290 $140
Lighting $5 $3 $2
Annualized 3 years capital costs $2,040 $2,000 $1,980
Annualized 10 years capital costs $880 $940 $540
Annualized 15 years capital costs $130 $60 $40
ANNUALIZED TCO $5,761 $5,146 $3,722
Data center A – PUE 2.2
Data center B – PUE 1.6
Data center C – PUE 1.05
Medium density (20 kW per rack) – air cooled
High density (50 kW per rack) – optimized air cooling, rear door
liquid cooling
High density (87 kW per rack) – direct hot liquid cooling, floating
Tamb

13. GREEN FIELD
Building for energy efficiency (i.e.
choice of location, free cooling…)
to utilize free cooling
TARGET PUE 1.2
EXISTING
Introducing an hot water cooling
installation in existing data center
TARGET PUE 1.4
EXTENSION WITH HIGH DENSITY
Separate room for hot water
cooling and installation which
floats with Tamb
TARGET PUE 1.05
1 Pflop/s installation in Mediterranean climate: hot summers, mild to
cold winters
Investment in data center
+ 4 M$ over a standard building
(PUE 2.2)
Energy savings: 1M $ per year
Payback 4.62 years
Investment in data center
+ 1M $ over standard equipment
(conservative, it can cost less)
Payback time: 1 year
+ 850K$ $ over standard
equipment
Payback time: 1.23 years

14. Green effects
1 Petaflop/s installation – CO2 savings
with water cooling compared to air
˜28000 tons of CO2 saved in 5 years.
Equivalent to:
3800 cars that do not circulate for 1 year
30100 saved adult trees
40 Km2 of rain forest left untouched