Your Partners & Co -Hosts:
NER Data Products
.
Greg A. Stover
Outline of Vendor Presentation
Senior Vice President
Cooling Infrastructure Optimization
The who, what, why and how?
AFCOM So. Cal. 5-28-09

Issues driving change
• Information growth continues to explode! 2006 = 160 Exabytes in existence, 400
in 2008, 1 Zetabyte by 2010 (IDC) That’s 57% per year
• Density
– Smaller, Faster, Hotter
– Virtualization – Big help, with side effects
• Power Consumption has doubled in the last 5 years, it will double again by 2011, power
density increasing 15-20% per year
• Watts in = Heat out! Hot Spots and Over Provisioning/Overcooling are on the rise
- Uptime continues to report the average Data Center is 2.6 times overcooled
- Yet 10% still having hot spots
• Greening Initiatives – Social vs Economic

Issues driving change
THE STANDARDS ARE COMING, THE STANDARDS ARE COMING!
(Green Grid, Uptime Institute, European Union, Google & Microsoft, DC Pulse, Vendors)
• PUE = Power Usage Effectiveness = Total Facility Power / IT Equipment Power
Of the total power being delivered, how much is being used by IT gear?
• DCiE = Data Center infrastructure Efficiency = IT equipment power / Total Facility Power x 100%
A DCiE value of 33% (equivalent to a PUE of 3.0) suggests that the
IT equipment consumes 33% of the power in the data center
• CADE = Corporate Average Data Center Efficiency = Facility Efficiency (FE) x Asset Efficiency (AE)
How well does your IT & Facilities investment convert Energy to useful work?
• CUE = Cooling Usage Effectiveness =
IT Load x 3.413=BTU’s/12,000= Tons of Cooling required/Tons Cooling capacity = Overcooling
How overcooled/over-provisioned are you?
Other newly released DCeP (TGG) & PPE (Gartner)

Issues driving change
Accountability is on the Horizon
- Availability has been all that's mattered
- Most of you haven’t paid the electric bill (facilities does)
- Data Centers have been like the CIA (budget?)

Issues driving change
• Government stepping In
- EPA report to Congress identified Data Centers/IT as a significant problem
- Data Center carbon footprint to exceed airlines in 2010
- expect voluntary Energy Star guidelines with reduction goals for now
- expect efficiency measurement metrics, benchmarking will be important
- Obama’s “Cap & Spend” – Accountability is imminent! inefficiency will cost!
- Hot off the Press – Energystar adopts PUE metric!

So now what? Moving forward
• Forget about the Jones’ – Who cares what others are doing?
• Make friends with Facilities ☺
• Pick a metric & Get Started
• Utilize best practices for measuring – Ashrae TC 9.9 & Green Grid
• Measure – Establish Baselines & benchmarking – Internal or
Outsource?
• Select tools to automate and standardize consistent measurement
• Prioritize – The options are many – What offers the greatest return?

Prioritization (your choices are many)
What are your options? What can you do? EPA report will share Top 50
• Implement Virtualization
• Deploy high efficiency Servers
• Deploy high efficiency Storage –
• Enable Power management features
• Deploy high efficiency PDU’s ( higher voltage AC or Moderate voltage DC)
• Right size Power Distribution
• Deploy high efficiency UPS Technology
• Deploy high efficiency chiller pumps
• Deploy high efficiency CRAC units
• Deploy Variable Speed Drives
• Hot and/or Cold Aisle Containment
• Use airside/waterside economizers – Where climate permits
• Direct liquid cooling (Water or other dialectic liquid)
• In Row Cooling
• Containers
• Build a new Data Center
Many are Disruptive, Labor intensive, Expensive!
Pick carefully

Where is the greatest potential return?
Lot’s of ways to say the same thing!
APC Version
Chatsworth
EYP Version

Where is the greatest potential return?

Cooling Realities – It’s often not a supply problem
Floor Tile Gymnastics & CFD Modeling
Mixing of hot exhaust with cold supply
air is biggest cause of inefficiency &
contributes to hot spots
On average, only 40% of available
cooling is utilized due to bypass and
over-provisioning to eliminate hot
spots
Typically, air is supplied well below
ASHRAE standards in attempt to
control temperature (77/80.6)
Despite this gross over-supply (2.5x),
10% of racks still experience hot
spots
Getting supply air where it needs to
be when it needs to be there combine
with eliminating Hot air is the key Cooling capacity and rack heat
density is often not the problem…
Slide courtesy of

EPA Data Center Report Said:
30% improvement in
infrastructure energy
efficiency from improved
airflow management
Airflow is the single
infrastructure improvement
that can be done without
disruption
Page 9 of EPA Final Report

CFD Modeling
Analyzing and understanding airflow & change
• Understand Heat, Airflow & Pressure behavior
• Foundation/data points for Baseline & Benchmarking
• root cause identification – constraints abound
• modeling the effects of change (Optimization Scenarios)
• failure analysis & redundancy
• Future layout & capacity planning
• Buy vs As a Service

The Modeling Process
STEP 1 - On-Site Audit
• Rack Location/Room Layout
• Rack by Rack IT equipment densities
• CRAC type and location
• Under-floor obstructions
• Total IT equipment power consumption
• Point temperatures throughout the room
• Number and location of perf. tiles
• Cutouts/Leakage
• Room dimensions
• Cable tray layouts
• Above-ceiling conditions
• External Factors (example: solar gain)
• Chilled water flow rate and temperature
• Electric rate (kW/hr)

The modeling Process
STEP 2 – CFD Modeling
• We construct an accurate
baseline model of your data
center (corresponding to the
information gathered during our
audit) using Computational Fluid
Dynamics software.
• The model predicts airflow and
temperature changes above and
below the floor during normal
operation and under failure
modes.
• We run multiple scenarios and
many, many iterations of the
model to develop a complete and
customized thermal solution for
your data center.

Modeling Options
STEP 3 – Project Review Recommended Solution
Option 1
Option 2
Option 3
Solution:
• Install networked under-floor fan tiles
in key locations
• Install under-floor air velocity
reducers
• Shut off 10-Ton CRAC unit
• Redistribute perforated tiles

The Solution
STEP 4 – Install the Solution
Install 2-foot-high containment panels
Install AdaptivCool™ CRAC at the back of these cabinets
controller
Strategically install
Install under floor air Install blanking panels in
AdaptivCool™ sensors,
velocity reducers critical areas
under floor air movers,
and return ducts

Thermal: Example
Results in this Data Center:
• Real-Time Dynamic Airflow and Temperature Control
• 24 x 7 Thermal Monitoring and Alarming
• Annual Energy Savings of $55,000
• Trending and Reporting of Thermal Conditions in the Data Center
• Active user interface
• Rack Intake Temperatures do NOT Exceed 77° F
Before
After

Rack Top Fans
Baseline Rack top @ 800 CFM
Modeling shows that significantly increasing rack top fan CFM does not affect return airflow
significantly Suggest to turn off rack top fans that are blowing cool air but leave rest on
Fail 2 CRACS Fail 2 CRACS
X X
X X

Baseline CFD Model
Normal Operation, 350 tons of cooling delivered
Cascading hot air
exhaust for Row K Air handlers AS4 thru
supplies Row M AS7 are on as well as
both overhead AS8 and
AS10 for total of 350-
tons of cooling
Data center room
temperature cold – Positioning of
result of over-cooling new 7.2 kW
by 3.6 times virtualized
blade cabinet Cascading hot air
exhaust for Row K
supplies Row M

Sample CFD Progression Results
Baseline Minimal Rack Moves Supply and return Full Re-layout
Supply optimization Optimization Supply
only optimization only

Baseline CFD Model
Normal Operation, All CRAC units running
Cooling vents
positioned
over what
could be
considered a
hot aisle
Hot air
exhaust
wraps around
cabinets into
cold aisle
Positioning of
Cascading hot
racks blocks
air exhaust for
hot air return
Row 3 supplies
path to
Row 2
CRAC units

Optimal Layout CFD Model
102kW load spread out between 121 cabinets
cooling is provided by CRACs 2 and 6
CRAC-6
CRAC-2
Hot air have
clear path
back to
Blanking panels are
CRACs
installed in any open
spaces in the cabinets
Cold aisles
maintain a 14
foot pitch The Production area
is untouched and
CRACS are left on

Optimal Layout with Additional
Load for a Total of 475kW
All 4 CRAC
units running for
total of 120 tons
475kW load requires
137 tons of cooling so
2 more CRAC units
are required

CFD Modeling
Buy vs As a Service
Purchase/own challenges
• Steep Learning Curve – training, how many, who?
• Analysis and interpretative experience – A little bit of a black art!
• Time consuming
• Total Cost
CFD As a Service - Potential Challenges
• Quality, experience, expertise of vendor – Get work samples
• You get what you pay for – Simulated data vs real data
• Data manipulation to tell own story – does it limit other options?
• Frequency
• Lack of understanding of your shop and goals

Cooling Infrastructure Optimization
We know the problems – What do you do?
• Again - Your Options are many
• We know what spins the electric Meter? We will focus on Cooling!
• Over provisioning & Hot Spots – You probably have one or the other
• Cooling Infrastructure Optimization (Options, Constraints & Realities)
– Basic Best Practices
– Containment Solutions (Curtains, Chimney/Ducted Cabinets (Passive
& Mechanical)
– Supplemental Air – Close Coupled Cooling
– Liquid Cooling Options
– Economizers (airside & waterside)
– Intelligent/Adaptive Air Movement/Mgt. (Room Scale)
They all work! They are all better than doing nothing!
The question is, which is right for you?

Basic Best Practices – Cooling efficiency
Minimize Air Mixing! “Bypass Air”
Separating Hot & Cold air is #1 issue/opportunity
• Hot & Cold Aisles – Rack orientation to each other & CRAC’s
• Fill the wholes – All of them (w/Plenum rated material)
- If it’s not delivering cold air directly to Server inlets – fill it!
(room envelope, fill around pipes, cable cutouts, use blanking panels)
- Remove Comfort tiles – Hot Aisles are suppose to be hot
• Eliminate/Remove under-floor delivery obstructions (old cables)
• Right size power cords – cable mgt.
• Remove out of rack equipment (put it in a rack or get it out of the rack rows)
• Raise Set points & Adjust humidity settings (Might want to model 1st ?)

Curtains/row containment – Hot or Cold
Considerations
• Aesthetics
• Fire Codes – no stds.
• Fire/smoke detection
• Fire Suppression costs
• Insurance Issues
• Pressurization
• Fusible link
maintenance
• Hot Aisle & Humans

Separate Supply Air from Exhaust Air
Cold Aisle Isolation Hot Aisle Isolation
105 ºF 75 ºF
75 ºF 105 ºF

Cabinet/Chimney/Rack Exhaust Ducting
Room air available for any rack
Rack Heat Containment
Heat maintained in individual rack
105 ºF
Placement not restricted to aisle
Exhaust duct increases pressure
drop – additional fan required 75 ºF
Similar restrictions to fire
protection and lighting
Allows raising supply air without
concern for working environment
Reference: ASHRAE Journal Article – Designing
Better Data Centers, December 2007

Chimney/Ducted Cabinets – Heat Containment
Passive, Mechanical, thermal Managed, pressure managed
Considerations
• Ceiling Height
• Plenum Design
• Fire Codes
• Disruption threshold
• Pressurization 105 ºF
75 ºF

Supplemental Air – Close Coupled
Considerations
• In the Row
- Infrastructure
requirements
- Leak detection
- Cost
• Above the Rack
- Infrastructure additions
- Leak detection
- Cost
Rumor has it that manufacturers of these
solutions now recommend containment as well

Liquid Cooling Enclosures
Considerations
• Infrastructure requirements
• Leak detection
• load
• Cost - benefit

Intelligent/Adaptive Air Movement/Mgt.
(Room Scale)
VFD’s & VSD’s Systems
Considerations
• Zone Cooling – Macro
• Not all units are retro-fitable
Intelligent Floor tiles and Return System
Considerations
• Adaptive Cooling – Micro
• Optimal on raised floors only

Considerations Summary
• Do I have the ability to bring chilled water, or additional chilled water
into my data center? (for liquid cooled or supplemental)
• Fire Code and Insurance considerations for curtain and containment
solutions
• Flexibility for room layout considerations with Chimney and close-
coupled systems
• Ability to retrofit older CRAC units and condenser units with VFD and
VSD systems
• Economic/budget considerations when weighing several of these
methods

You can see - The choices are many
What are the Constraints
• Corporate/Business Objectives
• Budget – Where'd all the money go
• Greenfield vs Legacy
• Room/Facility Design
• Local Codes
• Current IT Load – Future IT Load
• Current Infrastructure capacity vs required
• Remember – You can’t Manage what you can’t Measure!

Monitoring - Automation
Monitoring Tools
• IT Load – (Outlet, Strip, Cord, Cabinet, PDU)
• Total Facility Load
• Both (tie them together)
Planning & Management Tools
Saving for another day/presentation

Moving forward – Next Steps
• Forget about the Jones’ – Who cares what others are doing?
• Make friends with Facilities ☺
• Pick a metric & Get Started
• Measure – Establish Baselines & benchmarking – Internal or Outsource?
• Utilize best practices for measuring Data Points – ASHRAE TC 9.9 & Green Grid
• Assess Infrastructure – Model for Optimization – Pro-active!
• Select tools to automate and standardize consistent measurement
• Establish objective – Prioritize - The options are many
• Find the right fit! What offers the greatest return? Select a Technology

Infrastructure Optimization = Efficiency Gains
What Can you do with Efficiency Gains
(from raised set points, CRAC/CRAH optimization and improved Chiller
efficiency)
• Put the savings in your pocket
• Increase your capacity/Manageable load
• Increase redundancy
• Combinations of all of the Above
• Do the Cost benefit Analysis

Things to remember
One size doesn’t fit all

Things to remember
Get properly fitted – Assess Infrastructure – Have the Data

Things to remember
Sometimes it just doesn’t fit

Things to remember
More/bigger isn’t always better
Efficiency & Optimization are key!

NER your Trusted Advisor
Resources, Products, Solutions, Services
Questions?
Thank You!!!
Learn more @
www.nerdata.com