When it comes to high-power density data centers, all are not created equal. As organizations increasingly focus on heavy-duty, transactional workloads like big data analytics, they’re seeking space that can support upwards of 17kW per rack. Delivering these super high-power densities while ensuring tolerable working conditions requires careful planning and significant attention to air flow management, temperature control and electricity.

1. 1
Critical Design
Elements for
High-Power Density
Data Centers

2. 2
Critical Design Elements for High-Power
Density Data Centers
When it comes to high-power density data centers, all are not
created equal.
As organizations increasingly focus on heavy-duty,
transactional workloads like big data analytics, they’re seeking
space that can support upwards of 17kW per rack.
Delivering these super high-power densities while ensuring
tolerable working conditions requires careful planning and
significant attention to air flow management, temperature
control and electricity.
#DataCenter

3. 3
Artificial intelligence will exceed
human intellectual capacity and
control, thus radically changing
civilization in an event called
the singularity.
50th Anniversary of Moore’s Law
#DataCenter

4. 4
• In 1984, the number of Internet devices
was 1000
• In 2015, the number of Internet devices
exceed 30 Billion
• By 2020, it will exceed 50 Billion
Did you know?
#DataCenter

5. 5
There are two main issues regarding faster processors and those are:
1. Transmission delays on the chip
2. Heat build-up on the chip
Transmission delays As the size of the wires and transistors have gotten smaller over
the years, the time required to change states has gotten smaller, too. But there is some
limit -- charging and draining the wires takes time. That limit imposes a speed limit on
the chip.
HEAT - Every time the transistors in a gate change state, they leak a
little electricity. The faster a chip goes, the more heat it generates.
What is the issue with
Faster Processing? HEAT!

6. 6
High power density
computing is here if you
can design for it
Smaller and More Servers =
Lots of Heat & Power!
#DataCenter

7. 7
Courtesy of Schneider Electric
Most data centers
average to be low
density with some high
density cabinets only.
Most efficient density range
5-8kW
Enterprise kW / Rack High Density Study

8. 8
• A decade ago, standard densities were 3-5 kW per
rack
• Today, it is not uncommon to see 8-12 kW per rack
with some deployments reaching 15-20 kW per rack
• For this presentation, 10-20 kW is considered as high
density in a colocation facility
Colocation Data Center Power
Density Trends
#DataCenter

9. 9
High Density Benefits
Lowered Total Cost of Ownership (TCO)
• Emerson Network Power Study [2] – New Data Center
• Compare: 5 kw / rack (400 racks) vs. 20 kW / rack (100 racks) – Low Density vs.
High Density
• Floor Space: 10, 000 SF vs. 2,500 SF
• TCO difference after 5 years is $ 3, 660, 000 less for high density scenario
assuming similar maintenance cost
Categories Description
400 Racks
@5 kW/rack
100 Racks
@20 kW/rack
Capital Costs Building shell, rack, PDU 3,500,000 875,000
Cooling Capital Costs CRAC, high density modules, installation 830,000 1,900,000
Annual Cooling Operating
Costs Cooling energy costs 946,080 525,600
Maintenance No Data No Data No Data
Estimated TCO ( End of 5th Year) 9,060,400 5,403,000
Reduced
Space @
$150-200/SF
Reduced # of
PDUs @
$80-100K/
PDU
#DataCenter

10. 10
High Density Benefits
Increased Scalability
• Example: Customer starts up with 6 kW / rack, and after a few years has need
for 18 kW for their equipment
Current
Future
High Density Capable

11. 11
High Density Challenges
• More cooling required in smaller footprint
• Higher cooling equipment initial costs
• Stronger weight bearing structures required for heavier racks
• Response time in the event of a cooling failure
• Example: If the chiller pump failure or CRAH fan fails, data center operator will have
less than 1 minute for 20 kW / cab regions before temperatures exceed ASHRAE
recommended temperatures
• Solution: Feed CRAH fans and chiller pumps from UPS
• Power distribution
#DataCenter

12. 12
Not all racks in colocation data center are high density: equipment from the
past and present creates a mixed environment
• Can have 2 kW per rack and 20 kW per cabinet in same data center space
• These servers have different operating requirements (e.g. rack inlet temperatures and air flow rates)
• From a cost / watt perspective, average densities of 5-8 kW per rack is optimal [3]
Stranded Capacity (Space, Power & Cooling)
• Available floor space and racks but no remaining power or cooling
• Air handlers have remaining cooling capacity but not enough air flow rates to match IT equipment
requirement
• PDU has remaining electrical capacity but RPPs out of breaker positions, caused by low density racks
Resource Management (Reducing Stranded Capacity)
• Quantity and location of air handlers driven based on cabinet densities in each region
• Proper planning and book-keeping for power distribution to avoid overloading PDUS and to balance
phases
• Build out in modular pods to minimize stranded resources
Mixed Environment Challenges

13. 13
Cooling Strategies for Mixed Environments
• Blanking Panels & Rack Skirt
• Data Center Zoning
• Vertical Exhaust Ducts (VED)
• Cold or Hot Aisle Containment
• In Row Coolers (IRC)
• Other Strategies
#DataCenter

14. 14
Blanking Panels & Rack Skirts
Recommended for any kW / rack for proper
air flow management
12 kW / rack with and without blanking panels & rack skirts

15. 15
Data Center Zoning (Underfloor)
Similarly, owner can build out high density cabinets in contained pods (hot and/or cold aisle)

16. 16
High Density Cooling Solutions
• No Raised Floor –
(although it could be used
for routing chilled water
pipes)
• Row Cooling (Close-
coupled)
• Hot Aisle Containment
• 25 kW+
• Scalable (add cooling units
with increased rack
densities)
The challenge with any high density application is how to provide
that much volume of air (both supplying cold air and rejecting hot
air), typically 120-160 CFM/kW

17. 17
High Density Cooling Solutions
• Raised Floor Supply
• Chimney Racks
• Ceiling Plenum Return
• Cooling units can be located
outside
• Up to 20-25 kW
• Raised Floor Supply
• Fully Contained Hot Aisle
• Ceiling Plenum Return
• 25 kW+
• Needs extra space (to increase volume)

18. 18
High Density Cooling Solutions
• Hybrid Cooling
• Raised Floor Supply
• Row Cooling
• Hot Aisle Containment
• Ceiling Plenum Return
• 25 kW+
#DataCenter

19. 19
High Density Cooling Solutions
• Rear Door Heat
Exchangers
• Overhead Cooling Units
• Hybrid – combination of
different cooling
architectures
• Pipe routing – raised floor,
ceiling plenum
• 20kW+
#DataCenter

20. 20
High Density Cooling Solutions
• Direct Server Liquid Cooling
• Water can absorb about 4000
times more heat than air for
the same volume
• Supercomputers (100 kW+)
• Immersion Cooling (Dielectric
Fluid)

21. 21
• Minimize underfloor obstruction by planning route of conduit and
piping to limit the maximum obstruction in any given area
• Strict hot aisle-cold aisle configuration with containment
when necessary
• Balance tile flow rates to match required rack flow rates:
manually or use tile with automatic dampers
• Overhead coolers: similar to in row coolers but do not occupy
floor space
• Direct Liquid Cooling – provide water based cooling directly to
chip level
Other Strategies:
#DataCenter

22. 22
The Electrical Issue of High Power Density
• Proper management of power
cables and electrical
distribution system
• Management of air flow due
to # of conduits if placed
under floor
• Flexible distribution design
plan is needed
• Number of remote power
panels and their location
• Cost
#DataCenter

23. 23
• The very low loads are mainly rack enclosures with wiring patch panels,
switches, and hubs
• Loads in the 1 kW range are mainly sparsely populated rack enclosures
• Loads in the 2-3 kW range are mainly rack enclosures that are populated
with typical equipment but with significant unfilled rack space
• Loads in the 5 kW range are partially loaded with 1U servers, or contain a
mix of technologies
• Loads in the 7 kW + range are extremely rare but, according to customers,
are going to become more common with the recent density increases
resulting from server technology advancements
• Colocation Facilities must provide for all the above and in INAP’s case
up to 20kW/ Cabinet
Power densities in an average Enterprise
Data Center
#DataCenter

24. 24
The Electrical Issue of High Power Density
Checklist
1. Know your equipment power requirement (Voltage, Amps, Phase,
Primary, Redundant) needs to allow electrical design planning. If
you’re a collocation provider you must prepare for all types of
circuits
2. Be sure your team understands how to manage circuits and
determining overloading situations before they are installed
3. Pre-plan your electrical architecture with your rack plan layout to
manage conduit and circuit locations
4. Design your system to be flexible to avoid stranded capacity,
installation cost sticker shock and risk reduction
#DataCenter

25. 25
The Electrical Issue of High Power Density
Infrastructure deployment needs to consider:
• Sold / reserved customer power capacity
• Actual capacity being drawn by the racks
• Ensuring enough circuit / pole positions are available
Infrastructure topology should:
• Minimize upfront / Day 1 installation costs
• Avoid prematurely installing or stranding infrastructure
• Scale without significant cost premium to accommodate more racks & more power capacity
Customer load profiles at INAP data centers are trending towards an
INCREASE in kW / Rack or Power Density (w/sq ft)
• Increased risk of inadvertently overloading the infrastructure
• Increased risk of Customer power capacity demand out-pacing the ability to
deploy infrastructure
#DataCenter

26. 26
The Electrical Issue
We want to ensure that our power infrastructure does
not have STRANDED capacity
• The generators, switchgear, UPS are the most costly components; we want
the capacity of those to be “fully COMMITTED”
The Customer power circuit demands vary throughout
the Data Hall
• Each circuit requires a circuit breaker that uses “pole positions” within an
RPP panel
• To ensure we do not strand capacity, we need enough circuits to handle “low
density” racks
• “High density” racks require fewer pole positions and fewer RPPs
#DataCenter

27. 27
1. There are different SIZES of Circuits (capacity) that can be provided:
• The table below indicates a number of different breaker configurations and the resulting kW
• The most popular circuits are 208V/30A, 2-pole (5kW) & 208V/30A, 3-pole (8.6kW)
• Although the normal power draw across these circuits may be lower, there is nothing to prevent a
customer from drawing up to the maximum circuit load at any point in time
2. The sold or COMMITTED capacity is always significantly higher than the ACTUAL capacity
• If COMMITTED capacity EXCEEDS the infrastructure capacity then it is OVERSUBSCRIBED
• Even if you monitor the actual power consumption, a customer may increase their power draw without
warning
• There is no means to automatically shed load from the RPP, PDU, or UPS
The Electrical Issue

28. 28
1. Each Data Hall can be sub-divided into 3-zones (upper / middle / lower)
• Initially install ONE UPS and 3 PDU pairs (6 PDUs) for a data hall
• Capacity across the total datacenter is up to 1200kW
2. As customer racks are deployed in a given zone install additional RPPs to
support LOW power density
• Each zone initially provisioned with up to 500kW of capacity
• Up to 10 RPPs (4 panel with dual 400A inputs, RPP Option 3) per zone (5
A-side, 5 B-side) means there will be sufficient circuit capacity
• RPPs to be floor mounted to reduce length of branch circuits
3. Monitor the overall power consumption and install an additional UPS as required
4. As the power density in a given zone increases deploy additional PDUs and
RPPs to serve that increase in load
• Additional RPPs to be wall-mounted (2 panel with single 400A input, RPP
Option 1) for easier retrofit installation
The following slides indicate a phased
deployment approach
#DataCenter

29. 29
4-­‐Panel
RPP
(A-­‐
side)
4-­‐Panel
RPP
(B-­‐
side)
PDU
Pair
(Fed
by
UPS
1)
The Electrical Issue

30. 30
2-­‐Panel
RPP
(A-­‐
side)
PDU
Pair
(Fed
by
UPS
2)
The Electrical Issue

31. 31
3 Key Takeaways
1. The High Power Density option is available, but for the average
enterprise it remains a small footprint. Internap provides this
option for all customers
2. There are various types of cooling solutions dependent upon
the actual density that needs to be addressed
3. Electrical distribution in a high power density data center
requires pre-design and proper circuit management to
avoid issues
#DataCenter

32. 32
References
1. Clark, J. (2013, October 24). Raising Data Center Power Density. Retrieved March 19, 2015, from http://
www.datacenterjournal.com/it/raising-data-center-power-density/
2. High Density = Lower Cost ( Efficiency Without Compromise E-Book Series, Chapter 2). (n.d.). Retrieved March
19, 2015, from http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white
papers/emerson network power higher density equals lower cost.pdf
3. Brown, K., Torell, W., & Avelar, V. (n.d.). Retrieved March 19, 2015, from http://www.apcmedia.com/salestools/
VAVR-8B3VJQ/VAVR-8B3VJQ_R0_EN.pdf?sdirect=true
#DataCenter