Virtual Data Centre Design
A blueprint for success
IT has become the back bone
of every business. Advances
in computing have resulted in
economies of scale, allowing
large companies to integrate
business processes and
effectively establish standard
transactions through out the
enterprise. Against this
backdrop, building a data
centre to last 10 years is a
formidable challenge for major
Commerzbank, a major German bank, is no exception. Faced with the mammoth
task of creating a future-proof data centre in London, Gary Cane, Commerzbank
head of M&E, decided to use 3-D simulations techniques. This allows Commerzbank
to create a virtual model of the data centre and test the validity of the design
assumptions from an operational and cost point of view before committing to build
Most importantly, this simulation approach can provide the basis for all future
operational changes within the facility.
The Thermal Problem - Why Worry?
Power on the rise
Almost 40 years ago Gordon Moore at Intel made the famous observation that gate
density in microelectronics wills double every 18 months. His prediction has been
proven to be well founded and although we haven’t seen an exact parallel in the heat
produced because of other improvements in techniques, the trend has certainly been
followed. A decade ago a rack of electronics would typically consume a few hundred
watts of power which unsurprisingly converts directly into heat. Now, a typical rack
of electronics may consume 10 times that with 2 -3 kW in a rack being a common
occurrence with maximum heat dissipation reaching 10kW and occasionally even
Can air based cooling deliver?
The simple fact is that whilst historically the industry has used liquid cooling systems
to deal with hot equipment in the past, whenever possible, it has reverted to using
air to cool electronics not just because of the increased financial and operational
overheads of installing such a system, but also because of the not unnatural and
quite possibly justified psychological aversion to mixing water and electricity.
This may seem ridiculous because by comparison with water air is almost an
insulator. In numerical terms the difference seems quite phenomenal. For a 1
degree temperature rise, 1 litre of water can carry away about 3.5 times the amount
of heat that 1m3 of air can remove.
While the power dissipation remained low in the data centre this was not a problem,
but as the rate of heat output continues to rise, it becomes increasingly more
challenging to deliver the high volumes of air required to cool the equipment.
Fortunately as most of these facilities are virtually unoccupied the environments can
be designed for the equipment.
The result is that we can configure the system so that the room is cool where the
equipment needs it and warm elsewhere. This can mean cold aisles are cold and hot
aisles really are hot provided the equipment layout and configuration practices follow
the necessary procedures.
In practice while only a limited number of data centres suffer from a lack overall
cooling capacity, many more are facing the increasing need to cool hot spots with
locally significant requirements for cool air.
Since the historical approach has been to use an open plenum (the void created by
the raised floor) the ability to control the distribution of cool air to and within the
room has been limited not only because airflow is complex but also because we can’t
see it and therefore have limited understanding of the complex interactions that
Preventing the DC reaching the end of its life prematurely.
Changes in cooling requirement also present a new business dilemma. How can you
be certain the multimillion pound facility you build will have the necessary flexibility
to accommodate new equipment?
The fundamental backdrop it is critical to ensure the risk of thermally induced failure-
which ultimately threatens transaction processing at the heart of the business - is
What Can A Virtual Model Offer?
A virtual model of a data centre removes the invisible nature of air flow and allows
the company to take control of the cooling in their facility. By visualizing airflow and
heat transfer you can understand the relationship between cause and effect in the
This provides the ability to induce schemes to control and distribute the cool air thus
eliminating hot spots. It also allows the company to induce new equipment to the
facility. These simple benefits demonstrate a key need for facility IT managers to
work together in addressing a common threat – that of overheating.
To create a virtual model of a data centre that can accurately predict the key
performance characteristics of the facility from an airflow and temperature
perspective it is necessary to understand the complete configuration of the facility.
This includes the basis and details of the cooling system as well as the equipment
installed and its likely distribution.
As a result a virtual model may well offer the first real view of what the data centre
look and feel like in its 3 dimensional form will.
Not only does this technology offer the designer, and IT and facilities engineers a
technical assessment it also provides a tool that can illustrate the situation to any
members of the team who are less familiar with the technical issues but still involved
in any decision making process. This can include items at the initial design stage and
operational changes as they are introduced.
The Initial design stage
1. Define virtual data centre (VDC) as a model the can predict the likely
performance of the thermal environmental.
2. Help with the choice and the layout of the proposed air handler units to optimise
floor void pressurisation for the ventilation system.
3. Maximise the use of available space by configuring the floor tile layouts to suit
any proposed equipment layout and room architecture. This avoids the creation
of hotspots and thus minimise risk.
4. Ensure efficient delivery of the cool air and effective scavenging of the warm
exhaust air to improve energy efficiency
5. Define a method of balancing the power dissipation from cabinets to provide
adequate cooling to all equipment at all times.
6. Create a new and scientific method of communication between the IT and
Operational stage checklist:
1. Use the VDC to place new equipment in the facility and analyse the
environmental impact prior to making any decisions.
2. Maintain an up-to-date VDC.
Meeting the Business’ Needs
Balancing IT and Facility Requirements
The facility shown, Error! Reference source not found. 1) was the initial layout
for a proposed communications room.
Figure 1 Initial Proposed Layout
The proposed room is just over 24m long band approximately 8 .5m wide. The slab
to slab height is not large considering that a raised floor approach is to be used.
Besides, there are several structural beams that protrude downwards and reduce the
room’s overall height. These structural beams are aligned with the direction of flow
for return air moving from the hot exhausts from the equipment back towards the
Figure 2 Revised Proposed Configuration
This layout was considered sufficient to house about 50 cabinets together with two
short rows of patch connection racks. Although shown in the original drawing, the
fire suppression equipment is not likely to be housed in the room.
Importantly the cooling strategy was to use six down-flow cooing units to pressurise
a floor plenum and create cold aisles in front of the equipment. Although the
orthogonal rack layouts could were initially thought to be an easy access, a final
examination shown the opposite. In fact, such a layout could make it difficult to
separate the hot aisles from the cold aisles.
Figure 3 Power and Data Cables
As if expected, the building construction itself presented some constraints for the
way in which the equipment could be arranged. For example, the equipment must
be placed so that the structural columns, typical in this type of relatively open space,
do not limit access to the equipment cabinets or result in restricted access unsuitable
from either the equipment or health and safety perspective.
In view of the overall building services it was decided that the cooling units would be
better placed on the wall opposite to their original location. In addition it was
decided to move the patch connection racks to the other long side wall. This has the
advantage of less interruption to the main equipment cabinets.
In the revised proposal, (Figure 2), equipment cabinets are arranged in two rows
parallel to the length of the room. Several management practices were adopted
The designation of specified cable routes(Figure 3) allow:
Data into and out of the facility to/from the patch/distribution cabinets;
Power supply under the raised floor without crossing the data routes.
The use of floor bungs to limit the leakage of cool air from the ventilated floor
into the room. The objective here was to limit the size of the opening around the
cable to a hole no more than 25mm by 25mm.
Only fitting floor grilles in the cold aisle and to allow the warm air to return to the
cooling units over the top of the equipment.
Identifying Undesirable Risk
The revised proposal (Figure 4 & Figure 3) included:
35 No. floor grilles placed in a single row between the cabinets, supplying air
from 6 down-flow cooling units capable of delivering up to 40kW of cooling to
return air scavenged from the room via a top return and discharging 3m3/s of air
vertically downwards into the void;
Floor 25% open at the rear of the patch connection racks;
45 Cabinets housing 4kW of rack mounted equipment cooled front to back and
ventilated through perforated front and rear doors;
Figure 3 Rack/Cabinet Layouts
Figure 4 ACU, Floor Grille & Rack Locations
The revised layout appears to distribute the heat more uniformly along the length of
the room reflecting the installation and distribution of cooling. An early concern was
whether the system could provide good pressure distribution throughout the floor
void and therefore be able to deliver good cooling to all the cabinets and enclosed
equipment. The chief worry was the proximity of the cooling units at the left hand
end of the facility as shown where the room is most narrow.
Airflow modelling uses the fundamental concepts from physics of the conservation of
mass momentum and energy to predict the ventilation performance of the VDC and
therefore the resulting environment for personnel and equipment alike.
Because the solution depends on such fundamentals the validity of the results will be
dependent largely on the quality of the model as described by the user.
For example, the way in which the equipment is installed in the cabinets and how
much power each dissipates.
Results of the simulation for this revised proposal showed some very interesting
The short throw of the air from the cooling units, which is limited by the room
width, results in high stagnation pressure by the opposite wall.
The consequence is that there is a considerable flow of cool air from the floor
void into the room through the cable penetrations in the floor to the patch
connection racks where cooling is not needed.
Figure 5 shows that the velocity through the patch row is almost as great as the
velocity through the floor tile penetrations.
Figure 5 Flow from the Floor Void to the Room (Red) Coloured by Magnitude
of Upward Velocity
The picture of air flow (Figure 7) shows how these cools (blue) air simply short
circuits back to the cooling units having done little or no work.
A symptom of the problem is that the cooling load on the ACU’s to the left of the
room is low (they receive cool – blue air) while the ACU’s in the middle receive
warmer air (yellow) and the ACU’s to the right warmer air still (orange and red)
The difficulties at this right hand end of the room result from the fact that a
disproportionate amount of air leaves the floor void to the left of the room due to
the additional floor openings in the floor for the patch connection racks, and the
slightly asymmetric placement of the ACU’s resulting from the room geometry.
The warmer conditions at this end of the room are further exacerbated by the
higher equipment density but with gaps in one of the equipment rows.
Figure 6 Airflow Paths in the Room Coloured by Temperature
In a hot aisle / cold aisle approach equipment is always more vulnerable when it
is on the end of a row because being on the end introduced another path for
warm air from the hot aisle to return to the cold aisle creating undesirable mixing
of the hot and cold air before it is drawn into the rack mounted equipment to do
its cooling work. In this case (Figure 7) shows the temperatures in a plane just
in front of the equipment.
Figure 7 Temperatures in an Elevation Just Outside the Equipment Inlet
The air is generally warmer at the ends of the rows (including next to gaps).
The air is warmer towards the right hand end of the room (away from the patch
Equipment in the row with gaps is hotter than equipment in the equivalent row
In the row with gaps a number of racks experience inlet temperatures of between
25°C and 30°C which is often considered borderline whilst the row without gaps
generally experiences good conditions <25°C.
An elevation though one of the gaps clearly shows (Figure 9) the way in which the
warm air from the back of the equipment penetrates into the cold aisle. Cool air
from the floor grille is deflected to the inlet of the continuous row.
Figure 9 Temperatures in an Elevation Across the Two Rows of Equipment
Indeed, the solution highlighted several important issues. The design team discussed
these and proposed some key action points.
They are as follows:
Install shelving (Figure ) in the 2 smaller gaps not required for access to
reduce the ingress of warm air from the hot aisle.
Figure 10 Shelving to be Installed in Gaps
Install a vertical barrier (Figure 11) under the floor along the line of the
floor jack / pedestals one tile in front of the patch connection racks to
prevent leakage through the cable holes associated with patch
The structured cable layout under the floor means that there will only be
one penetration of this barrier by cables to open and re-seal when work
is done on the under-floor cabling. This was deemed more practical
method of reducing the leakage from the floor void than trying to seal
the individual floor penetrations for each of the patch connection racks.
Review the number of floor grilles to try and ensure best coverage of the
racks / cabinets at the end of the rows.
Figure 11 Barrier to be Installed in Floor Void
The changes have produced noticeable results. Firstly the loads on the ACU’s are
much more even. This is clearly characterised by the fact that the air temperatures
returning to the cooling units are much more even, (Figure 12).Notice now that the
highest return air temperatures do not reach 30°C but are more in the mid twenties.
Notice also that the air at the left hand end of the room is no longer about
20°C as it was in the original proposal but that now the temperature is elevated and
has done some work.
Figure 12 Airflow with the Improvements in Place
The inlet temperatures, Figure are also much improved with only very limited
locations where temperatures exceed the normally considered good range of up to
Figure 13 Inlet Temperatures with Improvements in Place
Benefits to the Business
The detailed case shows that it pays to create a 3-D virtual model of a data centre to
examine factors including airflow. Indeed, it’s this procedure that increased the
certainty the facility layout will work.
More important, should Commerzbank choose to proceed with the construction of
this facility they can be confident that the data centre will satisfy performance
requirements both now and in the future as more equipment is housed there.
In conclusion, undertaking this analysis has produced several business benefits. We
have identified modifications that will minimise the risk of equipment overheating.
This exercise also ensures that all cabinets housed within the facility are exposed to
good inlet conditions, a measure that cuts the time spent deciding where to install
But there are still opera issues. Clearly, it will still be important to follow god
practice when determining the configuration of the rack mounted equipment within
the cabinet and the total load per cabinet.
Nonetheless, this detailed analysis has successfully brought the IT and Facilities
Management teams together to achieve a common vision – and developed a virtually
future proof data centre.