This talk will cover the major drivers to go green and some important metrics, describe some of the measures that we’ve taken at Cardiff (as well as an example from elsewhere in the UK), starting where the power enters the building, looking at the Datacentre, checking out PC consumption and use on the desktop, and thinking about how to influence people and the organisation. These are just examples that may or may not be relevant for your context; e.g.‘free’ cooling might be relevant up a mountain in Scotland (and should be in Cardiff, but… more of that later) where the ambient temperature is low all year round. Free cooling might not be so relevant in warmer countries. However, what I hope to do is to show you what and how we have thought about the problem, and perhaps you will in turn find that useful. I’ll finish with a paradoxical claim - a tool that ensures that PCs use the maximum amount of electricity possible can be fairly considered to be as ‘Green’ as possible. I hope that I’ll be able to persuade you that this last claim is not ‘Greenwash’.
So, what do we mean by Green IT ? There are many answers out there, which range from ‘minimal or no impact on the environment’, through ‘environmentally sustainable’, to ‘use of resources in an efficient way’. I’m not going to debate the definition with you, but I will attempt to persuade you that, whichever definition you choose, it may not be as hard as you think to become a little bit (or perhaps a lot) greener, even though you know that ‘zero environmental impact’ is a long way off Going ‘Green’ is not a one-off set of actions: it is an on-going struggle to understand what is happening on a wide range of levels, measuring systems and analysing data, and then selecting the next target for improvement. I expect that we’re all familiar with ‘Greenwash’ – a hybrid engine that uses more petrol than conventional competitors, a two-stroke engined lawn mower that hypes its recycled packaging, the claim that a product is free of CFCs (which have been banned for years), thin-client energy comparisons using Pentium 4 energy examples (CPU energy has gone down by &gt;50% in recent years), and so on, and on, and on..... Sometimes it results in using more energy for greatly improved services. Is that Green ? Well, a bus uses more fuel per mile travelled than a car, but less fuel per passenger mile travelled – more energy, yes, but also more efficient. So, when selecting metrics to target for improvement, make sure you select the appropriate metric. So, there are large amounts of conflicting information and misinformation out there. How do we decide what to do ? How do you distinguish the irrelevant claim from the useful ?
There are a number of drivers for doing ‘Green IT’. Not all of these points apply to all environments. The relative priority of all of these drivers will vary, but energy prices are likely to become more significant as oil prices rise. Computers and computation are increasingly used in all areas of human endeavour, not just the hard sciences. This is being driven by a number of factors:- Growth in the quantities of data readily accessible over the Internet – census, genome databases, etc. Growth in data capture and mining of varying sorts – think health statistics or Amazon purchasing patterns. Personal music and photo collections are increasing so we now need TeraByte disks at home. All of these trends lead to a growth in demand for CPU & Storage, which all uses more physical space and electrical power, neither of them always available where you need them. For example, building new Datacentres in London with very high property prices is no longer very practicable. In many locations, there is insufficient electricity generating capacity to power the new Datacentres that are required, and the costs associated with this are also increasing as oil reserves dwindle. It is not only these fundamental constraints on the costs and availability of space and power that produce drivers to ‘go green’. Organisations are becoming increasingly concerned with their reputation. Enhancing the Reputation of Cardiff University & attracting better students by emphasising how ‘green’ we are is a definite driver for us. Sustainable IT is another great phrase which often means “sort out your future funding”, but is often translated to “being as green as possible”. Legal compliance is arriving as another driver, as governments strive to reduce their carbon emissions in line with international agreements. To complete this run through the main drivers, and this is a more personal view, I believe that we need to develop and improve our capabilities in these areas because we should (for the Planet, climate change, finite nature of most natural resources, legacy for our descendants, etc).
So, what should we measure ? The problem of deciding which metric to use is a bit like the old question of which came first, the chicken or the egg. You need to have some idea of what the numbers are before you can decide which numbers to measure (and thus target for improvement). So, how do you overcome that problem ?
This is where your ability to produce “guesstimates” really matters. Let’s start with electrical power, and a useful observation to help you cost things. At UK prices, 1kW used for a year costs approximately £1000. In your organisation, how many of you know how much power is used by all desktop PCs, and how much is used by servers ? For those of you who know (or have a rough idea), this could be the clue as to where to start your Green programme – part of your local context. If you don’t know these numbers, then perhaps you should ? Perhaps you can do some rough modelling (a “guesstimate”) as follows :- Measure the consumption (at idle) of your newest and oldest PCs (and monitors), and split the difference. Say you get 150W. Assume a duty cycle of 20% (8 hours per day, 220 days per year) – or if you have hard data, use that. Estimate (or better still know) how many PCs you have. From that, estimate the total PC power consumption. For 10,000 PCs, that equates to an electricity bill of around £0.3M p.a. at UK prices. Estimate your average server consumption – say 300W. Multiply by 2 to allow for cooling and other losses (more discussion on that later). Assume 100% duty cycle – most servers are on 24X7. Estimate (or count) your servers. For 300 servers, that equates to an electricity bill of around £0.3M. So, for the hypothetical numbers above, the servers are using about the same as the PCs. Now ask yourself where there might be possibilities for savings. Perhaps you could halve the duty cycle of PCs by encouraging more people to switch them off, or by implementing Windows power saving methods – that could halve the power consumption. Maybe moving to a thin-client solution would do even better. Perhaps you could improve the efficiency of your server cooling (more later), or would server virtualisation help ? The most promising initial target may not be with the largest consumer of electricity, but perhaps with the opportunity that gets you the necessary backing from the Finance Director. I can’t tell you what the right answer is; it all depends on your local context, and how far you are down this particular road. So much for the electricity consumption side of things. What else might we measure that has a part to play in our decision making processes ? Lifetime carbon costs of equipment are notoriously difficult to measure, but you can get useful info out there on the net. Paper consumption is a good one to tackle, as the embedded energy costs are high and many chemicals used in production are hazardous to the environment. Toner costs per printed page vary widely, with smaller printers being more expensive to run. Now what about the benefits that you get (or don’t get) from all of these IT services that use so much electricity ? Are you running old services that no-one really uses any more, and you just hadn’t noticed ? Perhaps they are running on old Pentium 4 based CPUs that had one of the highest power ratings out of the entire x86 series. Do you run multiple (physical) web servers based on Apache that are lightly loaded ? If so, run them as multiple Apache instances on one physical box (another form of virtualisation).
So, let’s consider some specific metrics such as:- Cost (Watts of electricity, capital investment, staff effort) CPU hours (how many hours of a Pentium did my computer job use) MFlops/Watt (Millions of Floating Point Operations (maths) per Watt of electricity) – Green Top 500 June 2010 – 773.38Mflops/Watt – Juelich supercomputer. Cardiff, 2008 ~ 200 Mflops/Watt. GB/Watt (GigaBytes of data stored per Watt of electricity) Gbps/Watt (Gigabits per second of network bandwidth per Watt of electricity) Cooling and power losses (PUE) Lifecycle Assessment – energy in production, distribution, disposal, recycling – can be significant. Pages printed (or not) Cost per page printed (printer, toner, paper) Physical space used saved – see SwaP = Performance/(Space X Power) For a research-based university, an important metric could be research papers published or cited. Note that this ‘Context’ affects your drivers. If you are running a supercomputer that supports some researchers, you might just use something as simplistic as the numbers of CPU hours or MFlops consumed during a research project. Alternatively, your Green IT metric might compare that papers published or cited to the total number of CPU hours or MFlops provided that supports that research. In both cases, an enhanced Mflops/Watt rating for your supercomputer would be a ‘Green benefit’ and thus could be a good metric. For the whole university, an on-going reduction in pages printed would probably be a good thing, so are you currently recording this number ? Or suppose you think outside the box. Could you use all that waste heat from the servers to heat your swimming pool, or warm your greenhouse ? Good for the UK, perhaps, but maybe you don’t need this in Australia ? The importance of local context again. To summarise on Metrics:- Use broad brush guesstimates, and then go for more accurate measurement of selected metrics. Pick the low hanging fruit and/or the largest wins, implement, measure, start again. Watch for ‘unintended consequences’.
What is PUE ? PUE=(Total power into the Datacentre)/(Total power into the computers in the Datacentre) A PUE of 1 is just about perfect. 2 is bad. 1.2 is ‘state of the art’. This graph shows how PUE was changing at Microsoft’s datacentres from 2004 to 2007.
So, where does all the power go ? The first lesson is to consider every step of the path that your electricity follows, from the point where it enters your building, probably to a high voltage (HV) transformer Our Estates department quotes an expected 3% saving for using the right HV transformer, at an additional cost of £500 For a new project, running at 50% load, the ROI is approximately 1 month !!!!!!! For a replacement, the work and disposal (old transformer) costs take that to a year or more – not too bad when the lifetime may be 10-20 years.
Look at losses when you get into the room itself following the energy all the way to where it leaves (usually as hot air).
Shown here is a top down view of a NetShelter SX rack with two InRow RC cooling units on either side. Both the rear of the NetShelter SX rack and the 2 InRow units are contained creating a closed loop system. As you can see: Fans of InRow cooling unit(s) distribute cool air to front of IT equipment Air passes through servers and absorbs heat Server exhaust air is prevented from escaping by rear containment All exhaust air is returned to InRow cooling unit(s) Proper airflow through the enclosure is ensured This particular configuration allows can allow for upwards of 25kW per rack providing 2N cooling redundancy. A variant of this configuration would be to use rear containment only allowing the cold supply air leaving the InRow cooling unit to disperse into the cold aisle. This would allow for sharing of cooling capacity with neighboring IT equipment.
There has been a gradual recognition that the more you can cool the hot air before you mix it with any cold air, the laws of thermodynamics tell you that it will take less energy. Thus hot aisle, cold aisle became common in Datacentres, followed by solutions such as APC’s Hot Aisle Containment, and other solutions such as cooled doors). These methods have varying pros and cons depending on your context again, but should all be considered.
Chilled water supplies are supposedly cheaper if they use free cooling – when it is below ~10 Centigrade, reduced refrigerant cycle, below 3.5, no cycle. But what about your context ? If it is hot all year round, free cooling may not be an option). Also, does it work in your environment anyway (perhaps not so well in Australia as half way up a mountain in Wales) ?
UPSs are not 100% efficient. When run at low percentage levels of their maximum, they can lose anything from 10% up to 25%, even if they are 95% efficient at full load. So, make sure you size your UPS at the Goldilocks level (just right). This is not as easy as it sounds. Our HPC cluster uses 115kW when performing the benchmarks used in acceptance tests, but under normal production use never exceeds 80% of that level. So, what’s the answer to that ? Perhaps it is to buy a modular UPS like the one we bought. http://hightech.lbl.gov/ups.html
What does this slide tell us ? 6% difference in efficiency between 2 specific UPS models. Generally, the smaller the UPS, the less efficient. For 100kW UPS running at 50% load, that 6% difference translates to £3000 p.a. http://www.apcmedia.com/salestools/VAVR-6LJV7V_R0_EN.pdf
What is our PUE, and why it should be better. What does the graph show ? No correlation between temperature and PUE; Free Cooling not working. We believe that this is because a buffer tank is needed, as the mechanical cooling was turning on and off too frequently, which wastes energy. We will shortly be fitting a buffer tank, and hope to see improvements as a result. Also, we have not yet raised operating temperatures, which will further reduce energy needed to cool everything. What are some of the other questions you might ask ? Server Power supplies – how efficient are yours (Blades vs Pizza boxes) ? How do you choose the CPUs for your servers ? Do you buy the fastest, or those with the best MFlops/Watt. What impact does the choice of memory or disks have on energy ? How much might be wasted by the overnight running of services only used in the day ? Are your servers doing useful work all of the time ? (virtualise, etc) How much space is consumed ? Space has a cost, and so do newer cooling methods. Sweating the assets on your cooling infrastructure can reduce the per unit cost of cooling.
These new chillers are supposed to have a higher CoP (Coefficient of Performance) and will be used in conjunction with the existing Free Cooling chillers.
To summarise our procurement, here are the Key Power Saving Solutions (Slide18), and the Key business benefits. (Slide 19) Note that there is a good ROI on the large efficient room-based UPS. Partly from buying ‘Just what you need’, rather than ‘Maximum you might need’ Cooling ROI, potentially very good. Time will tell when we have the numbers. Servers – ROI on quad-core difficult to assess, but we are VERY happy with a decision that effectively gave us twice the ‘grunt’ for the same money, took up less space, and will use less power.
How much energy can you save through server virtualisation and correct storage selection ? What is server virtualisation – running many virtual servers in one physical server. This relies on the fact that many physical servers are running at very low CPU utilisation levels, so you can run many virtual servers in a single physical server whilst delivering the same performance to users.
Watts X hours per day = watt-hours per day; X days per year = watt-hours per year; ÷ 1,000 = kWh per year; X energy cost per kWh = total annual energy cost.200W X 24hrs per day = 4800 watt-hours per day; X 365 days per year = 1752000 watt-hours per year; ÷ 1,000 = 1752 kWh per year X 12p per kWh = 21024p =£210.24 100W X 24hrs per day = 2400 watt-hours per day; X 365 days per year = 876000 watt-hours per year; ÷ 1,000 = 876 kWh per year X 12p per kWh = 10512p =£105.12
Watts X hours per day = watt-hours per day; X days per year = watt-hours per year; ÷ 1,000 = kWh per year; X energy cost per kWh = total annual energy cost.500W X 24hrs per day = 12000 watt-hours per day; X 365 days per year = 4380000 watt-hours per year; ÷ 1,000 = 4380 kWh per year X 12p per kWh = 52560p =£525.60 250W X 24hrs per day = 6000 watt-hours per day; X 365 days per year = 4380000 watt-hours per year; ÷ 1,000 = 2190 kWh per year X 12p per kWh = 26280p =£262.80
Cardiff used the following estimates as part of the business case for virtualising:- Total electricity cost reduction over 3 years from £425k down to £57k (and capital from £1.15M down to £191k So, having picked off the low hanging fruit by virtualising, where else can we look for savings in our server estate ? Can you make further energy reductions ? Yes. Look at automated VMotion & power down (save money at night and during the weekends). What about the newer CPUs and their ability to dynamically slow down the CPU ?
Cardiff won a grant to model and measure the energy impact of tiered storage – Project was called Planet Filestore. Capital Costs:- A 450GB 15k SAS disk costs about the same as a 2TB 7k2 SATA disk. So, if your top tier is Raid 10, SAS disks, mirrored across two sites, and your bottom tier is Raid 5 (8+1) SATA and not site-mirrored, you see an approx X16 difference in £/usable TB. (~X4 from the disk size, ~X2 from Raid5 vs Raid10, and another X2 from the site mirroring).
How much energy does the storage consume per usable TB ? Start with the X16 and then note that a 7k2 disk typically uses 15W, and a 15k disk uses 25W. So if your disks are all Tier 1 (say 400GB usable = 100GB in Raid10, site mirrored) = 1000 disks @ 25W = 25kW ~ £25,000 For HSM, with 10TB Tier 1 & 90TB Tier 2. Raid 5 array (8+1) of 2TB SATA delivers 16TB usable. Therefore need 6 arrays = 45 SATA disks. If Raid 5 array is (4+1), then 55 SATA disks. Use average of 50 disks. Tier 1&2 = £2500 + ((50 disks@15W) = 0.75kW) £750 = £3250 Energy savings of £21,750 for every 100TB
Compare the energy use of a PC left on 24X7, with one that is only turned on during working hours (say 8 hours per day, 200 days per year. That is equal to 20% of the time). So, lesson number one is to turn them off when you are not using them. Context changes over time – PC energy use 2005 PC – 200W-300W 2010 PC – 50W-100W 2005 CRT Monitor – 60W-150W 2010 LCD Monitor – 15W-30W
Should you implement one of the many packages that can provide Asset Management and Power Control & Recording (e.g. Nightwatchman) ? Very useful to prove to management that your changes are having an effect. Can you influence your users ? Perhaps the very recent Microsoft Research offering (Joulemeter, an alpha release piece of software, so be cautious) would help change user behaviour ? This is based on the idea that providing more information about energy usage can directly affect behaviour.
Don’t believe everything you read Data from 2007 (Stanford Research) concerning thin client energy benefits quoted a thin client @ 15W and a PC @ 300W.
What’s wrong with this ? Most PCs don’t use 300W anymore. 50-75W would be a better figure. See the paper for an up to date estimate of savings for a thin client scenario.
Deskjet energy costs vs centralised, larger printers “Over the life of a laser printer, Lexmark calculates some 80% of the costs will be for the paper it uses, around 6% for toner cartridges and around 8% for the electricity. For an ink-jet, it’s 29% for the paper costs, 7% for ink cartridges and 16% for electricity.” Perhaps you should stop using ink-jet printers ?
I’m presenting here some measured results in a UK University (Liverpool John Moores – LJMU). Around 20,000 undergraduates, with a more centralised management structure than Cardiff. This facilitated them in making a business case to save significant money by a comprehensive change to all printing facilities. They implemented a new chargeable printing system with students charged via software-based accounts that they can ‘top-up’, and staff charged via departmental accounts. They use Print Release stations to reduce actual pages printed, and gradually removed deskjet-type printers.
There were difficulties in changing user attitudes, but once people got used to the new system, they generally preferred it.
Network switches and routers don’t use much electricity, do they ? Well, that’s changing, as we start to see more devices using Power over Ethernet (PoE), and more desktops using Gbps connections. I’ve seen values for Gbps switches ranging from 0.5Gbps/W up to 3.5Gbps/Watt. So let’s do a guesstimate for Cardiff’s LAN. We have around 50,000 outlets, all of them connected to switches. Right now, they are mostly 100Mbps, but we’ll probably move to Gbps over the next few years, and unless switch designers do something about this, those 50,000 ports on edge switches could be using anything from 15kW up to around 100kW. A difference of around £85k p.a., and I haven’t added in the core and distribution switches. What about PoE ? The newer PoE standards deliver up to 25.5W, with some manufacturers claiming 50W capability. A 100m Cat 5 cable run will lose around 5W. PoE typically delivers around 50V, which is then DC-DC converted at the powered device incurring more losses – I’ve seen 10-20% quoted. Add that up and you could be wasting up to 10W out of the standard 25W. That’s a staggering 40% loss. One other thing to watch for is the cabling. At these power levels, you can’t tie together more than around 100 cables, or they’ll start to get too hot !! So, although switch manufacturers are starting to catch up with the Green marketing push and offering power saving features such as automated power down of unused ports, they still have a long way to go, and some of the major players have the least efficient kit.
Can your organisation help by changing your purchasing policies ? Yes. Here are just a few examples of how you can do this. Insist on your suppliers presenting information regarding aspects like EnergyStar ratings, or end-of-life disposal for PCs. Buy the Greener PC models. We have found these to cost approx £30 more, which pays for itself within 2 to 3 years. Discourage the purchase of the very fastest PCs – they may use 50% more power for a 10% performance improvement. For High Performance Computing (HPC) and general server purchasing, insist on suppliers contractually committing to consumption, and use their figures as part of your evaluation, perhaps in a TCO figure. Require your suppliers to only publicise energy-efficient systems when providing quotes to your organisation. Get them to configure energy saving settings as part of your supplied configuration.
I have already covered some examples of how to influence both your organisation and users in the previous examples. Perhaps the most important point here is to consider the audience, and what you are trying to achieve. Local context comes into play yet again. For example, in Cardiff, all electricity bills are paid centrally, so Heads of Academic Schools are not particularly interested in how much energy their PCs use, but the Head of Estates is. In other organisations, the bills are devolved, so you need to target a different audience for this information. Individuals may have little interest in the bigger picture concerning the energy impact of virtualisation on the total energy bills, but your management probably do. However, telling each PC user how much energy they have used today, and how much they might save by changing the power saving settings has a real chance of changing things. Technical types like me are probably influenced more easily when told the detail of how some energy saving measure works, whereas the Finance Director just wants to know the impact on the bottom line and the IT Director may want reassuring that it won’t take lots of staff time.
Finally, here is a counter-intuitive example of ‘more is less’. There is a freely available package called Condor, which makes all those spare CPU cycles in your PCs available for certain types of HPC calculation. The effect on your PCs is to make them use more energy, all of the time. So why might this be green ?
The basic answer is that you are getting a greater value from the extra energy that one of these PCs uses when doing some HPC work (provided that it is already turned on) than you would get from running a dedicated HPC service on a set of servers. Let’s show how that works by using a guesstimate again. We have measured the energy consumption of many different PC models to show this effect. Let’s consider a typical modern PC - a 2.4GHz Core2Duo PC. Its measured power usage when idle was 65W, and when running at 100% CPU utilisation was 81W. So, it only adds 16W to run your HPC work. Now compare that with a dedicated HPC system build from the same components – it will use the full 81W to do the same work – that is 5 times as much as the PC. Actually, it is not that bad, because the dedicated HPC system is probably more efficient, but note that it would have to be 5 times more efficient to match the PC. There is another environmental advantage to Condor that relates to the embodied manufacturing costs. You have this PC anyway, and all that Condor is doing is providing extra value by ‘sweating the asset’. It does not add to those costs in any way. It is also possible to run Condor in a fashion that does not get these benefits. I know of one University that only runs Condor on their PCs at night, by using ‘Wake on LAN’ to turn PCs on when a user submits jobs to the Condor system. They still ‘sweat their asset’ and have provided a useful HPC system with almost zero capital investment, but they don’t get the energy benefit identified above. What about a changing technical context ? We have observed that, as CPUs get more and more cores, the difference between 0% and 100% gets less, and so Condor becomes even more efficient compared with a dedicated cluster.
I have covered some of the many ways in which Cardiff University is trying to genuinely improve its Green credentials and presented overviews of some important ones. The implementation of Green policies will in many cases save money over the longer term, as well as benefiting the environment. Organisations should think carefully about the metrics they employ to measure success in this area, and should adopt a constant cycle of choosing the next target metric, implement selected improvements, measure the result, and then start again. In all cases, it is important to understand the local context, whether that be the ambient temperature of your location, your organisation’s prime drivers, or the need to capitalise on opportunities to maximise the usage of components. Remember that the context (technical, political, environmental, etc) can change. People can help if you encourage and inform them, and finally, don’t forget the wacky idea. Sometimes it is a winner. Questions ?
How do you make your Data Centre Greener ? I’ll be talking here about the experience that we gained when procuring a new supercomputer for Cardiff, along with the Datacentre to house it. One of the early pointers I found was a report to the US Congress in Aug 2007 on Datacentres. If you are interested in the issues and trends, look it up. http://hightech.lbl.gov/documents/DATA_CENTERS/EPA-datacenters.pdf
Ran 2 simultaneous tenders – 1 for HPS, 1 for environment – Difficult, required good relationship and cooperation between 2 suppliers (and our Estates dept). Crucial aspect to make it clear that tenders were to be evaluated on the basis of a 3 yr TCO, requiring suppliers to commit to power consumption figures, UPS efficiencies and cooling overheads. In addition, we wish to retro-fit additional metering to one of our existing machine rooms, so we will have a direct comparison.
Holistic approaches to green it gregynog colloquium - june 2011
Holistic Approaches to
Dr Hugh Beedie
ARCCA & INSRV
Going Green Can Save Money
What is Green IT ?
Drivers & Metrics
Data Centres, Power & Cooling
Servers, Virtualisation, and Storage
PCs, Monitors & Thin clients
Printing and Networks
Influencing user behavior & the organisation
‘Thinking outside the box’ – save energy with
What is Green IT ?
No environmental impact
Use of resources in an efficient way
Often faulty comparisons
Importance of your Context
Your situation, drivers & metrics…
Can change over time
Drivers – Why do Green IT?
Increasing demand for CPU & Storage
Lack of Space
Lack of Power
Increasing energy bills (oil prices doubled)
Enhancing the reputation of your University &
attracting better students
Sustainable IT & Legal Compliance
Because we should (for the Planet )
Metrics – Chicken and Egg
You can’t measure everything.
So, which numbers do you
measure, when you don’t know
which are important ?
Which use more, PCs or Servers ?
10,000 PCs using
150W, 8 hrs/day, 220 days/yr
UK electricity £300,000 p.a.
500 servers using
300W + losses (X 2)
24 X 7
UK prices £300,000 p.a.
So where do we start ?
Examples of Metrics
Cost (Watts, TCO, capital investment, etc)
Value (CPU Hrs, MFlops, Gbps, Papers)
Value/Cost is better ? (MFlops/Watt)
Cooling & power losses (PUE)
Average kWhr per PC per year
Pages printed (or not printed ?)
Cost per page (electricity, paper)
Power Usage Effectiveness
PUE=Total Power/Computer Power
What wastes the power ?
Power Conversion in HV Transformer
Before it gets to your room, you lose power.
Transformer Efficiency = 98% or 95%
Return On Investment (ROI)?
Capital Cost = £5500 or £5000 (650kVA)
New installation (@200kW), ROI = 1 month
Replacement, ROI = 1 year
Lifetime of investment = 20+ yrs !!!!!
Data Centre Consumption
Source: Intel Corp.
Drives , I/O
Cooling Inside the Room
Hot Aisle/Cold Aisle Layout
Hot Aisle Containment
Seals in hot air,
with room air Chamber
to hot aisle, locks
Prevents hot air
InRow RC unit
conditioner cools hot
exhaust air directly
Rack Air Containment
Server exhaust air is
escaping by rear
All exhaust air is
returned to InRow
Fans of InRow
cooling unit distribute
cool air to front of IT
Top Down View
APC Inline RC units
• Up to 20kW/rack
• Uniform cooling allows raising cold air temp
Cooling Inside the Room
Temperature (deg C)
<24 27.4 30.8 34.3 >37.8
Cooling – Outside the Room
3 Airedale Chillers
Ultima Compact Free
Variable speed fans
(We needed that )
System operated on
(12% of Year)
System operated on
(50% of Year)
System operated on
mechanical cooling only
(38% of Year)
Mechanical (compressor) cooling
Back Up Power
Full and half load
Modular – could
grow as we grew,
High Density Servers
Important if you spend on advanced
Can have shared power supplies (PSU)
Blade Servers have shared PSUs
Not always good for HPC
Bull supplied half-width, 1U servers.
2, quad core, 80W Harpertown CPUs
Cardiff’s PUE vs Temperature
Average PUE = 1.3
No correlation with temperature ?
Remove 1 120kW Free cooling chiller
Add 1 300kW Evaporative chiller
Expansion for HPC Wales
Key Power Saving Solutions
Low power servers
High Density Servers
(sweating the asset)
Chilled Water“Air Handling”
Virtualisation and Storage
Many virtual servers in one physical server
How much can you save ?
How much of that is energy ?
Fast disk=450GB, 25W, Slow disk=2TB, 15W
Tiering = Energy saving
Equipment FY Unit Total
100 RM servers - replacement cost, inc installation and peripherals
(assumes Dell PowerEdge 1950)
08/09 £2700 £270K
100 ‘Out of warranty’ servers – 2 year extended warranty 08/09 £500 £50K
250 Production Servers - replacement cost, inc installation and peripherals
(assumes Dell PowerEdge 1950)
09 - 11 £2700 £675K
1 Air conditioning upgrades in RWD 1.71 and Main 1.71 Computer
08/09 £100K £100K
6 12KVA and 16KVA UPS replacements 09/10 £9K £54K
Total hardware replacement costs (1 for 1) £1.15M
450 Cost of electricity for servers (200w per server @12p/KW hour) Annual £94K £283K
450 Cost of electricity for cooling (100w per server @12p/KW hour) * Annual £47K £142K
Total electricity costs £425K
Total hardware and electricity costs £1.24M
* Inefficient raised floor and ceiling mount air conditioning units
Equipment FY Unit Total
RM Servers – replacement cost, inc installation and peripherals
(assumes Dell PowerEdge P900 with 32GB RAM, 2TB Disk)
08/09 £5K £25K
‘Out of warranty’ servers– replacement cost, inc installation and
peripherals (assumes Dell PowerEdge P900 with 32GB RAM, 2TB Disk)
09/10 £5K £25K
Production Servers – replacement cost, inc installation and peripherals
(assumes Dell PowerEdge P900 with 32GB RAM, 2TB Disk)
09/11 £5K £65K
2 24TB Central Storage – assumes Sun Fire X4500 or Equivalent 08/09 £15K £30K
46 VMware ESX Licences – one per CPU 08 - 11 £1K £46K
Physical to Virtual (P2V) Software ? 08/09 ? ?
VMware backup software? 08/09 ? ?
Total hardware costs (VMware Infrastructure) £191K
25 Cost of electricity for servers (500w per server @12p/KW hour) Annual £13K £39K
25 Cost of electricity for cooling (250w per server @12p/KW hour)* Annual £6K £18K
Total electricity costs £57K
Total hardware and electricity costs £248K
* Inefficient raised floor and ceiling mount air conditioning units
Virtualisation Business Case
3 Year ROI with Virtualisation
Total hardware costs £1.15M
Total electricity costs £425K
Total costs £1.57M
Total hardware costs
Total electricity costs £57K
Total costs £248K
JISC-funded project to measure savings of..
Tiered storage – scenario
100TB needed for 30000 (staff and students)
High performance for 7000 concurrent users
Site mirrored Raid 10 450GB SAS vs
Non mirrored Raid 5 2TB SATA
~ factor of 16 in hardware cost
Therefore, use HSM (old data on Tier 2)
What about energy ?
Planet Filestore Project
Fast disk (FC/SAS)
Slow disk (SATA)
Tier 1 – DR
Tier 2 – non DR
100TB Tier 1 (DR)
10TB Tier 1 (DR)
90TB Tier 2(nonDR)
Energy saving = £21,750
PCs & Monitors
Where to start ?
Duty cycle guesstimate
Always on (100%)
On during working
Turn it off at night !!
What next for PCs ?
More stringent power saving policies
Asset Management (reporting)
Influencing user behaviour (Joulemeter)
What about Thin Clients ?
Year PC Thin Client
2005 300W 15W
2010 50W 5W
Still some scope, but not
what there used to be
Thanks to Liverpool John Moores
University (LJMU) for the next 2 slides
Carbon Footprint Savings
• Hundreds of devices removed
– Old lasers
– Fax machines
• Replaced with fleet of 220 devices
• Automatic power saving after 2 hours
• Use of recycled paper
• FollowMe documents held for 18 hours
Power used by switches
3.5Gbps/W to 0.5Gbps/W
Cardiff LAN has 50,000 outlets
100kW down to 15kW
Potential £85k p.a. by choosing well
PoE (up to 25W down a UTP cable)
5W in the cable, 5W DC-DC conversion
Use EnergyStar PCs
Discourage use of the fastest PCs
Use energy in your TCO calculations
Ensure suppliers only publicise green PCs
Context – who pays for electricity
Target your messages
• Their PC uses ???W
• Pages printed last week ??
Technical types – detail
• MFlops, kW, Geeks/Pint
• Bottom line ££££
What is Green, but makes your PC use
more electricity ?
Answer – Condor
So, what is Condor ?
Condor takes spare CPU Cycles for use in
Runs at low priority so users don’t notice
So why is it Green ?
Measure, Improve, Measure again
Look everywhere for waste
Context changes over time
Make sure you revisit your assumptions
Don’t just do the technical stuff
Think out of the box (Condor)
Cardiff’s New Datacentre
£3M grant to purchase a supercomputer
New Datacentre required with appropriate
power, cooling ..
Our Context ?
INSRV Sustainability Mission - To minimise
CU’s IT environmental impact and to be a
leader in delivering sustainable information
Limited experience in Datacentre build
Server & Data Center Energy Efficiency
(US Congress Report 2007)
What did Cardiff do ?
Ran 2 tenders, many contractors (& subs)
HPC & Environment (power, cooling)
Strong project management
TCO as key evaluation criterion
Value = HPC Benchmark figures
Winner = Highest Value/TCO