19. 19
19
Datacenter summary
● Fully ducted datacenters already here
● Liquid cooling moving to mainstream
– First liquid-to-air
• Rear door heat exchanger
• Pumped system in server
– Next liquid-to-liquid
● Power density increasing by one-half next ten years
27. 27
27
ITRS Cost Performance
● Die size remaining constant at 140 mm2
● Power density increasing 0.05 W/mc2
per year
● Maximum power increasing 7 watts per year
● Junction temperature remaining constant at 90 C
● Thermal resistance less than half in 15 years
33. 33
33
ITRS High Performance
● Maximum size remaining constant at 750 mm2
● Power density increasing .05 W/mm2
per year
● Maximum power increasing 37.5 watts per year
● Junction temperature decreasing ~ 1.5 C per year
● Thermal resistance less than one-fourth in 15 years
38. 38
38
Leakage effects
● Increasing with every technology node
● Temperature dependence becoming stronger
● Can be 50% of processor power in extreme cases
● Relevant metric is % total power change per C
– typical value 0.5 to 2%/C
39. 39
39
Michigan: Kim et al. 2003
Total Chip Dynamic and Static Power
Dissipation Trends Based on the ITRS
41. 41
41
Thermal Resistance Chain -
Components
Heatsink
Junction Temperature
Air Temperature
Package
}
}
Heatsink
TIM2
Die
Lid
TIM1
Substrate
42. 42
42
Thermal Resistance Chain -
Resistances
Heatsink fin-to-air
Heatsink base
Thermal interface material 2 (TIM2)
Package lid
Thermal interface material 1 (TIM1)
Silicon die
43. 43
43
Thermal Resistance Chain -
Resistances
Silicon die
Thermal interface material 1 (TIM1)
Package lid
Thermal interface material 2 (TIM2)
Heatsink base
Heatsink fin-to-air
44. 44
44
Thermal Resistance Chain - Silicon
Silicon thermal conductivity 100 – 125 W/mK
Non-negligible temperature dependence
Half to two-thirds that of aluminum
One-fourth to one-third that of copper
Resulting value ~ 0.01 C/W
Increases for nonuniform power
45. 45
45
Thermal Resistance Chain - TIM1
Epoxy filled with metal or ceramic particles
Must withstand solder reflow temperature ~ 245 C
Bulk material conductivity 5 W/mK
Non-negligible contact resistance
Fill ratio limited by allowable stiffness
Resulting value 0.03 C/W
Increases for nonuniform power
46. 46
46
Thermal Resistance Chain –
Nonuniform Power
Die power concentrated mostly in cores
Very little power in cache
Resulting resistances up to double uniform
48. 48
48
Thermal Resistance Chain - Lid
Copper thermal conductivity ~ 400 W/mK
Thickness limited by stiffness
Nearly eliminates nonuniformity of heat flux
Resulting value 0.03 C/W
Relatively independent of nonuniform power
49. 49
49
Thermal Resistance Chain - TIM2
Thermal grease conductivity ~ 3 W/mK
Must be removable and cleanable
Mineral or silicone oil with metal or ceramic fillers
Fill ratio limited by flowability
Minimum thickness determined by
• Contact force
• Lid and heatsink flatness
Resulting value ~ 0.03 C/W
51. 51
51
Thermal Resistance Chain – Heatsink
Base
Embedded heatpipes or vapor chamber
Copper with subatmospheric water
Water evaporates and condenses
Heatpipe acts as very conductive (125x copper)
52. 52
52
Thermal Resistance Chain – Heatsink
Fin-to-Air
Strongest dependence is on volume airflow rate
Resistance increased by finite fin conductivity (efficiency) η
Resistance increased by finite pressure drop (effectiveness)
Capacity and convective conductances:
Ccap proportional to volume airflow rate
Ccon proportional to pressure drop
Heatsink resistance α 1/η{Ccap[1-e-(Ccon/Ccap)
]}
53. 53
53
Thermal Resistance Chain – Heatsink
Fin-to-Air
Volume airflow rate dictated by system power
Industry average about 120 cfm/kW, now about 80 cfm/kW
Fin efficiency typically about 75 to 90%
Higher for low profile heatsinks
Heatsink effectiveness typically about 75 to 90%
Diminishing return on pressure drop at high levels
54. 54
54
Thermal Resistance Chain – Heatsink-
to-Air
Heatsink-to-air resistance much higher than 1/Ccap:
Resistance of heatsink base
Finite fin efficiency
Finite heatsink effectiveness
Typical values around twice 1/Ccap
Resulting value as low as 0.09 C/W
59. 59
59
Fixed parameters in the thermal
resistance chain
● Data center air temperature (typically 35ºC)
● Electrical interconnect specifications
– Impacts connector & board designs
– Impacts air path possibilities
● Total thermal load
– Defined by system specification
● Processor thermal load
– Defined by chip designs
● Junction temperature (typically 95ºC)
60. 60
60
Ideal versus Real Design Features
CPU upstream of memory
All airflow through CPU
Mostly open midplane
Large diameter fans
Sometimes downstream
Some bypass occurs
Little open area
Only room for small fans
61. 61
61
Alternative technologies
● Solder TIM1
– Used by Intel, AMD, Fujitsu
● Lidless package
– Used in notebooks
● Liquid-to-air cooling
– Used by IBM, Fujtsu
● Liquid-to-liquid cooling
– Used by IBM, Fujitsu
● Coolant on silicon
– Used by Cray
71. 71
71
Server summary
● Processor power thermal resistance decreasing
– Cost performance less than half in 15 years
– High performance less than one-fourth in 15 years
● Memory power doubling in 10 years
● Alternative technologies emerging and returning
– Thermal interface materials
– Liquid cooling
72. 72
72
Future trends
● Datacenter
– Ambient temperatures increasing
• Outside air all the time
– Air-to-liquid first step
● Server
– Power increasing
– Device temperatures decreasing
– Liquid-to-air first step
– Liquid-to-liquid next
