The document discusses the application of machine learning (ML) in climate science to enhance model accuracy, speed up processing, and discover new relationships in climate data. It emphasizes the integration of explainable AI techniques to understand ML predictions and improve the scientific workflow in atmospheric science. Key points include the shift from viewing ML as a black box to utilizing it as a valuable tool for generating new scientific insights.

1. LEARNING NEW CLIMATE SCIENCE BY
OPENING THE
MACHINE LEARNING BLACK BOX
@ZLabe
Zachary M. Labe
with Elizabeth A. Barnes
in the Department of Atmospheric Science
10 September 2021
Cognitive Brown Bag Series
CSU Department of Psychology

2. Machine Learning
is not new!
But…

3. Machine Learning
is not new!

4. Artificial Intelligence
Machine Learning
Deep Learning
Computer Science

5. Computer Science
Artificial Intelligence
Machine Learning
Deep Learning
Supervised
Learning
Unsupervised
Learning
Labeled data
Classification
Regression
Unlabeled data
Clustering
Dimension reduction

6. • Do it better
• e.g., parameterizations in climate models are not
perfect, use ML to make them more accurate
• Do it faster
• e.g., code in climate models is very slow (but we
know the right answer) - use ML methods to speed
things up
• Do something new
• e.g., go looking for non-linear relationships you
didn’t know were there
Very relevant for
research: may be
slower and worse,
but can still learn
something
WHY SHOULD WE CONSIDER
MACHINE LEARNING?

7. GROWING DATA

9. Adapted from: Kotamarthi, R., Hayhoe, K., Mearns, L., Wuebbles, D., Jacobs, J., & Jurado, J.
(2021). Global Climate Models. In Downscaling Techniques for High-Resolution Climate
Projections: From Global Change to Local Impacts (pp. 19-39). Cambridge: Cambridge University
Press. doi:10.1017/9781108601269.003
CLIMATE MODELS
Horizontal Grid
Vertical Levels
Past/Present/Future
Fully-Coupled System
20-40 Petabytes of data

10. ADAPTED FROM EYRING ET AL. 2016
CMIP6

11. GROWING TOOLS

14. Python tools for machine learning

15. Today’s weather or climate
scientist is far more likely to be
debugging code written in
Python… than to be poring over
satellite images or releasing
radiosondes.
“
D. Irving| Bulletin of the American Meteorological Society| 2016

16. Machine learning for weather
IDENTIFYING SEVERE THUNDERSTORMS
Molina et al. 2021
Toms et al. 2021
CLASSIFYING PHASE OF MADDEN-JULLIAN OSCILLATION
SATELLITE DETECTION
Lee et al. 2021
DETECTING TORNADOES
McGovern et al. 2019

17. Machine learning for climate
FINDING FORECASTS OF OPPORTUNITY
Mayer and Barnes, 2021
PREDICTING CLIMATE MODES OF VARIABILITY
Gordon et al. 2021, in review
TIMING OF CLIMATE CHANGE
Barnes et al. 2019

18. INPUT
[DATA]
PREDICTION
Machine
Learning

19. INPUT
[DATA]
PREDICTION
Machine
Learning

20. NSF AI Institute for Research
on Trustworthy AI in Weather,
Climate, and Coastal
Oceanography (AI2ES)
https://www.ai2es.org/

21. E.g.,
Research to
Operations (R2O)
Tornado Warning
Special Marine Warning
Severe Thunderstorm Warning
Flash Flood Warning

22. E.g.,
Establish robust,
responsible AI for
severe weather
detection
Tornado Warning
Special Marine Warning
Severe Thunderstorm Warning
Flash Flood Warning

23. Artificial Intelligence
Machine Learning
Deep Learning

24. X1
X2
INPUTS
Artificial Neural Networks [ANN]

25. Linear regression!
Artificial Neural Networks [ANN]
X1
X2
W1
W2
∑ = X1W1+ X2W2 + b
INPUTS
NODE

26. Artificial Neural Networks [ANN]
X1
X2
W1
W2
∑
INPUTS
NODE
Linear regression with non-linear
mapping by an “activation function”
Training of the network is merely
determining the weights “w” and
bias/offset “b"
= factivation(X1W1+ X2W2 + b)

27. X1
X2
∑
inputs
HIDDEN LAYERS
X3
∑
∑
∑
OUTPUT
= predictions
Artificial Neural Networks [ANN]
: : ::
INPUTS

28. Complexity and nonlinearities of the ANN allow it to learn many
different pathways of predictable behavior
Once trained, you have an array of weights and biases which can be
used for prediction on new data
INPUT
[DATA]
PREDICTION
Artificial Neural Networks [ANN]

29. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
TEMPERATURE

30. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
TEMPERATURE
We know some metadata…
+ What year is it?
+ Where did it come from?

31. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
We know some metadata…
+ What year is it?
+ Where did it come from?
[e.g., Barnes et al. 2019, 2020]
[e.g., Labe and Barnes, 2021]
TIMING OF EMERGENCE
(COMBINED VARIABLES)
RESPONSES TO
EXTERNAL CLIMATE
FORCINGS
PATTERNS OF
CLIMATE INDICATORS
[e.g., Rader et al. 2021, in prep]
Surface Temperature Map Precipitation Map
+
TEMPERATURE

32. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
We know some metadata…
+ What year is it?
+ Where did it come from?
[e.g., Barnes et al. 2019, 2020]
[e.g., Labe and Barnes, 2021]
TIMING OF EMERGENCE
(COMBINED VARIABLES)
RESPONSES TO
EXTERNAL CLIMATE
FORCINGS
PATTERNS OF
CLIMATE INDICATORS
[e.g., Rader et al. 2021, in prep]
Surface Temperature Map Precipitation Map
+
TEMPERATURE

33. THE REAL WORLD
(Observations)
What is the annual mean temperature of Earth?

34. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Anomaly is relative to 1951-1980

35. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Let’s run a
climate model

36. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Let’s run a
climate model
again

37. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Let’s run a
climate model
again & again

38. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
CLIMATE MODEL
ENSEMBLES

39. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
CLIMATE MODEL
ENSEMBLES
Range of ensembles
= internal variability (noise)
Mean of ensembles
= forced response (climate change)

40. What is the annual mean temperature of Earth?
Range of ensembles
= internal variability (noise)
Mean of ensembles
= forced response (climate change)
But let’s remove
climate change…

41. What is the annual mean temperature of Earth?
Range of ensembles
= internal variability (noise)
Mean of ensembles
= forced response (climate change)
After removing the
forced response…
anomalies/noise!

42. What is the annual mean temperature of Earth?
• Increasing greenhouse gases (CO2, CH4, N2O)
• Changes in industrial aerosols (SO4, BC, OC)
• Changes in biomass burning (aerosols)
• Changes in land-use & land-cover (albedo)

43. What is the annual mean temperature of Earth?
• Increasing greenhouse gases (CO2, CH4, N2O)
• Changes in industrial aerosols (SO4, BC, OC)
• Changes in biomass burning (aerosols)
• Changes in land-use & land-cover (albedo)
Plus everything else…
(Natural/internal variability)

44. What is the annual mean temperature of Earth?

45. Greenhouse gases fixed to 1920 levels
All forcings (CESM-LE)
Industrial aerosols fixed to 1920 levels
[Deser et al. 2020, JCLI]
Fully-coupled CESM1.1
20 Ensemble Members
Run from 1920-2080
Observations

46. So what?
Greenhouse gases = warming
Aerosols = ?? (though mostly cooling)
What are the relative responses
between greenhouse gas
and aerosol forcing?

47. Surface Temperature Map
ARTIFICIAL NEURAL NETWORK (ANN)

48. INPUT LAYER
Surface Temperature Map
ARTIFICIAL NEURAL NETWORK (ANN)

49. INPUT LAYER
Surface Temperature Map
ARTIFICIAL NEURAL NETWORK (ANN)
Collection of nodes (neurons)
that adjust their weights and
biases across layers in order to
learn signals for making
predictions
Learns nonlinear processes
through selected parameters
in the model

50. INPUT LAYER
HIDDEN LAYERS
OUTPUT LAYER
Surface Temperature Map
“2000-2009”
DECADE CLASS
“2070-2079”
“1920-1929”
ARTIFICIAL NEURAL NETWORK (ANN)

51. INPUT LAYER
HIDDEN LAYERS
OUTPUT LAYER
Surface Temperature Map
“2000-2009”
DECADE CLASS
“2070-2079”
“1920-1929”
BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI
ARTIFICIAL NEURAL NETWORK (ANN)

52. INPUT LAYER
HIDDEN LAYERS
OUTPUT LAYER
Layer-wise Relevance Propagation
Surface Temperature Map
“2000-2009”
DECADE CLASS
“2070-2079”
“1920-1929”
BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI
ARTIFICIAL NEURAL NETWORK (ANN)
[Barnes et al. 2020, JAMES]
[Labe and Barnes 2021, JAMES]

53. LAYER-WISE RELEVANCE PROPAGATION (LRP)
Volcano
Great White
Shark
Timber
Wolf
Image Classification LRP
https://heatmapping.org/
LRP heatmaps show regions
of “relevance” that
contribute to the neural
network’s decision-making
process for a sample
belonging to a particular
output category
Neural Network
WHY
WHY
WHY
Backpropagation – LRP

54. LAYER-WISE RELEVANCE PROPAGATION (LRP)
Volcano
Great White
Shark
Timber
Wolf
Image Classification LRP
https://heatmapping.org/
LRP heatmaps show regions
of “relevance” that
contribute to the neural
network’s decision-making
process for a sample
belonging to a particular
output category
Neural Network
WHY
WHY
WHY
Backpropagation – LRP

55. LAYER-WISE RELEVANCE PROPAGATION (LRP)
Volcano
Great White
Shark
Timber
Wolf
Image Classification LRP
https://heatmapping.org/
LRP heatmaps show regions
of “relevance” that
contribute to the neural
network’s decision-making
process for a sample
belonging to a particular
output category
Neural Network
Backpropagation – LRP
WHY
WHY
WHY

56. LAYER-WISE RELEVANCE PROPAGATION (LRP)
Image Classification LRP
https://heatmapping.org/
NOT PERFECT
Crock
Pot
Neural Network
Backpropagation – LRP
WHY

57. [Adapted from Adebayo et al., 2020]
EXPLAINABLE AI IS
NOT PERFECT
THERE ARE MANY
METHODS

58. [Adapted from Adebayo et al., 2020]
THERE ARE MANY
METHODS
EXPLAINABLE AI IS
NOT PERFECT

59. Visualizing something we already know…

60. Visualizing something we already know…
Input maps of sea surface
temperatures to identify
El Niño & La Niña
Use ‘LRP’ to see how the
neural network is making
its decision
[Toms et al. 2020, JAMES]

61. OUTPUT LAYER
Layer-wise Relevance Propagation
“2000-2009”
DECADE CLASS
“2070-2079”
“1920-1929”
BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI
WHY?
= LRP HEAT MAPS
[Labe and Barnes 2021, JAMES]

62. Layer-wise Relevance Propagation
BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI
WHY?
= LRP HEAT MAPS
Machine Learning
Black Box
[Labe and Barnes 2021, JAMES]

63. Layer-wise Relevance Propagation
BACK-PROPAGATE THROUGH NETWORK = EXPLAINABLE AI
WHY?
= LRP HEAT MAPS
Find regions of “relevance”
that contribute to the
neural network’s
decision-making process
[Labe and Barnes 2021, JAMES]

64. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS
Greenhouse gases fixed
to 1920 levels
[AEROSOLS PREVAIL]
Industrial aerosols fixed
to 1920 levels
[GREENHOUSE GASES PREVAIL]
All forcings
[STANDARD CESM-LE]
DATA

65. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS
Greenhouse gases fixed
to 1920 levels
[AEROSOLS PREVAIL]
Industrial aerosols fixed
to 1920 levels
[GREENHOUSE GASES PREVAIL]
All forcings
[STANDARD CESM-LE]
DATA

66. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS
Greenhouse gases fixed
to 1920 levels
[AEROSOLS PREVAIL]
Industrial aerosols fixed
to 1920 levels
[GREENHOUSE GASES PREVAIL]
All forcings
[STANDARD CESM-LE]
DATA

67. 1960-1999: ANNUAL MEAN TEMPERATURE TRENDS
Greenhouse gases fixed
to 1920 levels
[AEROSOLS PREVAIL]
Industrial aerosols fixed
to 1920 levels
[GREENHOUSE GASES PREVAIL]
All forcings
[STANDARD CESM-LE]
DATA

68. CLIMATE MODEL DATA PREDICT THE YEAR FROM MAPS OF TEMPERATURE
AEROSOLS
PREVAIL
GREENHOUSE GASES
PREVAIL
STANDARD
CLIMATE MODEL
[Labe and Barnes 2021, JAMES]

69. OBSERVATIONS PREDICT THE YEAR FROM MAPS OF TEMPERATURE
AEROSOLS
PREVAIL
GREENHOUSE GASES
PREVAIL
STANDARD
CLIMATE MODEL
[Labe and Barnes 2021, JAMES]

70. OBSERVATIONS
SLOPES
PREDICT THE YEAR FROM MAPS OF TEMPERATURE
AEROSOLS
PREVAIL
GREENHOUSE GASES
PREVAIL
STANDARD
CLIMATE MODEL
[Labe and Barnes 2021, JAMES]

71. [Labe
and
Barnes
2021,
JAMES]
ARE THE RESULTS ROBUST?
YES!
COMBINATIONS OF TRAINING/TESTING DATA

72. HOW DID THE ANN
MAKE ITS
PREDICTIONS?

73. HOW DID THE ANN
MAKE ITS
PREDICTIONS?
WHY IS THERE
GREATER SKILL
FOR GHG+?

74. RESULTS FROM LRP
[Labe and Barnes 2021, JAMES]

75. RESULTS FROM LRP
[Labe and Barnes 2021, JAMES]
WHAT IS
SIGNIFICANT?

76. 1. Shuffle ensemble member and year
dimensions (bootstrap-like method)
2. Apply true labels (unshuffled years)
3. Apply same ANN architecture and LRP
4. Repeat 500x by using different
combinations of training/testing data and
initialization seeds
5. Compute 95th percentile of the distribution
of LRP at all grid points
Uncertainty in LRP
[Labe and Barnes 2021, JAMES]

77. Uncertainty in LRP
Ultimately, we are trying to
mask noise in the LRP output
Identify robust climate pattern indicators!
[Labe and Barnes 2021, JAMES]

78. RESULTS FROM LRP
[Labe and Barnes 2021, JAMES]

79. RESULTS FROM LRP
[Labe and Barnes 2021, JAMES]

80. Higher LRP values indicate greater relevance
for the ANN’s prediction
AVERAGED OVER 1960-2039
Aerosol-driven
Greenhouse gas-driven
All forcings
Low High
[Labe and Barnes 2021, JAMES]

81. Greenhouse gas-driven
Aerosol-driven
All forcings
AVERAGED OVER 1960-2039
[Labe and Barnes 2021, JAMES]

82. DISTRIBUTIONS OF LRP
AVERAGED OVER 1960-2039
[Labe and Barnes 2021, JAMES]

83. DISTRIBUTIONS OF LRP
AVERAGED OVER 1960-2039
[Labe and Barnes 2021, JAMES]

84. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
TEMPERATURE
We know some metadata…
+ What year is it?
+ Where did it come from?

85. STANDARD EVALUATION OF
CLIMATE MODELS
Pattern correlation
RMSE
EOFs
Trends, anomalies, mean state
Climate modes of variability

86. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
TEMPERATURE
+ Where did it come from?

87. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
INPUT MAPS OF TEMPERATURE, PRECIPITATION, …

88. INPUT MAPS OF TEMPERATURE, PRECIPITATION, …
NEURAL NETWORK
----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
Classification Task

89. INPUT MAPS OF TEMPERATURE, PRECIPITATION, …
NEURAL NETWORK
WHICH MODEL DID THE MAP COME FROM?
----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
Classification Task

90. INPUT MAPS OF TEMPERATURE, PRECIPITATION, …
NEURAL NETWORK
WHICH MODEL DID THE MAP COME FROM?
----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
Classification Task
7 Classes – Large Ensemble GCMs
CMIP5 – MMLEA-SMILE Archive
16 Ensembles Per GCM
8th
class = Multi-Model Mean

91. COMPARING CLIMATE MODELS
LRP
(Explainable AI)
Raw data

92. What climate model
does the network
predict for each year
of observations?
Multi-Model Mean

93. APPLYING METHODOLOGY TO REGIONS
PREDICTION LRP (Explainable AI)

94. INPUT
[DATA]
PREDICTION
Machine
Learning
Explainable AI
Learn new
science!

95. MACHINE LEARNING IS JUST
ANOTHER TOOL TO ADD TO OUR
WORKFLOW.
1)

96. MACHINE LEARNING IS
NO LONGER A BLACK BOX.
2)

97. WE CAN LEARN NEW SCIENCE
FROM EXPLAINABLE AI.
3)

98. KEY POINTS
Zachary Labe
zmlabe@rams.colostate.edu
@ZLabe
1. Machine learning is just another tool to add to our scientific workflow
2. We can use explainable AI (XAI) methods to peer into the black box of machine learning
3. We can learn new science by using XAI methods in conjunction with existing statistical tools