27 March 2024… IARPC Collaborations, Modelers’ Community of Practice (Presentation): Explainable AI approach for evaluating climate models in the Arctic. Remote Presentation. References... Labe, Z. M., & Barnes, E. A. (2022). Comparison of climate model large ensembles with observations in the Arctic using simple neural networks. Earth and Space Science, 9(7), e2022EA002348, https://doi.org/10.1029/2022EA002348

1. Explainable AI approach for
evaluating climate models
in the Arctic
Zachary M. Labe
Postdoc at NOAA GFDL and Princeton University; Atmospheric and Oceanic Science
with Elizabeth A. Barnes (Colorado State University)
27 March 2024
IARPC Collaborations
Modelers’ Community of Practice
https://zacklabe.com/ @ZLabe

2. Relative to 1951-1980

4. STANDARD EVALUATION OF
CLIMATE MODELS
Pattern correlation
RMSE
EOFs
Trends, anomalies, mean state
Climate modes of variability
Negative Correlation Positive Correlation
PATTERN CORRELATION – T2M
PATTERN CORRELATION : NEAR-SURFACE AIR TEMPERATURE

5. INPUT PREDICTION
SO, WHAT ABOUT
MACHINE LEARNING?

6. Machine Learning
is not new!
“A Bayesian Neural Network for
Severe-Hail Prediction (2000)”
“Classification of Convective Areas
Using Decision Trees (2009)”
“A Neural Network for Damaging
Wind Prediction (1998)”
“Generative Additive Models versus
Linear Regression in Generating
Probabilistic MOS Forecasts of
Aviation Weather Parameters (1995)”
”A Neural Network for
Tornado Prediction
Based on Doppler
Radar-Derived
Attributes (1996)”
”The Diagnosis of
Upper-Level Humidity
(1968)”

7. Do it better
e.g., parameterizations in climate models are not
perfect, use ML to make them more accurate
Do it faster
e.g., code in climate models is very slow (but we
know the right answer) - use ML methods to speed
things up
Do something new
• e.g., go looking for non-linear relationships you
didn’t know were there
WHY ELSE SHOULD WE CONSIDER
MACHINE LEARNING?

8. Do it better
e.g., parameterizations in climate models are not
perfect, use ML to make them more accurate
Do it faster
e.g., code in climate models is very slow (but we
know the right answer) - use ML methods to speed
things up
Do something new
• e.g., go looking for non-linear relationships you
didn’t know were there
WHY ELSE SHOULD WE CONSIDER
MACHINE LEARNING?
Very relevant for
research: may be
slower and worse,
but can still learn
something

9. What is the annual mean temperature of Earth?

10. THE REAL WORLD
(Observations)
What is the annual mean temperature of Earth?
Data from
Berkeley Earth Surface Temperature
1930 2022

11. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Let’s run a
climate model
One ensemble member
2022
1930 2050
Data
from
NO
AA/G
FDL SPEAR

12. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Let’s run a
climate model
again!
Two ensemble members
Data
from
NO
AA/G
FDL SPEAR

13. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
Let’s run a
climate model
again & again!
Three ensemble members
Data
from
NO
AA/G
FDL SPEAR

14. What is the annual mean temperature of Earth?
THE REAL WORLD
(Observations)
CLIMATE MODEL
LARGE ENSEMBLE
30 ensemble
members in
NOAA/GFDL
SPEAR

15. What is the annual mean temperature of Earth?
Mean of ensembles
= forced response (climate change)
THE REAL WORLD
(Observations)
CLIMATE MODEL
LARGE ENSEMBLE
30
ensemble members
In GFDL SPEAR

16. What is the annual mean temperature of Earth?
Mean of ensembles
= forced response (climate change)
THE REAL WORLD
(Observations)
CLIMATE MODEL
LARGE ENSEMBLE
30
ensemble members
In GFDL SPEAR
Range of ensembles
= internal variability (noise)
Mean of ensembles
= forced response (climate change)

17. But let’s remove
climate change…
Climate Change Signal
(ensemble mean)
Observations
Ensemble
Members

18. Ensemble
Members
Mean of
anomalies
After removing the
forced response…
= anomalies/noise!

19. Ensemble
members in
GFDL SPEAR
Maps of a given time period for each ensemble
Inputs for machine learning

20. Ensemble
members in
GFDL SPEAR
Training Data:
24 ensemble members
Maps of a given time period for each ensemble

21. Training Data:
24 ensemble members
Validation Data:
4 ensemble members

22. Training Data:
24 ensemble members
Validation Data:
4 ensemble members
Testing Data:
2 ensemble members

23. 2-m Actual Air Temperature (°C)
THERE ARE MANY CLIMATE MODEL LARGE ENSEMBLES…
Annual mean 2-m temperature
7 global climate models
16 ensembles each
ERA5 (observations)

24. INPUT
[DATA]
PREDICTION
Machine
Learning

25. ----ANN----
2 Hidden Layers
10 Nodes each
Ridge Regularization
Early Stopping
TEMPERATURE
We know some metadata…
+ What year is it? (Labe & Barnes, 2021)
+ Where did it come from?

26. TEMPERATURE
We know some metadata…
+ What year is it? (Labe & Barnes, 2021)
+ Where did it come from?
Train on data from the
Multi-Model Large
Ensemble Archive

27. TEMPERATURE
We know some metadata…
+ What year is it? (Labe & Barnes, 2021)
+ Where did it come from?
NEURAL NETWORK
CLASSIFICATION TASK
HIDDEN LAYERS
INPUT LAYER
INPUT LAYER
OUTPUT LAYER
HIDDEN LAYERS

28. CLIMATE MODEL
MAP
[DATA]
Machine
Learning
CLASSIFICATION

29. CLASSIFICATION
Machine
Learning
CLIMATE MODEL
MAP
[DATA]

30. CLASSIFICATION
Machine
Learning
CLIMATE MODEL
MAP
[DATA]
Explainable AI
Learn new
science!

31. WHY
LAYER-WISE RELEVANCE PROPAGATION (LRP)
Volcano
Timber
Wolf
Image Classification LRP
LRP heatmaps show regions
of “relevance” that
contribute to the neural
network’s decision-making
process for a sample
belonging to a particular
output category
Neural Network
WHY
Backpropagation – LRP
https://heatmapping.org/

32. WHY
LAYER-WISE RELEVANCE PROPAGATION (LRP)
Volcano
Timber
Wolf
Image Classification LRP
LRP heatmaps show regions
of “relevance” that
contribute to the neural
network’s decision-making
process for a sample
belonging to a particular
output category
Neural Network
WHY
Backpropagation – LRP
https://heatmapping.org/

33. EXPLAINABLE AI (XAI)
THERE ARE MANY
METHODS
A bird!
XAI
[Adapted from Adebayo et al., 2020]

34. THERE ARE MANY
METHODS
EXPLAINABLE AI (XAI)
[Adapted from Adebayo et al., 2020]

35. Visualizing something we already know…
ENSO

36. Neural
Network
[0] La Niña [1] El Niño
Input a map of sea surface temperatures
[Toms et al. 2020, JAMES]

37. Visualizing something we already know…
Input maps of sea surface
temperatures (SST) to
identify El Niño or La Niña
Use ‘LRP’ to see how the
neural network is making
its decision
[Toms et al. 2020, JAMES]
Layer-wise Relevance Propagation
Composite SST Observations
LRP [Relevance]
SST Anomaly [°C]
0.00 0.75
0.0 1.5
-1.5
Warmer
Colder
High
Low

38. APPLYING METHODOLOGY TO THE ARCTIC

39. NEURAL NETWORK
CLASSIFICATION TASK
HIDDEN LAYERS
INPUT LAYER
OUTPUT LAYER
TEMPERATURE MAP
LABE AND BARNES 2022, ESS

40. APPLY SOFTMAX OPERATOR
IN THE OUTPUT LAYER
RANK
LABE AND BARNES 2022, ESS

41. APPLY SOFTMAX OPERATOR
IN THE OUTPUT LAYER
[ 0.71 ]
[ 0.05 ]
[ 0.01 ]
[ 0.01 ]
[ 0.03 ]
[ 0.11 ]
[ 0.08 ]
RANK
LABE AND BARNES 2022, ESS

42. APPLY SOFTMAX OPERATOR
IN THE OUTPUT LAYER
[ 0.71 ]
[ 0.05 ]
[ 0.01 ]
[ 0.01 ]
[ 0.03 ]
[ 0.11 ]
[ 0.08 ]
RANK
[ 1 ]
[ 4 ]
[ 7 ]
[ 6 ]
[ 5 ]
[ 2 ]
[ 3 ]
LABE AND BARNES 2022, ESS

43. APPLY SOFTMAX OPERATOR
IN THE OUTPUT LAYER
[ 0.71 ]
[ 0.05 ]
[ 0.01 ]
[ 0.01 ]
[ 0.03 ]
[ 0.11 ]
[ 0.08 ]
RANK
[ 1 ]
[ 4 ]
[ 7 ]
[ 6 ]
[ 5 ]
[ 2 ]
[ 3 ]
Confidence/Probability
LABE AND BARNES 2022, ESS

44. RANKING CLIMATE MODEL PREDICTIONS FOR EACH YEAR IN OBSERVATIONS
LABE AND BARNES 2022, ESS

45. COMPARING CLIMATE MODELS IN THE ARCTIC
High
Low
RECENT ARCTIC AMPLIFICATION
LABE AND BARNES 2022, ESS

46. High
Low
HISTORICAL PERIOD
COMPARING CLIMATE MODELS IN THE ARCTIC
LABE AND BARNES 2022, ESS

47. High
Low
DIFFERENCE IN LAYER-WISE RELEVANCE PROPAGATION
COMPARING CLIMATE MODELS IN THE ARCTIC
LABE AND BARNES 2022, ESS

48. KEY POINTS
1. Artificial neural network can identify which climate model produced an annual mean map of
near-surface temperature in the Arctic
2. Classification network is evaluated using input from atmospheric reanalysis as a method of
comparing climate models and observations
3. XAI method reveals regional temperature patterns the artificial neural network is using to
classify observations with different climate models
Zachary Labe
zachary.labe@noaa.gov
Labe, Z. M., & Barnes, E. A. (2022). Comparison of climate model large
ensembles with observations in the Arctic using simple neural networks.
Earth and Space Science, 9(7), e2022EA002348
https://doi.org/10.1029/2022EA002348