This lecture presented during the CLIM Transition Workshop

1. UNIVERSITY OF
CALIFORNIA

2. SAMSI CLIM 2018
Progress and activities in the Parameter
Optimisation working group
Ben Timmermans (LBNL, coordinator)
Ben Lee (Penn. State)
Murali Haran (Penn. State)
Charles Jackson (University of Texas)
Andrew Gettelman (NCAR)
15th May 2018

3. Outline

The Group

Background and scope

Discussion points and conclusions (to date)

Ongoing group activities

4. Group members

Me (LBNL)

UQ for numerical modeling of atmospheric climate extremes

Ben Lee (Penn. State)

Fast calibration methods for numerical models with large parameter spaces

Charles Jackson (U. Texas at Austin)

Use of observations to constrain climate models (paleoclimate / atmosphere)

Andrew Gettelman (NCAR)

Atmospheric process modeling, model development (CAM / CESM)

Murali Haran (Penn. State)

Statistician, statistical computing and cross-disciplinary climate research

5. Sacks et al., 1989

6. Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-
Resolution CESM to Microphysics and Horizontal Resolution”
Recent motivating example

7. (Recent) motivating example
Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-
Resolution CESM to Microphysics and Horizontal Resolution”

8. (Recent) motivating example
Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-
Resolution CESM to Microphysics and Horizontal Resolution”

9. (Recent) motivating example
Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-
Resolution CESM to Microphysics and Horizontal Resolution”

10. (Recent) motivating example
Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-
Resolution CESM to Microphysics and Horizontal Resolution”

11. (Recent) motivating example
Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-
Resolution CESM to Microphysics and Horizontal Resolution”

12. Historical background & context

13. 
Early research into the design and analysis of computer experiments
motivated calibration and optimisation.

Sacks et al. (1989), Santner et al. (2001)

Probabilistic Bayesian approach pioneered by Kennedy & O'Hagan (2001)

Probabilistic approach using a Gaussian process for 'model discrepancy'

Explicitly account for 'model inadequacy' (i.e. discrepancy between simulation
output and observation)

Typically results in a complex 'posterior' that dictates inference via MCMC.

Efficient surrogate models or emulators typically required

Various discussion by and follow-up work: Bayarri, Trucano, Higdon, Lucas,
Rougier, Williamson, Chang, others...
Background

15. 
Data assimilation approaches

Ensemble Kalman filters (Carrassi et al., 2017)

Parallel sampling (does not get around model cost, but enhances posterior analysis)

Learning in a forecast setting (Schneider et al., 2017)
Recent advances

16. Discussion points

17. 
How to chose methodology?

Expensive numerical model: Emulator required, but restrictions input / output
dimensionality

Cheap numerical model (< few minutes, on a few CPU cores): choices available

Problem specific

Emulators remain impractical (or at least very complex) for
certain classes of problem

Expertise not available (black art!)
Practical problems

18. 
We are testing (climate) models with the same data used to
construct them.

While there may be improvements in climate modeling for recent climate,
we still lack a clear methodology for understanding what measures are
important for projections.

Measures of model skill seem unrelated to projection uncertainty (Kloche
& Pincus, 2011)

Formulation of the likelihood / cost function.

Considerable uncertainty remains over the choice of observational
constraints

Model bias and structural error.

In principle it can be explored but in practice it remains challenging.
Philosophical problems

19. 
Most studies represent a 'first pass' of a problem, considering the
mean (e.g. climate sensitivity)

Can we / should we optimize w.r.t. higher moments of the output
(e.g. extremes)
Further thoughts...

20. Conclusions

21. Keep things simple ... ?

Employ cheap models where possible (e.g. single column
model).

Use high quality observational data.

Failures are then easier to track down and cheaper to identify.

22. Keep things simple ... ?

23. Calibration of single column model
Observations Default run
Cost = 88.5
Calibrated run
Cost = 76.7
Cloud fraction
Cost function: cloud fraction short-wave radiation liquid water path

24. Group Activities

25. • Test case: a climate model run for 1 week to represent a single aircraft flight track
• Case study: supercooled liquid clouds in the S. Hemisphere
• Model does a good job on representing clouds in the right place, but a bad job of
representing supercooled liquid
Aircraft Observation (10s=3km, 360s~100km)
CESM Grid Box (100km @ 60N)
Flight track with 10s obs ( )
HIPPO Project flight tracks. Focus on RF05
1. Cloud phase experiment (AG & CJ)

26. Blue Shading:
CESM Ice clouds
Gray Shading:
CESM Liquid Clouds
GV Altitude
Pressure
+ HIPPO Ice (Ni>0)
+ HIPPO Liq (Nc >0)
+ HIPPO Liq T<0C
Latitude
◊
+
◊
+
+
Section along H4RF05 (Jun) Flight Track
Model broadly reproduces
cloud in the right place, but
misses supercooled liquid
Ice number concentration
is well reproduced by the
model
Model misses supercooled
liquid at -20˚C

27. 2. Wind-wave model calibration (BT & BL)

Forcing winds: Patricola, 2018 (Nature, in review)

Timmermans et al., 2018 (Oceanography, in review)

29. 2. Wind-wave model calibration (BT & BL)

30. 
Rhoades et al. (2018) JAMES “Sensitivity of Mountain Hydroclimate Simulations in Variable-Resolution CESM
to Microphysics and Horizontal Resolution”

Sacks et al. (1989) “Design and Analysis of Computer Experiments” Statistical Science

Kennedy & O'Hagan (2001) “Bayesian calibration of computer models” Journal of the Royal Statistical
Association B

Santner et al. (2003) “The Design and Analysis of Computer Experiments” Springer, New York

Oakley & O'Hagan (2004) “Probabilistic sensitivity analysis of complex models: a Bayesian approach” J. R.
Statist. Soc. B

Bayaari et al., (2007) “A Framework for Validation of Computer Models” Technometrics

Chang et al. (2014) “Fast dimension-reduced climate model calibration and the effect of data aggregation”
Annals of Appl. Stat.

Williamson, D. (2015) “Exploratory ensemble designs for environmental models using k-extended Latin
Hypercubes” Environmetrics

Molteni F. (2003) “Atmospheric simulations using a GCM with simplified physical parametrizations. I. Model
climatology and variability in multi-decadal experiments” Clim. Dyn.

Schneider et al. (2017) “Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and
Targeted High-Resolution Simulations” GRL

Timmermans et al. (2018) “Hurricanes, ocean warming and big waves” Oceanography (in review)

Patricola, C. (2018) “Future change in hurricanes under ocean warming” Nature (in review)
Selected references

31. Extreme output

We are concerned with a range of extremes:

Risk Ratio.

Counting events of a certain “magnitude” or over a threshold.

High quantiles.

Fitted parameterised extreme value distributions (GEV).

32. Experiment design: CAM parameters

Parameters known to induce variability (E.g. Qian et al. 2015)

33. Experimental period

Fewer longer runs (e.g. 5 to 10 years)
vs

More abundant shorter runs (e.g. 1 to 3 year)

What period?

Angelil et al. (2017) Independent assessement of attribution
statements... (2010 - 2013)

34. 2 years spin-up
Nat-Hist (1) / All-Hist (1)
Defaults
2007 -> 2009 2007 -> 2009
Spin-up Nat-Hist (1 mem.) Spin-up All-Hist (1 mem.)
2009 -> 2010/10 (150) 2009 -> 2010/10 (150)
Spin-up PPEs Spin-up PPEs
2010/10 -> 2013/10 2010/10 -> 2013/10
(150 x 28) (150 x 28)
= 2 x 11,000 years

35. PPE: Initial Findings

44. Sensitivity Analysis for Numerical Models

Numerous sources of uncertainty affect numerical models.

Representation of physics, adjustment via tunable
paremeters.

Input parameters, X ∈ ℝn
: 10 < n < ?

Inference for (and similar):

45. Use of emulators

Computational expense necessitates the use of emulators.

Lots of options available: Regression methods, Kriging (GP
models), neural networks, ...

But we want tractable and usable methods to alleviate the
complexity of the numerical model.

Some formulations allow for closed form solutions for UA / SA
(Oakley & O'Hagan, 2004).

Emulators are valuable well beyond UA / SA (optimisation, ...).

46. Numerical models

ICTP AGCM runs at ~4
degrees

47. Output: Precipitation in Nebraska

48. Model performance (Nebraska)

49. Quantile response (Nebraska)

50. Gaussian process model (Nebraska)

51. GEV fit (Nebraska, member 15)

52. Pareto parameter response (Nebraska)

53. Gaussian process models (Nebraska)

54. Summary

For a low resolution model of intermediate complexity, simple
regression emulation approaches would seem reasonable for
high (< 0.999) quantiles.

Response in scale and shape parameters appears weak: internal
variability / fitting uncertainty dominates.

Need to determine relationship between tail characteristics,
internal variability and its relationship to parameter variability.
(“Threshold index” for parameters?)

55. Statistical tools

Statistical time series models that can generate heavy tails:
Generalised Auto-Regressive Conditionally Heteroskedastic
models (GARCH)
 Process at
is auto-regressive in variance, not mean.
ϵ = unit variance white noise process
ω, α, β = coefficients

56. GARCH model fitting

57. Emulators

Simple hierarchical model set-up:

58. Emulators: Heirarchical

59. Emulators: Uncertainty Analysis

60. Summary

For a low resolution model of intermediate complexity, simple
emulation approaches for extremes would seem reasonable.

Will this remain the case for more complex models? (Probably
not)

Can we make use of statistical tools (e.g. GARCH models) to
investigate experiment design and hierarchical approaches.
(Possibly?)

61. References

Qian et al. (2015) “Parametric sensitivity analysis of precipitation at global and local
scales in the Community Atmosphere Model CAM5” JAMES

Molteni F (2003) “Atmospheric simulations using a GCM with simplified physical
parametrizations. I. Model climatology and variability in multi-decadal experiments.”
Clim. Dyn.

Kennedy & O'Hagan (2000) “Predicting the output from a complex computer code
when fast approximateions are available.” Biometrika

Oakley & O'Hagan (2004) “Probabilistic sensitivity analysis of complex models: a
Bayesian approach” J. R. Statist. Soc. B

Timmermans et al. (2017) “ Impact of tropical cyclones on modeled extreme wind-wave
climate” GRL

Wehner et al. (2015) “Resolution dependence of future tropical cyclone projections of
CAM5.1 in the U.S. CLIVAR hurricane working group idealized configurations” Journal
of Climate

62. References

Catching Waves: A Comparison of Models

https://www.youtube.com/watch?v=lkyCc1MMEVE