The main goal of calibration is usually to improve the predictive performance of the simulator but the values of the parameters in the model may also be of intrinsic scientific interest in their own right. As an example of the latter we will discuss CO2 retrievals from the the Orbiting Carbon Observatory 2 (OCO-2). In order to make appropriate use of observations of the physical system it is important to recognize model discrepancy, the difference between reality and the simulator output. We illustrate through a simple example that an analysis that does not account for model discrepancy may lead to biased and over-confident parameter estimates and predictions. The challenge with incorporating model discrepancy in statistical inverse problems is being confounded with calibration parameters, which will only be resolved with meaningful priors. For our simple example, we model the model-discrepancy via a Gaussian process and demonstrate that through accounting for model discrepancy our prediction within the range of data is correct. We will then discuss the effect of model discrepancy in CO2 retrievals. This is joint work with Anthony O'Hagan, University of Sheffield, and Jonathan Hobbs and Amy Braverman at the Jet Propulsion Laboratory.

1. Model discrepancy and physical parameters in
calibration and prediction of computer models
Jenný Brynjarsdóttir
Joint work with Anthony O’Hagan, University of Sheffield, UK, and
Jon Hobbs and Amy Braverman, Jet Propulsion Laboratory
SAMSI Opening Workshop, August 20 - 24, 2018
Model Uncertainty: Mathematical and Statistical (MUMS)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 1 / 39

2. Introduction
SAMSI Program on Uncertainty Quantification
2011-2012
Subprograms:
Methodology, Climate Modeling, Engineering and Renewable
Energy and Geosciences
Chia, Nate, Jenny, Pierre, Andreas, Alex, and Ying
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 2 / 39

3. Introduction
Outline
Introduction
Example 1: Simple machine showing the effect of model
discrepancy on
Estimating physical parameters
Interpolation - Predicting within the scope of the data
Extrapolation
Example 2: Model discrepancy in remote sensing of CO2
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 3 / 39

4. Introduction
Uncertainty Quantification (UQ)
for deterministic science-based models
Computer Model
η(x, θ)
=
Model
Discrepancy
(MD)
Reality
ζ(x)
x: controllable inputs
θ: unknown inputs
physical parameters with
true value θ∗
tuning parameters
Obs: Zi = ζ(xi) + i
i = 1, . . . , n
i i.i.d. N(0, σ2)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 4 / 39

5. Introduction
Uncertainty Quantification (UQ)
for deterministic science-based models
Computer Model
η(x, θ)
=
Model
Discrepancy
(MD)
Reality
ζ(x)
UQ objectives include:
Calibration: estimate θ∗
Predict ζ(x)
interpolation or
extrapolation
Obs: Zi = ζ(xi) + i
i = 1, . . . , n
i i.i.d. N(0, σ2)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 4 / 39

6. Introduction
Uncertainty Quantification (UQ)
for deterministic science-based models
Computer Model
η(x, θ)
=
Model
Discrepancy
(MD)
Reality
ζ(x)
Calibration:
Zi = η(xi, θ) + i
i i.i.d. N(0, σ2)
Ignores Model discrepancy
Kennedy & O’Hagan (2001):
Zi = η(xi, θ) + δ(xi) + εi
Model δ(x) as a zero-mean
Gaussian Process
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 4 / 39

7. Confounding between Model Discrepancy and parameters
Confounding between θ and δ(·)
Jim: “Hopelessly confounded!” - “Impossible!”
KOH-approach:
Z = η(x, θ) + δ(x) +
For any θ there is a δ(x) that gives us ζ(x):
ζ(x) = η(x, θ) + δ(x)
⇒ θ and δ(x) are not identifiable.
Even if we learn ζ(x) well (large sample size), calibration only
gives a joint posterior distribution for θ and δ(·) over the manifold
Mζ = {(θ, δ(·)) : ζ(x) = η(x, θ) + δ(x); x ∈ Xobs}
Need more information ⇒ need to think carefully about priors on θ
and/or δ(x)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 5 / 39

8. Confounding between Model Discrepancy and parameters
Mission impossible?
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 6 / 39

9. Confounding between Model Discrepancy and parameters
Confounding between θ and δ(·)
KOH-approach:
ζ(x) = η(θ, x) + δ(x)
Pick any θ = t,
then set
δ(x) = ζ(x) − η(t, x)
for all x
Model discrepancy (δ)
Parameter(θ)
Prior
Posterior
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 7 / 39

10. Confounding between Model Discrepancy and parameters
Confounding between θ and δ(·)
KOH-approach:
ζ(x) = η(θ, x) + δ(x)
Pick any θ = t,
then set
δ(x) = ζ(x) − η(t, x)
for all x
Model discrepancy (δ)
Parameter(θ)
Prior
Posterior
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 7 / 39

11. Confounding between Model Discrepancy and parameters
Confounding between θ and δ(·)
KOH-approach:
ζ(x) = η(θ, x) + δ(x)
Pick any θ = t,
then set
δ(x) = ζ(x) − η(t, x)
for all x
Model discrepancy (δ)
Parameter(θ)
Prior
Posterior
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 7 / 39

12. Example 1: Simple Machine
The Simple Machine
Produces work according to the amount of effort put into it
Computer model:
η(x, θ) = θx
Does not account
for losses due to
friction, etc.
θ has physical
meaning:
efficiency in a
frictionless world
gradient at zero
true value:
θ∗
= 0.65
0 1 2 3 4 5 6
01234
x (effort)
y(work)
q
Simple Machine η(x, 0.65)
True process ζ(x)
Observations
q
q
q
q
q
q
q
q
q
q
q
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 8 / 39

13. Example 1: Simple Machine
Parameter estimation
0.45 0.55 0.65 0.75
050100150
No MD
θ
Posteriordensity
θ*
11 obs.
31 obs.
61 obs.
0.45 0.55 0.65 0.75
010203040
GP prior on MD
θ
Posteriordensity
θ*
11 obs.
31 obs.
61 obs.
0.45
010203040
Cons
Posteriordensity
a) b) c)
Posterior densities do not cover the true value of θ.
Posterior mean: 0.562. True value: θ∗
= 0.65
Ignoring model discrepancy: More observations only make us
more sure about the wrong value!
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 9 / 39

14. Example 1: Simple Machine
Confounding between θ and δ(·)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 10 / 39

15. Example 1: Simple Machine
What do we know about Model Discrepancy for our
Simple Machine?
Model does not account for friction
What does that mean for the δ(x) function?
1 The function δ(x) is very smooth
2 δ(0) = 0
3 δ(x) < 0 for all x > 0
4 δ(x) is decreasing,
δ (x) < 0 for all x > 0
5 δ (0) = 0
We want a prior for δ that reflects this
0 1 2 3 4 5 6
01234
x (effort)
y(work)
q
Simple Machine η(x, 0.65)
True process ζ(x)
Observations
q
q
q
q
q
q
q
q
q
q
q
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 11 / 39

16. Example 1: Simple Machine
Priors for the Model Discrepancy (δ)
0 1 2 3 4
−2012
ψ = 0.3
x
Realizations
0 1 2 3 4
−2012
ψ = 1
x
Realizations
0 1 2 3 4
−3−1123
ψ = 0.3
x
Realizations
0 1 2 3 4
−3−1123
ψ = 1
x
RealizationsConstrained prior:
δ(0) = 0 and δ (0) = 0,
δ (0.5) < 0 and δ (1.5) < 0
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 12 / 39

17. Example 1: Simple Machine
Parameter estimation
0.45 0.55 0.65 0.75
050100150
No MD
θ
Posteriordensity
θ*
11 obs.
31 obs.
61 obs.
0.45 0.55 0.65 0.75
010203040
GP prior on MD
θ
Posteriordensity
θ*
11 obs.
31 obs.
61 obs.
0.45 0.55 0.65 0.75
010203040
Constr. GP prior on MD
θ
Posteriordensity
θ*
11 obs.
31 obs.
61 obs.
a) b) c)
Constrained GP prior on the model discrepancy (MD):
Posterior mean is slightly biased.
Unlike before, posterior densities cover the true value of θ.
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 13 / 39

18. Example 1: Simple Machine Prediction
Prediction - Interpolation
Posterior of ζ(x0) = θ ∗ x0 + δ(x0) at x0 = 1.5
0.80 0.85 0.90 0.95
050100150
No MD
ζ(1.5)
Posteriordensity
ζ*(1.5)
11 obs.
31 obs.
61 obs.
0.80 0.85 0.90 0.95
050100150
GP prior on MD
ζ(1.5)
Posteriordensity
ζ*(1.5)
11 obs.
31 obs.
61 obs.
0.80 0.85 0.90 0.95
050100150
Constr. GP prior on MD
ζ(1.5)
Posteriordensity
ζ*(1.5)
11 obs.
31 obs.
61 obs.
a) b) c)
The interpolations get better with more data
Ignoring model discrepancy: More observations only make us
more sure about the wrong value!
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 14 / 39

19. Example 1: Simple Machine Prediction
Prediction - Extrapolation
Posterior of ζ(x0) = θ ∗ x0 + δ(x0) at x0 = 6
2.5 3.0 3.5 4.0 4.5
051015202530
No MD
ζ(6)
Posteriordensity
ζ*(6)
11 obs.
31 obs.
61 obs.
2.5 3.0 3.5 4.0 4.5
0.00.51.01.52.02.53.0
GP prior on MD
ζ(6)
Posteriordensity
ζ*(6)
11 obs.
31 obs.
61 obs.
2.5 3.0 3.5 4.0 4.5
0.00.51.01.52.02.53.0
Constr. GP prior on MD
ζ(6)
Posteriordensity
ζ*(6)
11 obs.
31 obs.
61 obs.
a) b) c)
Constrained GP prior on the model discrepancy (MD):
Posterior densities do not cover the true value of ζ(6).
Does worse than the go-to method we used before
Extrapolation is always dangerous!
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 15 / 39

20. Example 1: Simple Machine Prediction
Prediction - Extrapolation
Posterior of ζ(x0) = θ ∗ x0 + δ(x0) at x0 = 8
3 4 5 6 7
05101520
No MD
ζ(8)
Posteriordensity
11 obs.
31 obs.
61 obs.
3 4 5 6 7
0.00.51.01.5
GP prior on MD
ζ(8)
Posteriordensity
11 obs.
31 obs.
61 obs.
3 4 5 6 7
0.00.51.01.5
Constr. GP prior on MD
ζ(8)
Posteriordensity
11 obs.
31 obs.
61 obs.
a) b) c)
Constrained GP prior on the model discrepancy (MD):
Posterior densities do not cover the true value of ζ(8).
Does worse than the go-to method we used before
Extrapolation is always dangerous!
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 16 / 39

21. Example 1: Simple Machine Prediction
Why was the extrapolation worse for Constrained GP?
0 1 2 3 4 5 6
01234
x (effort)
y(work)
q
Simple Machine η(x, 0.65)
True process ζ(x)
Observations
Fitted line
qqqqqqqqqqqqq
qqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 17 / 39

22. Example 2: Orbiting Carbon Observatory - 2
Example 2: Orbiting Carbon Observatory - 2
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 18 / 39

23. Example 2: Orbiting Carbon Observatory - 2
CO2 measurements from space
q
q
qq
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
qqq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqqqq
q
q
q
qq
qq
q
q
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
qq
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qqqq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
0.760 0.765 0.770
0.000.010.020.03
Oxygen band
Radiance
qqqq
q
q
q
q
qqqqqqqqqqqq
qqqqq
qqqq
qqqqqqqqqqqq
q
qqqqqqq
q
qqq
qqqqqqqqqqqq
q
qqq
q
q
qqqqqq
qq
qqqqq
q
qqqq
q
qq
qqq
q
q
q
qq
q
qq
q
qqqq
q
q
qqqqqqqqqqq
q
q
qq
qq
q
qq
qqqqqqqq
q
q
q
qqqq
q
q
q
qqqqqqq
q
q
q
q
qq
q
q
q
q
qqq
qq
qqq
q
q
q
q
qqqq
q
qqqqqqq
q
q
q
qq
qqqq
q
qqq
qq
qqq
q
qqq
q
q
q
q
q
qqqqqqq
q
q
qqqqq
q
q
q
q
q
qqqq
q
qqqqqq
qq
q
q
q
q
q
q
q
q
qqqq
qq
q
qqqqq
qq
q
q
q
q
qqq
q
q
qqqqqq
qqqqq
q
q
q
q
qqqq
qq
q
q
qqqqqq
q
q
q
q
qq
qqq
qqq
q
qqqq
q
q
q
q
q
q
q
qq
q
qqqqq
q
qqq
q
q
q
q
q
q
q
qq
q
q
qqqqq
q
q
q
q
q
q
q
q
qq
q
qq
q
qq
q
q
q
q
q
qq
q
q
q
q
qqq
q
qqqq
q
q
q
q
q
q
q
q
q
qqqqq
q
q
q
q
q
q
q
q
q
qqqq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qqqq
qqq
q
q
q
q
q
q
q
qqqq
q
q
qqq
q
q
q
q
q
q
q
q
qqqq
qq
q
q
q
q
q
q
qq
qqq
q
q
q
qq
q
q
q
q
q
q
q
q
qqqqq
q
q
q
q
q
q
q
qq
qqq
qq
q
q
q
q
q
q
q
qq
qqqqq
q
q
q
q
q
q
q
q
qqqq
qqq
q
q
q
q
q
qq
qqqqq
q
q
q
q
q
q
q
qq
qqqqqqqqqq
q
qq
q
q
qqq
q
q
qqq
q
q
q
q
q
qq
qqqqq
q
q
q
q
q
q
q
qqqq
q
q
q
q
q
q
q
q
qq
qqq
q
q
q
q
q
q
qqq
qq
q
q
q
q
q
q
q
qqqq
q
q
q
q
q
q
qq
qqq
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
qqq
q
q
q
q
q
qqq
q
q
q
q
q
q
qqq
q
q
q
q
q
qqq
q
q
q
q
qq
q
qq
q
q
q
q
q
q
q
qq
q
q
q
qq
q
q
q
q
q
q
qqq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qqqq
q
q
q
qqqq
qq
q
qqq
q
q
qqqq
q
q
q
q
qq
q
q
qq
qqqqq
q
qqqqqqqqqqqqqq
qq
q
qqq
q
q
qqq
q
qqq
q
qqqqqqq
qq
qqqqqqqqqqqqqqq
qq
q
qqqqqqqqqqq
qqqqqqqqqqq
qq
qqqq
qqqqqqqqqqqqqqqqqqq
q
qqqqqqqqqq
qqqqqqqq
q
q
qqqqqqqq
qqq
qqqqqq
q
q
q
qqq
qqqqqq
q
qqqqqqqq
q
qq
qqq
qqqqqqqqqqqqqqq
1.590 1.595 1.600 1.605 1.610 1.615 1.620
0.0000.0150.030
Strong CO2 band
Radiance
q
qqqqqqqqqqqqqqqqqqq
q
q
q
qqqqqq
q
q
qqqqqqqqqqqqqqqq
q
qqq
qq
q
q
q
qqqqqqq
q
qqqqqqqqqqqqqqqqqqqq
qq
qqqqqqqqq
q
qqqqqqqqqq
q
q
qqqqq
q
q
q
q
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
qqqq
q
qqqqqqqq
q
q
q
q
qqqqqq
q
q
qq
q
q
qqqqqqq
q
qqqqqqqqqqq
qqqqqqqqq
q
q
qqqqq
qqq
q
q
q
q
q
q
qq
q
q
q
q
qqqq
qq
q
qqqqqqqq
q
qqqqq
q
qqqqqqqq
qq
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqqqq
q
qqqqqq
q
q
qq
q
qq
q
qqqq
qqqqqq
q
qq
qq
q
q
q
q
q
qq
q
q
q
q
q
q
qq
qq
qqqqqq
q
q
qqqqqqqqqqqq
q
qqq
qq
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qq
qqqq
q
q
qqqqqqqqqqqq
q
qqqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qqqqqqqqqqqqqqqq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
qqqqq
q
qqqq
q
qq
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
q
qq
q
q
qq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qqq
qq
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
qqqqqqqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
qqq
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
q
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
qqqqqqq
2.05 2.06 2.07 2.08
0.0000.0100.020
Weak CO2 band
wavelength (micron)
Radiance
Y = observed radiances at 3048 wavelengths
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 19 / 39

24. Example 2: Orbiting Carbon Observatory - 2
State Vector and forward model
X =













X1
...
X20
X21
...
X50
















CO2 mole fraction in
different layers of
the atmosphere



Other variables, such as:
surface pressure, aerosols
water vapor, temperature offset
albedo, chlorophyll fluorescence
Y: noisy obs. of natures transformation of atmosphere
Y = F(ν) + measurement error, ν = wavelengths
Don’t know F exactly, use a “full physics” model instead:
Y = F(ν, x) +
But F = F
QOI: Column averaged CO2 XCO2 = h X1:20
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 20 / 39

25. Forward model
The forward model FFP
Computationally feasible
Simplification of a dream model FD
Solves radiative transfer equations (integro-differential equations)
Spectrally-dependent surface and atmosphere optical properties
Cloud and aerosol single scattering optical properties
Gas absorption and scattering cross-sections
Surface reflectance (albedo)
Spectrum effects
Solar model, Fluorescence, Instrument Doppler shift
Instrument model
Spectral dispersion, Instrument line shape (ILS) function,
Polarization response
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 21 / 39

26. CO2 retrieval
Optimal Estimation
Rodgers, 2000
Find the state vector ˆx that minimizes
c = y − FFP
(x, B)
T
S−1
e y − FFP
(x, B) +(x−µa)T
S−1
a (x−µa)
using Levenberg-Marquardt algorithm
and provide an estimate of uncertainty:
ˆS = (KT
S−1
e K + S−1
a )−1
where K =
δFFP(x, B)
δx x=ˆx
Use N(ˆx, ˆS) as an estimate of the true state X
ˆxCO2 = hT ˆx1:20 and Var(XCO2) = hT ˆSCO2h
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 22 / 39

27. CO2 retrieval
Optimal estimation ≈ Bayesian Retrieval
Bayesian model:
Y|X ∼ N(FFP
(X, B), Se)
X ∼ N(µa, Sa)
Posterior density: p(x|y) ∝ exp −1
2 c where
c = y − FFP
(x, B)
T
S−1
e y − FFP
(x, B) +(x−µa)T
S−1
a (x−µa)
assuming that B, Se, µa and Sa are known.
x = posterior mode
ˆS ≈ posterior covariance matrix
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 23 / 39

28. CO2 retrieval
Model discrepancy for CO2 Retrieval
X is a physical parameter in this model:
Yi = FFP
i (X, B) + i, i = 1, 2, . . . , 3048
where
i
ind
∼ N(0, σ2
i ), X ∼ N(µa, Σa)
We know there is model discrepancy
FFP
= F
FFP
= FD
The problem with estimating physical parameters:
If model discrepancy is not accounted for, parameter estimation is
biased and over confident
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 24 / 39

29. CO2 retrieval
Current approach to model error in the OCO-2 mission
1 Calculate 3 EOFs from residuals and fit
Y(νi) = FFP
i (z, B) + Uz + i, i iid. N(0, σ2
i )
cost function for OE procedure becomes
c = (y−F(x, b)−Uz) Σ−1
ε (y−F(x, b)−Uz)+(x −µa) Σ −1
a (x −µa)
State vector now includes EOF amplitudes, x = (x , z ) .
Akin to the Kennedy & O’Hagan approach
Y(νi ) = FFP
i (X, B) + δ(νi ) + i , i iid. N(0, σ2
i )
2 Apply bias correction for XCO2 after the retrieval using ground
measurements of XCO2
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 25 / 39

30. CO2 retrieval
Bias correction for OCO-2
TCCON stations that measure XCO2 from the ground
OCO-2 in target mode over TCCON stations
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 26 / 39

31. CO2 retrieval
Model discrepancy for CO2 Retrieval
Kennedy and O’Hagan approach for OCO-2:
Y(νi) = FFP
i (X, B) + δ(νi) + i, i = 1, 2, . . . , 3048
We need a model discrepancy term that describes the actual
model discrepancy (not residuals)
Strategy
Borrow information between spatial locations
Use ground measurements to learn δ at that location
Transfer that δ to nearby locations
Row rank modeling of δ
Explore the form of model discrepancy with simulation studies
FD
, FFP
, FSURR
, FCS
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 27 / 39

32. CO2 retrieval
Strategy
*o
Elf
→
←
a.⇐*¥
Ts
Tt
Ts
i
-
-
-
e.
e.
sbwee
O
g
Q
%7
--
,←
+
+
+
c
'
→
A
oss
s
⇐***
G.
-
+
W
.
+
+
adore
of
→
nn
a.
OO
q,=
°
u
+
of
g-
1-
a
§
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 28 / 39

33. CO2 retrieval
Preliminary study
Model error in the clear-sky model due to parameter misspecification
What does the model error look like?
Simplified "clear sky" forward model, FCS
21-dim state vector, fewer parameters
Generate two soundings (without measurement error)
ytrue
= FCS
(Xtrue
, b)
yWM
= FCS
(Xtrue
, bWM
)
The difference between ytrue and yWM gives the model
discrepancy that is due to that particular misspecification of b.
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 29 / 39

34. CO2 retrieval
Model discrepancy
2.05 2.06 2.07 2.08
0.0000.0020.0040.006
Strong CO2 band
Wavelength
Difference
1.590 1.600 1.610 1.620
0.0000.0020.0040.0060.0080.010
Weak CO2 band
Wavelength
Difference
0.760 0.765 0.770
0.0000.0020.0040.0060.008
Oxygen A band
Wavelength
Difference
Model discrepancy is very spiky
⇒ Gaussian Process model for δ is not appropriate
But: The spikes in δ are at the same positions as absorption lines
Suggests basis vector model for δ:
δ = Uz
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 30 / 39

35. CO2 retrieval
Basis vectors
2.05 2.06 2.07 2.08
0.000.050.100.15
Strong CO2 band
Wavelength
1.590 1.595 1.600 1.605 1.610 1.615 1.620
0.000.050.100.15
Weak CO2 band
Wavelength
0.760 0.765 0.770
0.000.050.100.15
Oxygen A band
Wavelength
Chose ≈ 50 absorption lines for each band
Used the Laplace pdf to create basis vectors (U):
f(ν) =
1
2β
exp −
|ν − µ|
β
, ν, µ ∈ R and β > 0
µ = absorption line, β controls spread, truncated at ± 4 stdev.
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 31 / 39

36. CO2 retrieval
Preliminary study
Generated 3 true state vectors xtrue from N(µa, Σa)
Yi = FCS
(xtrue
i , btrue
) + ε i = 1, 2, 3
Location 1: Validation site
Have independent observation of XCO2
Use them to define an informative prior on X
Use a vague prior on Z
Get the posterior distribution p(Z | Y1)
Locations 2 and 3:
Vague prior on X
Prior on Z: posterior from location 1
Working model: tweaked b by 1%
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 32 / 39

37. CO2 retrieval
Retrieval
Without model discrepancy (MD) :
Y|X ∼ N(FCS
(X, bWM
), Σε)
X ∼ N(µa, Σa)
With model discrepancy:
Y|X, Z ∼ N(FCS
(X, bWM
) + UZ, Σε)
X ∼ N(µa, Σa)
Z ∼ N(µZ , ΣZ )
Vague prior: Large variances
Informative prior: Small variances
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 33 / 39

38. CO2 retrieval
Results for validation site
386 388 390 392 394
01234
Sounding 1
Xco2
Density
Informative prior on X
Non−inform. MD prior
No MD
Prior on Xco2
True value
Vague prior for Z: µZ = 0 and ΣZ = I.
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 34 / 39

39. CO2 retrieval
Model discrepancy at validation site
0 50 100 150
−0.40.00.20.4
Element of Z
q q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q q
q
q
q
q
q q q
q
q
q
q q
q
q q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q q
q
q
q
q
q q
q q q
q
q
q
q q q
q
q
q
q
q
q
q
q
q
q
q
q q
q q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q q q
q
q q
q
q
q
q
q
q
Strong CO2 band Weak CO2 band A band
90% posterior credible intervals of Z1 from this retrieval
prior 90% credible interval: (−1.645, 1.645)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 35 / 39

40. CO2 retrieval
Results for locations 2 and 3
386 388 390 392 394
01234
Sounding 1
Xco2
Density
Informative prior on X
Non−inform. MD prior
No MD
Prior on Xco2
True value
386 388 390 392 394
0.00.20.40.60.81.0
Sounding 2
Xco2
Density
Informative MD prior
Non−inform. MD prior
No MD
Prior on Xco2
True value
386 388 390 392 394
0.00.20.40.60.81.0
Sounding 3
Xco2
Density
Informative MD prior
Non−inform. MD prior
No MD
Prior on Xco2
True value
Locations 2 and 3: the posterior recovers the true value of XCO2
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 36 / 39

41. Conclusions
Discussion
Learning about model discrepancy at one location has the
potential to greatly improve retrievals in other locations.
Lots of details have to be figured out
How well does model discrepancy extrapolate across space?
Computational efficiency (choice of basis vectors)
UQ-test bed
Realistic surrogate model, FSURR
Templates of simulated (X, Y) pairs that reflect the range of physical
conditions around the globe
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 37 / 39

42. References
References
Jenný Brynjarsdóttir and Anthony O’Hagan.
Learning about physical parameters: The importance of model
discrepancy.
Inverse Problems, 30, 2014.
M. C. Kennedy and A. O’Hagan.
Bayesian calibration of computer models.
Journal of the Royal Statistical Society B, 63(Part 3):425–464, 2001.
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 38 / 39

43. Bayes In Space
Thanks!
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 39 / 39

44. Extra
Analysis without accounting for model discrepancy
To estimate the parameter we fit
zi = xiθ + i, i iid. N(0, σ2
).
Prior: π(θ, σ2) ∝ 1/σ2
Posterior: tn−1 distribution
Don’t need any MCMC
Posterior distribution does not
cover the true value of θ.
Posterior mean: 0.562
True value: 0.65
More observations only make
us more sure about the wrong
value!
0.45 0.55 0.65 0.75
050100150
No MD
θ
Posteriordensity
θ*
11 obs.
31 obs.
61 obs.
0
010203040
Posteriordensity
b)
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 40 / 39

45. Extra
Analysis with Model Discrepancy (MD)
As in Kennedy & O’Hagan (2001), we model δ(x) as a Gaussian
process:
δ(x) ∼ GP(0, σ2
c(·, ·|ψ)) where c(x1, x2|ψ) = exp −
x1 − x2
ψ
2
Bayesian model:
Z | θ, δ, σ2
∼ N Xθ + δ, σ2
I
δ | σ2
, ψ ∼ N 0, σ2
Λ(ψ)
θ, σ2
, σ2
, ψ ∼ π(θ, σ2
, σ2
, ψ)
We are interested in both θ and δ so we want to sample the posterior
distributions of both. Problem:
Full conditional of δ can have a numerically singular covariance
matrix
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 41 / 39

46. Extra
Complete Bayesian model
Our solution: Sample δ(xi) at few locations, and set the rest equal
to their conditional mean
xo = (0.2, 0.96, 1.72, 2.48, 3.24, 4.00) and δo = δ(xo)
Set δr = E (δr | δ(xo))
A complete formulation of the model:
Z|δo, θ ∼ N(Xθ + H(ψ)δo, σ2
In)
δo ∼ N 0, σ2
Λ(ψ)o,o
θ ∝ 1, σ2
∼ IG(a , b ), σ2
∼ IG(a, b),
ψ ∼ trGamma(0,4)(aψ, bψ)
where
H(ψ) =
I6
Λ(ψ)r,oΛ(ψ)−1
o,o
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 42 / 39

47. Extra
The prior distributions
p(θ) ∝ 1
σ2
∼ InvGamma mode = 0.22
, mean = 0.32
ψ ∼ TrGamma[0,4] (mean = 1, var = 0.2)
σ2
∼ InvGamma mode = 0.0092
, mean = 0.012
MCMC: Gibbs sampler with a
Metropolis-Hastings step for ψ
0 1 2 3 4 5 6
01234
x (effort)
y(work)
q
Simple Machine η(x, 0.65)
True process ζ(x)
Observations
Fitted line
qqqqqqqqqqqqq
qqqqqqqqq
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
Jenný Brynjarsdóttir (CWRU) Model discrepancy August 22, 2018 43 / 39