This talk was presented as part of the panel discussion on Day 2 of the MUMS Opening Workshop.

1. Calibration in the Face of Model Discrepancy:
...and in the face discontinuity or local features
Geogios Karagiannis
University of Durham
August 21, 2018
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 1 / 17

2. Introduction to the problem
The problem set-up
Real complex system Z :
x
ζpxq
ùñ y for x P X and y P R
Computer model (simulator) S :
x
Spx,tq
ùñ η for px, tq P X ˆ Θ and y P R
Interest, learn:
parameter θ P Θ, s.t. Spx, θq « ζpxq, @x P X
emulator ζpxq|y, η, x, .. @x P X
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 2 / 17

3. Introduction to the problem
Kennedy & O’Hagan SBC method
Assume observables tpxi , yi qui and tpxi , ti , ηi qui from model
y “ ζpxq ` y ; y „ Np0, σ2
y q
η “ Spx, tq ` η ; η „ Np0, σ2
ηq
Link Z and S as
Dθ P Θ, ζpxq “ Spx, θq ` δpxq
Specify priors
Sp¨, ¨q „ GPpµS p¨|βS q, cS p¨, ¨|ϕS qq;
δp¨q „ GPpµδp¨|βδq, cδp¨, ¨|ϕδqq; pβ, ϕq „ πpdβ, dϕq
θ „ πpdθq
[Kennedy and O’Hagan, 2001]
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 3 / 17

4. Introduction to the problem
Issue in the fully Bayesian approach:
Bayesian inference is actually performed on
pMζ “ tθ, δp¨qu : ζpxq “ Spx, θq ` δpxq for all x P Xq
regardless the size of the available data
[Brynjarsdóttir and O’Hagan, 2014]
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 4 / 17

5. Introduction to the problem
Issue in the fully Bayesian approach:
Bayesian inference is actually performed on
pMζ “ tθ, δp¨qu : ζpxq “ Spx, θq ` δpxq for all x P Xq
regardless the size of the available data
[Brynjarsdóttir and O’Hagan, 2014]
In more realistic scenarios:
1 Scale discrepancy:
ζpxq “ ρpxq Spx, θq ` δpxq
2 Even more general like :
ζpxq “ Gp Spx, θq, θq
3 etc...
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 4 / 17

6. Model calibration in the presence of discontinuities
Application: AX Cold Flow
Bayesian treed model calibration [Konomi et al., 2017]
Outputs d : Pressure drop PDT3820.
Experimental Inputs x: Gas Velocity (GV ), [15, 60] SLPM
CFD Model parameters t:
θ1 : Coef. of restitution, particle-particle (Res.PP), θ1 „ 1.997Bp2.5, 2.5q `
θ2 : Coef. of restitution, particle-wall (Res.PW ), θ2 „ 1.997Bp2.5, 2.5q ` 0.8
θ3 : Frict. angle, particle-particle (FA.PP), θ3 „ 20Bp1.2, 2.5q ` 25
θ4 : Frict. angle, particle-wall (FA.PW ), θ4 „ 20Bp1.2, 2.5q+25
θ5 : Packed bed void fraction (PBVF), θ5 „ 0.1Bp2.5, 2.5q ` 0.3
θ6 : Particle size (PSize), θ6 „ 20Bp1.2, 2.5q ` 105.0
Table: Summary of inputs, outputs, and the CFD model parameters with their
corresponding prior using the AX sorbent
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 5 / 17

7. Model calibration in the presence of discontinuities
Application: AX Cold Flow
We can see local features, difficult to be represented accurately by
stationary GP models
Gas Velocity
15 20 25 30 35 40 45 50 55 60
PressureDrop
800
1000
1200
1400
1600
1800
Experimental
Simulation
They can be represented by more complex GP structures, but we must
fuind the suitable one...
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 6 / 17

8. Model calibration in the presence of discontinuities
How we addressed that?
We partitioned the input & calibration space via a binary space
partition P :“ PpT q[Chipman et al., 1998, Gramacy and Lee, 2008]
D3
D1
D2
s
2
θ
s1
t
x
{x, s1}
{t, s2}
{D1, d1}
D[:, t] < s2
{D2, d2}
D[:, t] > s2
D[:, x] < s1
{D3, d3}
D[:, x] > s1
...and we fit basic GP models
f pd|T , φ, θq “
Kź
k“1
Npdk|µkpφk, θq, Vkpφk, θqq
πpT , φ1:k, θq “ πpT qπpφ1:k, θ|T q
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 7 / 17

9. Model calibration in the presence of discontinuities
Application: the emulator
The resulting emulator ...
15 20 25 30 35 40 45 50 55 60
900
1000
1100
1200
1300
1400
1500
1600
1700
Gas Velocity
PDT3820
(a) Bayes treed Calibr.
15 20 25 30 35 40 45 50 55 60
900
1000
1100
1200
1300
1400
1500
1600
1700
Gas Velocity
PDT3820 (b) Standard Bayes Calibr.
... but not that interested in this here...
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 8 / 17

10. Model calibration in the presence of discontinuities
Application: Discrepancy representation?
We represent the discrepancy function
Gas Velocity
15 20 25 30 35 40 45 50 55 60
MeanDiscrepancy
-200
-150
-100
-50
0
(c) Proposed Bayes Calibr.
Gas Velocity
15 20 25 30 35 40 45 50 55 60
MeanDiscrepancy
-200
-150
-100
-50
0
50
(d) Standard Bayes Calibr.
We can produce more accurate estimation of the discrepancy function
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 9 / 17

11. Model calibration in the presence of discontinuities
Application: Calibration?
0.3
0.32
0.34
0.36
0.38
0.4
02040
θ5
Pr
0 0.5 1 1.5 2
x 10
4iteration
(e) Bayes treed Calibr.
0.3
0.32
0.34
0.36
0.38
0.4
01020
θ5
Pr
0 0.5 1 1.5 2
x 10
4iteration
(f) Standard Bayes Calibr.
20 30 40 50 60
0.3
0.32
0.34
0.36
0.38
0.4
x
1
t
5
1
1.2
1.4
1.6
(g) Bayes treed Calibr.
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 10 / 17

12. Model calibration in the presence of discontinuities
Application: Calibration?
105
110
115
120
125
130
135
00.20.4
θ6
Pr
0 0.5 1 1.5 2
x 10
4iteration
(h) Bayes treed Calibr.
105
110
115
120
125
130
135
00.10.2
θ6
Pr
0 0.5 1 1.5 2
x 10
4iteration
(i) Standard Bayes Calibr.
20 30 40 50 60
105
110
115
120
125
130
135
x
1
t
6
1
1.2
1.4
1.6
(j) Bayes treed Calibr.
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 11 / 17

13. Model calibration in the presence of discontinuities
Application: Speculations & observations
Possibly
In the cases that there is discontinuity, or local features
BTC might be able to mitigate the non-identifiability issue
Why?
...because it can model the discrepancy more ‘accurately’
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 12 / 17

14. Some remedies to non-identifiability
Some remedies to non-identifiability
Use subjective strongly informative priors
a luxury we do not always have.
Perform modular Bayesian model calibration:
[Liu et al., 2009, Arendt et al., 2012]
learning separately each component
Spx, tq from simulation data
δpxq from experimental data
θ from experimental data
Modularization within full Bayesian analysis? [Just an idea]
modelling together real system & comp. model
avoiding using the same data 2 times
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 13 / 17

15. Some remedies to non-identifiability
Modularization within full Bayesian analysis
Split the data-set in two sets; set A & set B
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 14 / 17

16. Some remedies to non-identifiability
Modularization within full Bayesian analysis
Split the data-set in two sets; set A & set B
Use Set A to create informative priors by modularisation:
tpxA
i , tA
i , ηA
i qui úùñ perform Bayes analysis and get
φS
|ηA
„ πpφS
|ηA
q
{Spx, tq „ ppSx,t |ηA
q
tpxA
i , yA
i qui úùñ perform Bayes analysis and get
yδpxq “ yζpxq ´ {Spx, θ˚q, and yζpxq „ ppζx |yA
q,
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 14 / 17

17. Some remedies to non-identifiability
Modularization within full Bayesian analysis
Use Set B tpxB
i , tB
i , ηB
i qui & tpxB
i , yB
i qui to perform fully Bayesian
analysis as in K&O given the prior information specified from Set A:
parameter est. φS , φζ „ πpφS , φζ|ηA, ηB, yA, yBq
calibration: θ „ πpθ|ηA
, ηB
, yA
, yB
q
prediction: yζpxq „ ppζx |ηA
, ηB
, yA
, yB
q,
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 15 / 17

18. Conclusions
Conclusions
To sum-up
When discrepancy exists it may be beneficial to be ‘detected’ and be
taken into account
Bayesian treed model calibration procedure might can mitigate
non-identifiability ... in the face of discontinuity or local features
Modularization within full Bayesian analysis may be an interesting
direction for further investigation...
Thanks
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 16 / 17

19. Conclusions
References
Paul D Arendt, Daniel W Apley, and Wei Chen. Quantification of model
uncertainty: Calibration, model discrepancy, and identifiability. Journal
of Mechanical Design, 134(10):100908, 2012.
Jenný Brynjarsdóttir and Anthony O’Hagan. Learning about physical
parameters: The importance of model discrepancy. Inverse Problems, 30
(11):114007, 2014.
Hugh A Chipman, Edward I George, and Robert E McCulloch. Bayesian
cart model search. Journal of the American Statistical Association, 93
(443):935–948, 1998.
Robert B Gramacy and Herbert K H Lee. Bayesian treed gaussian process
models with an application to computer modeling. Journal of the
American Statistical Association, 103(483):1119–1130, 2008.
Marc C Kennedy and Anthony O’Hagan. Bayesian calibration of computer
models. Journal of the Royal Statistical Society: Series B (Statistical
Methodology), 63(3):425–464, 2001.
Bledar A Konomi, Georgios Karagiannis, Kevin Lai, and Guang Lin.
Geogios Karagiannis University of DurhamCalibration in the Face of Model Discrepancy: ...and in the face discontinuity or loAugust 21, 2018 17 / 17