School of something
FACULTY OF OTHER
The treatment of uncertainty in
Alison S. Tomlin
Energy and Resources Research Institute
SPEME, Faculty of Engineering
Why track uncertainties?
• In climate modelling the current expectation is that
uncertainties will be tracked within model predictions.
• Phrases such as “high confidence”, “likely”, “more likely
than not” etc. are presented along side mean predictions
and confidence limits.
• This is critical since such predictions are to be used in
decision making and policy formulation.
• Urban air quality models are also used to help form policy
on pollution mitigation strategies.
• So why are uncertainties in such model predictions not
Hierarchy of model complexity
e.g. Gaussian, or
Types of model uncertainty
➢ Missing physical/chemical processes within the model.
➢ Over simplification of processes to save compute power.
e.g. reduced chemistry, averaged turbulence representations.
➢ Influence of grid resolution.
➢ Complex models contain large numbers of parameters
that are all uncertain.
e.g. kinetic reaction rates, roughness lengths, mixing
timescales, emissions, flux rates etc.
➢ Parametric uncertainties can be propagated through
sampling methods using multiple model runs.
➢ Structural uncertainty more difficult to assess:
the “Unknown unknowns” dilemma.
➢ Neither approach commonly used in atmospheric pollution
Can we learn from other
Climate Change community tests for structural uncertainties
via the inter-comparison of models and observations.
Could perform sensitivity
analysis of model components
Model inter-comparison for
➢ A few examples of inter-comparisons between models of
the same type e.g. Gaussian based, different RANS
➢ Few exercises that compare models of different structure to
assess their fitness for purpose.
What does fit for purpose mean?
• Within parametric uncertainties model overlaps with error
bars of measured data.
• Model can be extrapolated to new conditions (not easy for
over-fitted models) i.e. predictive.
• Other criteria? Doesn’t have to be physically realistic.
Evaluation of complex models
• Need to evaluate if models are fit for purpose.
• Comparison of model with experimental or field data for
simple to complex scenarios.
• how much confidence we can place in simulations.
• If lack of agreement then how do we find contributing
• Sensitivity and uncertainty analysis help to answer these
- need strong feedback loop between model
evaluation and methods for model improvement.
Sensitivity and uncertainty
• Uncertainty analysis (UA)
estimates the overall predictive
uncertainty of a model given the
state/or lack of knowledge about its
• UA puts error bars on predictions.
Sensitivity analysis (SA) determines
how much each input parameter
contributes to the output uncertainty
Parameter Si, x=2.2 m
Structure function coeff. C0 0.7976
Mixing time-scale coeff. α 0.1853
Σ Si 0.9843
How can uncertainties be
• The most consistent ways to trace uncertainties are
1. To run several models with potentially different structures and
2. To run each model using an ensemble or random sample of input
parameters in order to generate a distribution of predicted target
Estimated maxEstimated min
Scatter plots for urban RANS dispersion
The examples show possible nonlinearities and large scatter –
obscuring overall sensitivity to parameter.
Wind direction° (input) Wind direction° (input)
High Dimensional Model
• Developed to provide detailed mapping of the input variable space to
selected outputs – a meta-model.
• Output is expressed as a finite hierarchical function expansion:
• Meta-model built using quasi random sample and approximation of
component functions by orthonormal polynomials.
• Used to generate partial variances and therefore sensitivity indices
• Si then ranked to give parameter importance.
Component functions for
Wind direction°Surface roughness length
Closing the loop
• High ranked parameters could then be re-estimated using ab
initio modelling studies or simple experiments which help to
isolate their effects.
• If the parameter can be re-estimated with better certainty
then the error bars of the prediction are reduced.
• In some cases, even within the error bars, there is no
overlap between experimental and modelled outputs.
suggests structural problems with the model
• For this reason comparisons of model vs observations
should include error bars on BOTH!
Example: York street canyon
simulation of [NO2]
Complex model couplings
CFD flow model
Flow and turbulence
Parameter set varied in global
Model parameters (26 in total):
➢velocity structure function coefficient co
➢mixing time-scale coefficient α
➢surface roughness z0 for inlet, surface and wall
➢temperature dependant rate parameters for NO/NO2/O3 reactions,
photolysis rate parameters for JO3 and JNO2
➢wind direction θref
Sample sensitivity results
(θref= 110-130): average over 6 in-canyon
road-side locations: mean [NO2]
Summary of findings
• Based on the assumed uncertainties in input
parameters the predicted roadside increments in
[NO2] can vary by around a factor of 2.
• Using the HDMR approach the influence of
traffic related parameters e.g. demand, primary
NO2 fraction, can be assessed within the overall
• For sites away from the traffic queue the wind
direction is the single most important parameter
affecting predicted NO2.
• Model parameters such as surface roughness lengths are influential.
• Chemical parameters (for NO+O3) are important for sites near the
• Background O3 appears not to be too influential for this site.
• Propagating uncertainties within pollution dispersion models needs to
become more routine.
➢ Doesn’t come without a computational cost.
•Uncertainties can be significant e.g. causing a factor of 2 variability in
•Using global sensitivity/meta-modelling approaches it is still possible to
trace the influence of controllable parameters e.g. traffic demand on
target outputs using component functions.
•Wind speed and direction are key parameters causing variability and yet
we have very few truly urban met stations.
We can learn from other communities!
HDMR software with graphical
user interface freely available
Main author: Tilo Ziehn
• Need to be able to model formation of secondary pollutants such as NO2
in order to test the impact of emission reduction strategies.
• Computational fluid dynamics (CFD) models becoming more common in
pollution and emergency response management.
• Different turbulence closure schemes in use:
Reynolds Averaged Navier Stokes (RANS), Large Eddy Simulation
• Also different dispersion schemes:
Lagrangian particle, stochastic fields, eddy diffusivity.
• Urgent need to evaluate where different approaches are “fit for purpose”
and the impacts of parametrisations of turbulence, mixing and kinetics.
Convergence with sample size
0 20 40 60 80 100 120
Sample size in Sobol sequence
64 samples enough to
give accurate estimates
of mean and variance of
What we want to know is which of the 26 input
parameters contribute most to this predictive variance
due to their estimated uncertainties i.e.
what are the global sensitivities for each parameter?