Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Treatment of Uncertainty in Models


Published on

Alison Tomlin

Published in: Environment
  • Be the first to comment

  • Be the first to like this

The Treatment of Uncertainty in Models

  1. 1. School of something FACULTY OF OTHER The treatment of uncertainty in dispersion models Alison S. Tomlin Energy and Resources Research Institute SPEME, Faculty of Engineering
  2. 2. Why track uncertainties? • In climate modelling the current expectation is that uncertainties will be tracked within model predictions. • Phrases such as “high confidence”, “likely”, “more likely than not” etc. are presented along side mean predictions and confidence limits. • This is critical since such predictions are to be used in decision making and policy formulation. • Urban air quality models are also used to help form policy on pollution mitigation strategies. • So why are uncertainties in such model predictions not routinely traced??
  3. 3. Hierarchy of model complexity Direct numerical simulation with detailed chemistry. Computationally expensive Detailedchemicalandphysical representations Large Eddy Simulation Reynolds Averaged CFD + Lagrangian Semi-empirical e.g. Gaussian, or network models
  4. 4. Types of model uncertainty Structural uncertainty: ➢ Missing physical/chemical processes within the model. ➢ Over simplification of processes to save compute power. e.g. reduced chemistry, averaged turbulence representations. ➢ Influence of grid resolution. Parametric uncertainty: ➢ Complex models contain large numbers of parameters that are all uncertain. e.g. kinetic reaction rates, roughness lengths, mixing timescales, emissions, flux rates etc.
  5. 5. Propagating Uncertainties ➢ Parametric uncertainties can be propagated through sampling methods using multiple model runs. ➢ Structural uncertainty more difficult to assess: the “Unknown unknowns” dilemma. ➢ Neither approach commonly used in atmospheric pollution dispersion models!
  6. 6. Can we learn from other communities? Climate Change community tests for structural uncertainties via the inter-comparison of models and observations.
  7. 7. Could perform sensitivity analysis of model components
  8. 8. Examples of process sensitivity analysis
  9. 9. Model inter-comparison for dispersion models? ➢ A few examples of inter-comparisons between models of the same type e.g. Gaussian based, different RANS models. ➢ Few exercises that compare models of different structure to assess their fitness for purpose. What does fit for purpose mean? • Within parametric uncertainties model overlaps with error bars of measured data. • Model can be extrapolated to new conditions (not easy for over-fitted models) i.e. predictive. • Other criteria? Doesn’t have to be physically realistic.
  10. 10. Evaluation of complex models • Need to evaluate if models are fit for purpose. • Comparison of model with experimental or field data for simple to complex scenarios. MUST INCLUDE • how much confidence we can place in simulations. • If lack of agreement then how do we find contributing causes? • Sensitivity and uncertainty analysis help to answer these questions: - need strong feedback loop between model evaluation and methods for model improvement.
  11. 11. Sensitivity and uncertainty analysis • Uncertainty analysis (UA) estimates the overall predictive uncertainty of a model given the state/or lack of knowledge about its input parameters. • UA puts error bars on predictions. Sensitivity analysis (SA) determines how much each input parameter contributes to the output uncertainty (usually variance). Parameter Si, x=2.2 m Structure function coeff. C0 0.7976 Mixing time-scale coeff. α 0.1853 Σ Si 0.9843
  12. 12. How can uncertainties be traced? • The most consistent ways to trace uncertainties are 1. To run several models with potentially different structures and parameterisations. 2. To run each model using an ensemble or random sample of input parameters in order to generate a distribution of predicted target outputs. Parameter 1 Parameter2 Estimated maxEstimated min
  13. 13. Scatter plots for urban RANS dispersion model The examples show possible nonlinearities and large scatter – obscuring overall sensitivity to parameter. Wind direction° (input) Wind direction° (input) Verticalvelocityinstreet(output) Rooflevelturbulence(output)
  14. 14. High Dimensional Model Representations (HDMR) • Developed to provide detailed mapping of the input variable space to selected outputs – a meta-model. • Output is expressed as a finite hierarchical function expansion: • Meta-model built using quasi random sample and approximation of component functions by orthonormal polynomials. • Used to generate partial variances and therefore sensitivity indices • Si then ranked to give parameter importance. )x,...,x,xx,xf)(xfff n21 nji jiij n i ii    1 12...n 1 0 (f...)()(x
  15. 15. Component functions for previous example Wind direction° Verticalvelocityinstreet Rooflevelturbulence Wind direction° Verticalvelocity Wind direction°Surface roughness length
  16. 16. Closing the loop • High ranked parameters could then be re-estimated using ab initio modelling studies or simple experiments which help to isolate their effects. • If the parameter can be re-estimated with better certainty then the error bars of the prediction are reduced. • In some cases, even within the error bars, there is no overlap between experimental and modelled outputs. suggests structural problems with the model • For this reason comparisons of model vs observations should include error bars on BOTH!
  17. 17. Example: York street canyon simulation of [NO2] Signal controlled intersection TKE, wind vectors for θref=90° to canyon axis
  18. 18. Complex model couplings Traffic micro-simulation model Vehicle Characteristics Traffic Network Information Instantaneous emissions model Vehicle speeds Vehicle acceleration CFD flow model (MISKAM k-ε) Meteorology Building layouts Emissions Flow and turbulence Dispersion model Pollution concentrations
  19. 19. Parameter set varied in global sensitivity study Model parameters (26 in total): ➢velocity structure function coefficient co ➢mixing time-scale coefficient α ➢surface roughness z0 for inlet, surface and wall ➢temperature dependant rate parameters for NO/NO2/O3 reactions, photolysis rate parameters for JO3 and JNO2 ➢wind direction θref ➢temperature ➢background [O3] ➢NO:NOx ratio ➢traffic demand Model parametrisation Physical/meteorological Traffic emissions
  20. 20. Sample sensitivity results (θref= 110-130): average over 6 in-canyon road-side locations: mean [NO2] 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 Averagesensitivityindex Off-Peak Peak
  21. 21. Summary of findings • Based on the assumed uncertainties in input parameters the predicted roadside increments in [NO2] can vary by around a factor of 2. • Using the HDMR approach the influence of traffic related parameters e.g. demand, primary NO2 fraction, can be assessed within the overall data scatter. • For sites away from the traffic queue the wind direction is the single most important parameter affecting predicted NO2. • Model parameters such as surface roughness lengths are influential. • Chemical parameters (for NO+O3) are important for sites near the junction. • Background O3 appears not to be too influential for this site.
  22. 22. Conclusions • Propagating uncertainties within pollution dispersion models needs to become more routine. ➢ Doesn’t come without a computational cost. •Uncertainties can be significant e.g. causing a factor of 2 variability in predicted concentrations. •Using global sensitivity/meta-modelling approaches it is still possible to trace the influence of controllable parameters e.g. traffic demand on target outputs using component functions. •Wind speed and direction are key parameters causing variability and yet we have very few truly urban met stations. We can learn from other communities!
  23. 23. HDMR software with graphical user interface freely available Main author: Tilo Ziehn
  24. 24. Example • Need to be able to model formation of secondary pollutants such as NO2 in order to test the impact of emission reduction strategies. • Computational fluid dynamics (CFD) models becoming more common in pollution and emergency response management. • Different turbulence closure schemes in use: Reynolds Averaged Navier Stokes (RANS), Large Eddy Simulation (LES). • Also different dispersion schemes: Lagrangian particle, stochastic fields, eddy diffusivity. • Urgent need to evaluate where different approaches are “fit for purpose” and the impacts of parametrisations of turbulence, mixing and kinetics.
  25. 25. Convergence with sample size 0.00E+00 1.00E+11 2.00E+11 3.00E+11 4.00E+11 5.00E+11 6.00E+11 7.00E+11 8.00E+11 9.00E+11 0 20 40 60 80 100 120 NO2concentration(moleculescm-3) Sample size in Sobol sequence Mean Variance 64 samples enough to give accurate estimates of mean and variance of predicted [NO2] What we want to know is which of the 26 input parameters contribute most to this predictive variance due to their estimated uncertainties i.e. what are the global sensitivities for each parameter?