SlideShare a Scribd company logo

Comparing Goodness-of-fit Measures for Calibration of Models Focused on Extreme Events (EGU 2012)

Hydrological models have been increasingly used during the last decades for flood forecasting and water resources management, among others. Therefore, a proper assessment of the reliability of hydrological simulations is of utmost importance to enhance societal confidence on model predictions. To date, several goodness-of-fit measures have been proposed to assess model performance and to measure the agreement between observations and simulated equivalents. Despite serious and well-known limitations, many single-objective goodness-of-fit measures are still of widespread use. As an example, the Nash-Sutcliffe efficiency (NSE) has been highly criticised as an inappropriate benchmark for comparing modelling results to observations, nonetheless, it is still one of the most common performance measures used by both practitioners and environmental scientists. This work examines how different goodness-of-fit measures reported in literature behave when used within a single-objective optimisation procedure, both for the identification of the model’s parameters and in the reproduction of high- and low-flow events. By doing so, several parameters of the semi-distributed Soil and Water Assessment Tool (SWAT) 2005 are calibrated, by using a novel global optimisation technique called Particle Swarm Optimization (PSO) and changing only the goodness-of-fit measure to be optimised. In particular, the Kling-Gupta efficiency (KGE), the volumetric efficiency (VE) the NSE, the index of agreement (d), along with modified and relative versions of the last two measures are compared for a calibration of SWAT aiming at reproducing low flows. On the other hand, KGE, NSE, d, and the coefficient of persistence (cp) are compared for a calibration aiming at reproducing high flows. In addition, a relatively new goodness-of-fit measure is introduced, which, when used along with some ad-hoc weighting scheme, allows capturing model errors in the high or low spectrum of the analysed time series with little influence of errors in other portions of the signal. Optimal parameter values presented a considerable variation depending on the objective function used in PSO. Discharge values obtained during calibration are used to compute Empirical Cumulative Distribution Functions (ECDFs) for three different quantiles representative of simulated low, medium and high flows. Simulated quantiles computed with the new goodness-of-fit function in combination with the ad-hoc weighting scheme were closer to their observed counterparts. Results provide quantitative guidance about the bias of the calibrated hydrological model to reproduce low and high flows when different well-known goodness-of-fit measures are used as objective function during calibration. The latter facilitates the elaboration of standards about which benchmarks to use when trying to represent different extreme events.

1 of 1
Download to read offline
Joint
        Research
                                                                              Comparing Goodness-of-fit Measures for Calibration
                                                                              Comparing Goodness-of-fit Measures for Calibration                                                                                                                                                                                                                                                                  of
                                                                                                                                                                                                                                                                                                                                                                                                  of
         Centre
     EGU2012-11549
     Session: HS1.3
                                                                                      Models Focused on Extreme Events
                                                                                      Models Focused on Extreme Events
      Apr 23th, 2012                                                                                                                                    Mauricio Zambrano-Bigiarini and Alberto Bellin
                                                                                                                                                        Mauricio Zambrano-Bigiarini and Alberto Bellin
  1) Motivation                                                                                                                                                                                                                                                                                                                                             4) Results
  Despite serious and well-known limitations ( e.g., Legates and
  McCabe, 1999), many single objective goodness-of-fit measures
  are still of widespread use. As an example, the Nash-Sutcliffe                                     3.1) Nash-Sutcliffe efficiency : 3.6) Relative Nash-Sutcliffe efficiency :                                                Nomenclature
  efficiency (NSE) has been highly criticised as an inappropriate                                          (Nash and Sutcliffe, 1970)
                                                                                                                             N
                                                                                                                                                              (Krausse et al., 2005)                                           ●   Si : i-th simulated value
                                                                                                                                                                                                               2
  benchmark for comparing modelling results to observations                                                                                                                              N
                                                                                                                                                                                                     Oi−S i
                                                                                                                                                                                                 (            )
                                                                                                                                            2
                                                                                                                         ∑ ( O i − Si )                                              ∑                                         ●   Oi : i-th observed value
  (e.g., Schaefli and Gupta 2007), nonetheless, it is still one of                                        NSE =1−        i =1
                                                                                                                                                                                     i =1             Oi
                                                                                                                           N
                                                                                                                                            2                        rNSE =1−                                                  ●   j : arbitrary power, i.e, positive integer
  the most common performance measures used by both                                                                      ∑ ( O i− O )                                                    N
                                                                                                                                                                                                     Oi− O     2


  environmental scientists and practitioners.
                                                                                                                         i =1                                                        ∑
                                                                                                                                                                                     i =1
                                                                                                                                                                                                 (     O      )                ●


                                                                                                                                                                                                                               ●
                                                                                                                                                                                                                                   Ō : mean observed value
                                                                                                                                                                                                                                   Ôi : median of the observed values in
                                                                                                     3.2) Index of Agreement:                             3.7) Relative Index of Agreement:                                            the same month than Oi
  2) Aim                                                                                                   (Willmot, 1981)
                                                                                                                                 N
                                                                                                                                                               (Krausse et al., 2005)
                                                                                                                                                                                             N                     2
                                                                                                                                                                                                                               ●     r     : Pearson's product-moment
                                                                                                                                                                                                         Oi− S i
                                                                                                                                                                                                     (             )
                                                                                                                                                2
                                                                                                                             ∑ ( O i − Si )                                                                                              correlation coefficient
  To provide practical guidance about how different goodness-                                                                                                                            ∑
                                                                                                          d =1−               i =1                                                                        Oi                        α : ratio between the standard
  of-fit measures reported in literature perform when used within                                                                                                                         i =1                                 ●
                                                                                                                   N
                                                                                                                                                    2
                                                                                                                                                                     rd =1−    N                                           2             deviation of simulations (σs) and
                                                                                                                  ∑ (∣Si− O∣+∣O i− O∣)                                                   ∣Si − O∣+ ∣Oi −O∣
  a single-objective optimisation procedure, both for the
  identification of model parameters and in the reproduction of
                                                                                                                  i =1                                                        ∑
                                                                                                                                                                              i =1
                                                                                                                                                                                     (           O                     )                 observations (σo)
                                                                                                                                                                                                                               ●   β : ratio between the mean of the
  high- and low-flow events.                                                                         3.3) Coefficient of Persistence: 3.8) Modified Nash-Sutcliffe efficiency :                                                       simulations (μs) and observations
                                                                                                           (Kitanidis & Bras, 1980)                           (Legates and McCabe, 1999)                                              (μo)
                                                                                                                         N                                                                   N
                                                                                                                                        2

  3) Methodology                                                                                                       ∑ ( S i− O i )                                                    ∑ ∣Oi −Si∣j                           ●   ωi : weight in [0,1] applied to both
                                                                                                                    i =2
                                                                                                          cp =1−    N                                                mNSE =1− i =1                                                    observed and simulated values at                                                                                                                            Fig 08. Boxplots summarizing the 5th and 50th
  3.1) Study Area
                                                                                                                                                                                N
                                                                                                                                                                                                                                      time step i
                                                                                                                   ∑ ( O i− Oi−1 )2                                                      ∑ ∣Oi −O∣j                                                                                                                                      Fig 07. Boxplots summarizing the parameter values
                                                                                                                                                                                                                                                                                                                                                                                                  percentiles of daily discharge obtained with calibrations
                                                                                                                   i =2                                                                  i =1                                  ●   λ : number in [0, 1] representing the                                                                                                                          focused on low flows. Only the best half of the total
                                                                                                                                                                                                                                       weight given to the high-flow part                                                                obtained with calibrations focused on low flows. Only
                                                                                                                                                                                                                                                                                                                                                                                                  parameter sets obtained during calibration are
                                                                                                     3.4) Volumetric Efficiency:                          3.9) Modified Index of Agreement :                                           of the signal, with λ close to 1                                                                  the best half of the total parameter sets obtained
                                                                                                                                                                                                                                                                                                                                                                                                  considered for each calibration exercise. Grey vertical
                                                                                                           (Criss and Winston, 2008)                          (Legates and McCabe, 1999)                                               when focusing on high-flow                                                                        during calibration are considered for each calibration
                                                                                                                                                                                                                                                                                Fig 05. Boxplots summarizing the parameter values                                                                 lines indicate the value of the observed 5th and 50th
                                                                                                                                                                                          N                                            events, and λ close to zero when                                                                  exercise.
                                                                                                                         N                                                                                                                                                      obtained with calibrations focused on high flows. Only                                                            percentiles.
                                                                                                                       ∑ ∣Oi −Si∣                                                        ∑ ( O i− S i ) j                              focusing in low-flow conditions
                                                                                                                                                                                                                                                                                the best half of the total parameter sets obtained
                                                                                                                                                                                         i =1
                                                                                                          VE =1− i=1 N                                               d j=1−                                                        OL ,OH : user-defined thresholds used        during calibration are considered for each calibration
                                                                                                                                                                                                                                                                                                                                                    8) Conclusions
                                                                                                                                                                                                                               ●
                                                                                                                                                                               N
                                                                                                                                                                                                                       j                                                        exercise.
                                                                                                                             ∑ Oi                                             ∑ (∣Si −O∣+ ∣Oi −O∣)                                   to separate low and high values,
                                                                                                                                                                                                                                     respectively. In order to avoid
                                                                                                                             i =1                                             i =1
                                                                                                                                                                                                                                     subjectivity in the selection of OL                                                                        ●
                                                                                                                                                                                                                                                                                                                                                    Large underestimation of observed low flows (~40%) were
                                                                                                     3.5) Kling-Gupta efficiency:                         3.10) Weighted Seasonal Nash-Sutcliffe:                                    and OH , we use the flow duration                                                                              obtained with simulated values calibrated by using NSE and the
                                                                                                                                                                (this work)                  N
                                                                                                            (Gupta et al., 2009)                                                                                                     curve criterion proposed by Yilmaz
                                                                                                                                                                                         ∑ ∣ω i ( Oi −Si )∣j                         et al., 2008                                                                                                   KGE. Such underestimation is commonly masked out by the
                                                                                                       KGE =1− √( r −1)2 + (α −1)2+ (β−1)2                           wsNSE =1− i=1
                                                                                                                                                                                N                                                                                                                                                                   good overall fit in terms of NSE, KGE and other statistics.
                                                                                                                                                                                         ∑ ∣ω i (O i− Oi )∣j
                                                                                                                                                                                                      ̂
                                                                                                                 Cov s, o σs     μs                                                      i =1

                                                                                                              r = σ σ α = σ o β= μ o
                                                                                                                                                                                                                                                                                                                                                    Low-flows calibrations:

                                                                                                                                                                 {                                                             }
                                                                                                                   s o                                                          λ             ,    O i⩾OH

                                                                                                     3.6) Seasonal Nash-Sutcliffe:                           ω i = (1−λ )+
                                                                                                                                                                           (2 λ −1)( Oi− OL )
                                                                                                                                                                                               , OL < Oi < O H
                                                                                                                                                                                                                                                                                                                                                ●
                                                                                                                                                                                                                                                                                                                                                    rNSE and wsNSE (j=1, λ=0) perform the best (in terms of 5th
                                                                                                                                                                                O H −OL
                                                                                                           (Adapted from Garrick et al., 1978)
                                                                                                                                                                              1−λ             ,    O i⩾OL
                                                                                                                                                                                                                                                                                                                                                    percentile), and both of them provide also a good
                                                                                                                                 N

                                                                                                                             ∑ ∣Oi −Si∣j                                                                                                                                                                                                            representation of medium flows (50th percentile) with all the
                                                                                                          sNSE =1− iN
                                                                                                                    =1
                                                                                                                                                                                                                                                                                                                                                    other measures overestimating them.
                                                                                                                             ∑ ∣O i−O i∣j
                                                                                                                                    ̂                                                                                                                                                                                                           ●
                                                                                                                                                                                                                                                                                                                                                    NSE, KGE and d tend to underestimate GW_DELAY in
                                                                                                                             i =1

                                                                                                                                                                                                                                                                                                                                                    comparison to to the other goodness-of-fit measures.
Fig 01. Location of the Ega River Basin, meteorological stations, and discharge station used for
the calibration of the upper catchment.
                                                                                                                                                                                                                                                                                                                                                    High-flows calibrations:
  3.2) Calibration Procedure                                                                                                                                                                                                                                                                                                                    ●
                                                                                                                                                                                                                                                                                                                                                    wsNSE (j=2, λ=0.95), d and KGE perform the best (in terms of
  The Soil and Water Assessment Tool (SWAT) version 2005                                                                                                                                                                                                                                                                                            95th percentile). At the same time, only wsNSE provides a good
  was calibrated for the period Jan/1961-Dec/1970, using the                                                                                                                                                                                                                                                                                        representation of medium flows (50th percentile), with all the
  first year as warming up period.                                                                                                                                                                                                                                                                                                                  other goodness-of-fit measures overestimating them.
  A set of 9 parameters was selected for calibration:                                                                                                                                                                                                                                                                                           ●
                                                                                                                                                                                                                                                                                                                                                    The optimal value and variability of parameters related to the
                         Parameter                                         Min       Max             Fig 02. Discharge time series corresponding to the outlet of the upper part of the Ega River Basin (Ega en Estella stream gauge, Q071).                                                                                                        slow response of the catchment (GW_DELAY and ALPHA_BF)
 Base flow alpha factor [days]                            ALPHA_BF 1.00E-1        9.90E-1            Horizontal blue and red lines show the discharge values used to separate high and low flows, respectively (see Fig 03).
 Manning's “n” value for the main channel [-]             CH_N2        1.60E-2    1.50E-1
                                                                                                                                                                                                                                                                                                                                                    was very similar among all the goodness-of-fit measures tested
                                                                                                                                                                                                                                                                                   Fig 06. Boxplots summarizing the 50th and 95th
 Initial SCS CN II value [-]                              CN2          4.00E+1    9.50E+1                                                                                                                                                                                          percentiles of daily discharge obtained with                     (both close to zero), with KGE and wsNSE presenting the
 Saturated hydraulic conductivity [mm/hr]                 SOL_K        1.00E-3    1.00E+3                                                                                                                                                                                          calibrations focused on high flows. Only the best half           largest spread. However, those values are very different from
                                                                                                                                                                                                                                                                                   of the total parameter sets obtained during calibration
 Available water capacity, [mmH2O/mm soil]                SOL_AWC      1.00E-2    3.50E-1
                                                                                                                                                                                                                                                                                   are considered for each calibration exercise. Grey               the ones obtained during the calibration focused on low flows.
 Effective hydraulic conductivity in main channel [mm/hr] CH_K2        0.00E+0    2.00E+2                                                                                                                                                                                          vertical lines indicate the value of the observed 5th
 Soil evaporation compensation factor [-]                 ESCO         1.00E-2    1.00E+0                                                                                                                                                                                          and 50th percentiles.
 Surface runoff lag time [days]                           SURLAG       1.00E+0    1.20E+1                                                                                                                                                                                            References:
 Snowfall temperature [°C]                                SFTMP        -5.00E+0 5.00E+0                                                                                                                                                                                              ●
                                                                                                                                                                                                                                                                                         Criss, R., Winston, W., 2008. Do Nash values have value? Discussion and alternate proposals. Hydrological Processes 22, 2723–2725
  Calibration was carried out using Particle Swarm Optimisation                                                                                                                                                                                                                      ●
                                                                                                                                                                                                                                                                                         Garrick, M., Cunnane, C., Nash, J.E., 1978. A criterion of efficiency for rainfall-runoff models. Journal of Hydrology 36, 375–381
                                                                                                   Fig 03. Daily flow duration curve corresponding to                                                                                                                                    Kennedy, J., and R. Eberhart (1995), Particle swarm optimization, in Proceedings IEEE International Conference on Neural Networks, 1995, vol. 4, pp. 1942–1948,
  (PSO, Kennedy and Eberhart, 1995),    with 20 particles, 300
                                                                                                                                                                                                                                                                                     ●


                                                                                                   the outlet of the upper part of the Ega River Basin           Fig 04. Weighting values used in wsNSE, which take into account the skewness of the
                                                                                                                                                                                                                                                                                     ●
                                                                                                                                                                                                                                                                                         Kitanidis, P.K., Bras, R.L., 1980. Real-time forecasting with a conceptual hydrologic model 2. applications and results. Water Resources Research 16, 1034–1044.
  iterations, ω=1/(2log2), c1=c2=0.5+log2, linearly decreasing                                     (Q071). Verticl black lines show the discharge                observed daily discharges. Red and blue lines correspond to the weights used when                                   ●
                                                                                                                                                                                                                                                                                         Krause, P., Boyle, D., Bäse, F., 2005. Comparison of different efficiency criteria for hydrological model assessment. Advances in Geosciences 5, 89–97..
  Vmax from 1.0 to 0.5, and random topology (see EGU2012-10950                                     values used to separate high and low flows                    focusing on low- and high-flow events, respectively. On the left panel, the horizontal axis                         ●
                                                                                                                                                                                                                                                                                         Legates, D., McCabe Jr., G., 1999. Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resources Research 35,
                                                                                                   following Yilmaz et. al., 2008)                               represent discharges values of the Ega River Basin (Q071), while on the right panel the                                 233–241
  for further details).
                                                                                                                                                                                                                                                                                         Schaefli, B., Gupta, H., 2007. Do Nash values have value?. Hydrological Processes 21, 2075–2080.


www.jrc.europa.eu
                                                                                               Mauricio Zambrano-Bigiarini                                       horizontal axis represent the empirical CDF of the observed discharge values.                                       ●

                                                                                                                                                                                                                                                                                     ●
                                                                                                                                                                                                                                                                                         Willmott, C., 1981. On the validation of models. Physical Geography 2, 184–194.
                                                                                               European Commission • Joint Research Centre • Institute for Environment and Sustainability                                                                                            ●
                                                                                                                                                                                                                                                                                         Yilmaz, K., Gupta, H., Wagener, T., 2008. A process-based diagnostic approach to model evaluation: Application to the NWS distributed hydrologic model. Water
                                                                                               Tel. +39 0332 789588 • Email: mauricio.zambrano@jrc.ec.europa.eu                                                                                                                          Resources Research 44, W09417.
Ad

Recommended

3 d-test of hypothesis
3 d-test of hypothesis3 d-test of hypothesis
3 d-test of hypothesisRiccardo Rigon
 
2 a-idro-geomorfologia-the basics
2 a-idro-geomorfologia-the basics2 a-idro-geomorfologia-the basics
2 a-idro-geomorfologia-the basicsRiccardo Rigon
 
Global Sensitivity Analysis for the Calibration of a Fully-distributed Hydrol...
Global Sensitivity Analysis for the Calibration of a Fully-distributed Hydrol...Global Sensitivity Analysis for the Calibration of a Fully-distributed Hydrol...
Global Sensitivity Analysis for the Calibration of a Fully-distributed Hydrol...Mauricio Zambrano-Bigiarini
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 

More Related Content

Recently uploaded

Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdfWriting Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdfMr Bounab Samir
 
UniSC Moreton Bay Library self-guided tour
UniSC Moreton Bay Library self-guided tourUniSC Moreton Bay Library self-guided tour
UniSC Moreton Bay Library self-guided tourUSC_Library
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptxMaryPotorti1
 
Evaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxEvaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxgarvitnanecha
 
Bayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesBayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesTushar Tank
 
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfDiploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfSUMIT TIWARI
 
Overview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfOverview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfChristalin Nelson
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...Nguyen Thanh Tu Collection
 
Chromatography-Gas chromatography-Principle
Chromatography-Gas chromatography-PrincipleChromatography-Gas chromatography-Principle
Chromatography-Gas chromatography-Principleblessipriyanka
 
EDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdfEDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdfElizabeth Walsh
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaFaheema Hasan
 
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapitolTechU
 
UniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUSC_Library
 
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...Rabiya Husain
 
Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfOverview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfChristalin Nelson
 
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)Rabiya Husain
 
Introduction of General Pharmacology PPT.pptx
Introduction of General Pharmacology PPT.pptxIntroduction of General Pharmacology PPT.pptx
Introduction of General Pharmacology PPT.pptxRenuka N Sunagad
 
mean stack mean stack mean stack mean stack
mean stack mean stack  mean stack  mean stackmean stack mean stack  mean stack  mean stack
mean stack mean stack mean stack mean stackNuttavutThongjor1
 
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTScatch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTSCarlaNicolas7
 
Appendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptxAppendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptxRenuka N Sunagad
 

Recently uploaded (20)

Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdfWriting Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
Writing Agony Letter & If type O+1 & Diphthongs + Text “Arab Science”.pdf
 
UniSC Moreton Bay Library self-guided tour
UniSC Moreton Bay Library self-guided tourUniSC Moreton Bay Library self-guided tour
UniSC Moreton Bay Library self-guided tour
 
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx2.20.24 Asian Americans and the Myth of the Model Minority.pptx
2.20.24 Asian Americans and the Myth of the Model Minority.pptx
 
Evaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptxEvaluation and management of patients with Dyspepsia.pptx
Evaluation and management of patients with Dyspepsia.pptx
 
Bayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with ExamplesBayesian Analysis Fundamentals with Examples
Bayesian Analysis Fundamentals with Examples
 
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdfDiploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
Diploma 2nd yr PHARMACOLOGY chapter 5 part 1.pdf
 
Overview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdfOverview of Databases and Data Modelling-1.pdf
Overview of Databases and Data Modelling-1.pdf
 
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
50 ĐỀ THI THỬ TỐT NGHIỆP THPT TIẾNG ANH 2024 CÓ GIẢI CHI TIẾT - GIỚI HẠN KHO...
 
Chromatography-Gas chromatography-Principle
Chromatography-Gas chromatography-PrincipleChromatography-Gas chromatography-Principle
Chromatography-Gas chromatography-Principle
 
EDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdfEDL 290F Week 1 - Meet Me at the Start Line.pdf
EDL 290F Week 1 - Meet Me at the Start Line.pdf
 
BTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic LeukemiaBTKi in Treatment Of Chronic Lymphocytic Leukemia
BTKi in Treatment Of Chronic Lymphocytic Leukemia
 
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptxCapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
CapTechTalks Webinar Feb 2024 Darrell Burrell.pptx
 
UniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tourUniSC Fraser Coast library self-guided tour
UniSC Fraser Coast library self-guided tour
 
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
Barrow Motor Ability Test - TEST, MEASUREMENT AND EVALUATION IN PHYSICAL EDUC...
 
Overview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdfOverview of Databases and Data Modelling-2.pdf
Overview of Databases and Data Modelling-2.pdf
 
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
HOW TO DEVELOP A RESEARCH PROPOSAL (FOR RESEARCH SCHOLARS)
 
Introduction of General Pharmacology PPT.pptx
Introduction of General Pharmacology PPT.pptxIntroduction of General Pharmacology PPT.pptx
Introduction of General Pharmacology PPT.pptx
 
mean stack mean stack mean stack mean stack
mean stack mean stack  mean stack  mean stackmean stack mean stack  mean stack  mean stack
mean stack mean stack mean stack mean stack
 
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTScatch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
catch-up-friday-ARALING PNLIPUNAN SOCIAL JUSTICE AND HUMAN RIGHTS
 
Appendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptxAppendicular SkeletonSystem PPT.....pptx
Appendicular SkeletonSystem PPT.....pptx
 

Featured

ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at WorkGetSmarter
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...DevGAMM Conference
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationErica Santiago
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellSaba Software
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming LanguageSimplilearn
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...Palo Alto Software
 
9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation9 Tips for a Work-free Vacation
9 Tips for a Work-free VacationWeekdone.com
 
I Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from PrinceI Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from PrinceEmpowered Presentations
 
How to Map Your Future
How to Map Your FutureHow to Map Your Future
How to Map Your FutureSlideShop.com
 

Featured (20)

ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
 
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
The Pixar Way: 37 Quotes on Developing and Maintaining a Creative Company (fr...
 
9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation9 Tips for a Work-free Vacation
9 Tips for a Work-free Vacation
 
I Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from PrinceI Rock Therefore I Am. 20 Legendary Quotes from Prince
I Rock Therefore I Am. 20 Legendary Quotes from Prince
 
How to Map Your Future
How to Map Your FutureHow to Map Your Future
How to Map Your Future
 

Comparing Goodness-of-fit Measures for Calibration of Models Focused on Extreme Events (EGU 2012)

  • 1. Joint Research Comparing Goodness-of-fit Measures for Calibration Comparing Goodness-of-fit Measures for Calibration of of Centre EGU2012-11549 Session: HS1.3 Models Focused on Extreme Events Models Focused on Extreme Events Apr 23th, 2012 Mauricio Zambrano-Bigiarini and Alberto Bellin Mauricio Zambrano-Bigiarini and Alberto Bellin 1) Motivation 4) Results Despite serious and well-known limitations ( e.g., Legates and McCabe, 1999), many single objective goodness-of-fit measures are still of widespread use. As an example, the Nash-Sutcliffe 3.1) Nash-Sutcliffe efficiency : 3.6) Relative Nash-Sutcliffe efficiency : Nomenclature efficiency (NSE) has been highly criticised as an inappropriate (Nash and Sutcliffe, 1970) N (Krausse et al., 2005) ● Si : i-th simulated value 2 benchmark for comparing modelling results to observations N Oi−S i ( ) 2 ∑ ( O i − Si ) ∑ ● Oi : i-th observed value (e.g., Schaefli and Gupta 2007), nonetheless, it is still one of NSE =1− i =1 i =1 Oi N 2 rNSE =1− ● j : arbitrary power, i.e, positive integer the most common performance measures used by both ∑ ( O i− O ) N Oi− O 2 environmental scientists and practitioners. i =1 ∑ i =1 ( O ) ● ● Ō : mean observed value Ôi : median of the observed values in 3.2) Index of Agreement: 3.7) Relative Index of Agreement: the same month than Oi 2) Aim (Willmot, 1981) N (Krausse et al., 2005) N 2 ● r : Pearson's product-moment Oi− S i ( ) 2 ∑ ( O i − Si ) correlation coefficient To provide practical guidance about how different goodness- ∑ d =1− i =1 Oi α : ratio between the standard of-fit measures reported in literature perform when used within i =1 ● N 2 rd =1− N 2 deviation of simulations (σs) and ∑ (∣Si− O∣+∣O i− O∣) ∣Si − O∣+ ∣Oi −O∣ a single-objective optimisation procedure, both for the identification of model parameters and in the reproduction of i =1 ∑ i =1 ( O ) observations (σo) ● β : ratio between the mean of the high- and low-flow events. 3.3) Coefficient of Persistence: 3.8) Modified Nash-Sutcliffe efficiency : simulations (μs) and observations (Kitanidis & Bras, 1980) (Legates and McCabe, 1999) (μo) N N 2 3) Methodology ∑ ( S i− O i ) ∑ ∣Oi −Si∣j ● ωi : weight in [0,1] applied to both i =2 cp =1− N mNSE =1− i =1 observed and simulated values at Fig 08. Boxplots summarizing the 5th and 50th 3.1) Study Area N time step i ∑ ( O i− Oi−1 )2 ∑ ∣Oi −O∣j Fig 07. Boxplots summarizing the parameter values percentiles of daily discharge obtained with calibrations i =2 i =1 ● λ : number in [0, 1] representing the focused on low flows. Only the best half of the total weight given to the high-flow part obtained with calibrations focused on low flows. Only parameter sets obtained during calibration are 3.4) Volumetric Efficiency: 3.9) Modified Index of Agreement : of the signal, with λ close to 1 the best half of the total parameter sets obtained considered for each calibration exercise. Grey vertical (Criss and Winston, 2008) (Legates and McCabe, 1999) when focusing on high-flow during calibration are considered for each calibration Fig 05. Boxplots summarizing the parameter values lines indicate the value of the observed 5th and 50th N events, and λ close to zero when exercise. N obtained with calibrations focused on high flows. Only percentiles. ∑ ∣Oi −Si∣ ∑ ( O i− S i ) j focusing in low-flow conditions the best half of the total parameter sets obtained i =1 VE =1− i=1 N d j=1− OL ,OH : user-defined thresholds used during calibration are considered for each calibration 8) Conclusions ● N j exercise. ∑ Oi ∑ (∣Si −O∣+ ∣Oi −O∣) to separate low and high values, respectively. In order to avoid i =1 i =1 subjectivity in the selection of OL ● Large underestimation of observed low flows (~40%) were 3.5) Kling-Gupta efficiency: 3.10) Weighted Seasonal Nash-Sutcliffe: and OH , we use the flow duration obtained with simulated values calibrated by using NSE and the (this work) N (Gupta et al., 2009) curve criterion proposed by Yilmaz ∑ ∣ω i ( Oi −Si )∣j et al., 2008 KGE. Such underestimation is commonly masked out by the KGE =1− √( r −1)2 + (α −1)2+ (β−1)2 wsNSE =1− i=1 N good overall fit in terms of NSE, KGE and other statistics. ∑ ∣ω i (O i− Oi )∣j ̂ Cov s, o σs μs i =1 r = σ σ α = σ o β= μ o Low-flows calibrations: { } s o λ , O i⩾OH 3.6) Seasonal Nash-Sutcliffe: ω i = (1−λ )+ (2 λ −1)( Oi− OL ) , OL < Oi < O H ● rNSE and wsNSE (j=1, λ=0) perform the best (in terms of 5th O H −OL (Adapted from Garrick et al., 1978) 1−λ , O i⩾OL percentile), and both of them provide also a good N ∑ ∣Oi −Si∣j representation of medium flows (50th percentile) with all the sNSE =1− iN =1 other measures overestimating them. ∑ ∣O i−O i∣j ̂ ● NSE, KGE and d tend to underestimate GW_DELAY in i =1 comparison to to the other goodness-of-fit measures. Fig 01. Location of the Ega River Basin, meteorological stations, and discharge station used for the calibration of the upper catchment. High-flows calibrations: 3.2) Calibration Procedure ● wsNSE (j=2, λ=0.95), d and KGE perform the best (in terms of The Soil and Water Assessment Tool (SWAT) version 2005 95th percentile). At the same time, only wsNSE provides a good was calibrated for the period Jan/1961-Dec/1970, using the representation of medium flows (50th percentile), with all the first year as warming up period. other goodness-of-fit measures overestimating them. A set of 9 parameters was selected for calibration: ● The optimal value and variability of parameters related to the Parameter Min Max Fig 02. Discharge time series corresponding to the outlet of the upper part of the Ega River Basin (Ega en Estella stream gauge, Q071). slow response of the catchment (GW_DELAY and ALPHA_BF) Base flow alpha factor [days] ALPHA_BF 1.00E-1 9.90E-1 Horizontal blue and red lines show the discharge values used to separate high and low flows, respectively (see Fig 03). Manning's “n” value for the main channel [-] CH_N2 1.60E-2 1.50E-1 was very similar among all the goodness-of-fit measures tested Fig 06. Boxplots summarizing the 50th and 95th Initial SCS CN II value [-] CN2 4.00E+1 9.50E+1 percentiles of daily discharge obtained with (both close to zero), with KGE and wsNSE presenting the Saturated hydraulic conductivity [mm/hr] SOL_K 1.00E-3 1.00E+3 calibrations focused on high flows. Only the best half largest spread. However, those values are very different from of the total parameter sets obtained during calibration Available water capacity, [mmH2O/mm soil] SOL_AWC 1.00E-2 3.50E-1 are considered for each calibration exercise. Grey the ones obtained during the calibration focused on low flows. Effective hydraulic conductivity in main channel [mm/hr] CH_K2 0.00E+0 2.00E+2 vertical lines indicate the value of the observed 5th Soil evaporation compensation factor [-] ESCO 1.00E-2 1.00E+0 and 50th percentiles. Surface runoff lag time [days] SURLAG 1.00E+0 1.20E+1 References: Snowfall temperature [°C] SFTMP -5.00E+0 5.00E+0 ● Criss, R., Winston, W., 2008. Do Nash values have value? Discussion and alternate proposals. Hydrological Processes 22, 2723–2725 Calibration was carried out using Particle Swarm Optimisation ● Garrick, M., Cunnane, C., Nash, J.E., 1978. A criterion of efficiency for rainfall-runoff models. Journal of Hydrology 36, 375–381 Fig 03. Daily flow duration curve corresponding to Kennedy, J., and R. Eberhart (1995), Particle swarm optimization, in Proceedings IEEE International Conference on Neural Networks, 1995, vol. 4, pp. 1942–1948, (PSO, Kennedy and Eberhart, 1995), with 20 particles, 300 ● the outlet of the upper part of the Ega River Basin Fig 04. Weighting values used in wsNSE, which take into account the skewness of the ● Kitanidis, P.K., Bras, R.L., 1980. Real-time forecasting with a conceptual hydrologic model 2. applications and results. Water Resources Research 16, 1034–1044. iterations, ω=1/(2log2), c1=c2=0.5+log2, linearly decreasing (Q071). Verticl black lines show the discharge observed daily discharges. Red and blue lines correspond to the weights used when ● Krause, P., Boyle, D., Bäse, F., 2005. Comparison of different efficiency criteria for hydrological model assessment. Advances in Geosciences 5, 89–97.. Vmax from 1.0 to 0.5, and random topology (see EGU2012-10950 values used to separate high and low flows focusing on low- and high-flow events, respectively. On the left panel, the horizontal axis ● Legates, D., McCabe Jr., G., 1999. Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resources Research 35, following Yilmaz et. al., 2008) represent discharges values of the Ega River Basin (Q071), while on the right panel the 233–241 for further details). Schaefli, B., Gupta, H., 2007. Do Nash values have value?. Hydrological Processes 21, 2075–2080. www.jrc.europa.eu Mauricio Zambrano-Bigiarini horizontal axis represent the empirical CDF of the observed discharge values. ● ● Willmott, C., 1981. On the validation of models. Physical Geography 2, 184–194. European Commission • Joint Research Centre • Institute for Environment and Sustainability ● Yilmaz, K., Gupta, H., Wagener, T., 2008. A process-based diagnostic approach to model evaluation: Application to the NWS distributed hydrologic model. Water Tel. +39 0332 789588 • Email: mauricio.zambrano@jrc.ec.europa.eu Resources Research 44, W09417.