EURO-­‐BASIN,	  www.euro-­‐basin.eu	     Introduc)on	  to	  Sta)s)cal	  Modelling	  Tools	  for	  Habitat	  Models	  Devel...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development     Outline   ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tio...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—riso...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—riso...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—riso...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—riso...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—riso...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—riso...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples     O...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples     Z...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples     A...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples     P...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples     Z...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek—     Outli...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek—     Weka ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek—     Weka ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek—     Weka ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es    ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es    ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es    ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es    ...
i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es    ...
EURO-­‐BASIN,	  www.euro-­‐basin.eu	     Introduc)on	  to	  Sta)s)cal	  Modelling	  Tools	  for	  Habitat	  Models	  Devel...
Upcoming SlideShare
Loading in …5
×

Model Validation, performance measures, models comparison and Weka (open source software for data mining) by JA Fernandes

3,277 views

Published on

Key lecture for the EURO-BASIN Training Workshop on Introduction to Statistical Modelling for Habitat Model Development, 26-28 Oct, AZTI-Tecnalia, Pasaia, Spain (www.euro-basin.eu)

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,277
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
97
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Model Validation, performance measures, models comparison and Weka (open source software for data mining) by JA Fernandes

  1. 1. EURO-­‐BASIN,  www.euro-­‐basin.eu   Introduc)on  to  Sta)s)cal  Modelling  Tools  for  Habitat  Models  Development,  26-­‐28th  Oct  2011  
  2. 2. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  3. 3. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  4. 4. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Introduction ƒlides ˜—sed m—inly in ‡itten —nd pr—nk @PHHSAY €érez et —lF @PHHSAY ellen @PHHWAY pern—ndes @PHIIA y˜je™tiveX to me—sure how well — model represents truthF „ruth ™—nnot ˜e —™™ur—tely me—suredX o˜serv—tionsF uestionsX How well the model ts the observations (goodness-of-t)? How well the model forecast new events (generalisation)? How superior is one model compared to another? Which is more important, precision or trend? enswersX Validation procedures. Metrics or performance measures. Statistical tests.
  5. 5. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Model prediction (P), observations (O), true state (T) —A model with no skill ˜A ide—l model ‚eprodu™ed from ƒtow et —lF @PHHWA —nd ellen @PHHWA
  6. 6. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Goodness-of-t vs generalisation pittingX N: Total number of cases Training-set Test-set Chances of over-tting. qener—liz—tion → tr—inEtest splitX N: Total number of cases Training-set Test-set Hold-out (commonly 66%-33% split) (Larson, 1931) Hold-out depends on how fortunate the train-test split is.
  7. 7. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion K-fold cross-validation (CV) €erform—n™e is the —ver—ge of k models @v—™hen˜ru™h —nd wi™keyD IWTVY ƒtoneD IWURAF ell d—t— is eventu—lly used for testingF ƒtill sensitive to d—t— splitX str—ti(edD repe—ted @fou™k—ert —nd pr—nkD PHHRAF ‚eprodu™ed from €érez et —lF @PHHSAF
  8. 8. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Leave-one-out cross-validation (LOOCV) N: Total number of cases ... ... ... ... ... ... ... ... x modelsD xEI ™—ses for tr—ining —nd I ™—se for testing @wosteller —nd „ukeyD IWTVAF ƒuit—˜le for sm—ll d—t—setsD more ™omput—tion—lly expensiveF †—ri—n™e of the error is the l—rgestD ˜ut less ˜i—sedF st ™—n ˜e used for more st—˜le p—r—meters @less v—ri—n™eA
  9. 9. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Bootstrapping (0.632 bootstrap) e ™—se h—s — HFTQP pro˜—˜ility of ˜eing pi™ked for tr—iningEset @ifronD IWUWAF error a H.TQP ∗ etest @gener—lis—tionA C H.QTV ∗ etraining @(tAF et le—st IHH res—mplingsD some studies suggest IHHHHF ‚eprodu™ed from €érez et —lF @PHHSAF
  10. 10. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Sumarizing Real performance Real performance Estimated performance Accuracy Estimated performance Precision sn™re—sing d—t— p—rtitions le—ds to FFF more accurate performance estimation (+). more variance in the performance estimation, less precise (-). more computationally expensive (-). uEfold ™rossEv—lid—tionX tr—deEo @‚odríguez et —lFD PHIHAF
  11. 11. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Pipeline validation in lter methods Discretize Factor Factors Selection Naive Bayes
  12. 12. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Pipeline validation in lter methods Discretize Factor Factors Selection Naive Bayes
  13. 13. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Pipeline validation in lter methods Discretize Factor Factors Selection Naive Bayes Full 10x5cv Dataset
  14. 14. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Pipeline validation in lter methods Discretize Factor Train 1 Factors Selection Performance Test 1 Naive Bayes estimation ... (Fold 1) ... ... Discretize Factor ... Train 5 Factors Selection Performance Performance estimation Test 5 Naive Bayes estimation Full 10x5cv (Fold 5) Dataset
  15. 15. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Pipeline validation in lter methods Discretize Factor Train 1 R Factors Selection E Performance P Test 1 Naive Bayes estimation E ... (Fold 1) ... ... A Discretize Factor ... T Train 5 Factors Selection 1 Performance Performance estimation Test 5 Naive Bayes estimation (Repeat 1) Full 10x5cv (Fold 5) ... Dataset Train 1 Whole methodology R performance estimation . . . E 10 repeats average P Test 1 E ... A Performance estimation T Train 5 (Repeat 10) 10 . . . Test 5
  16. 16. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodel†—lid—tion Pipeline validation in wrapper methods Model validation Train 1 Discretize Discretize CFS with Model building Class Predictors LOOCV Test 1 Naive Bayes Performance estimation Bootstrapping (Bootstrap 1) ... ... (100) ... ... Discretize Discretize CFS with Class Train 100 Class Predictors LOOCV cut-off points Performance estimation Test 100 Naive Bayes evaluation (Bootstrap 100) Apply selected class discretization Train 1 Discretize CFS with Predictors LOOCV R E Performance P Test 1 Naive Bayes estimation E (Fold 1) ... A T Train 5 1 . . . Performance Performance estimation Test 5 estimation (Repeat 1) (Fold 5) 5 folds average Full 10x5cv ... Dataset Train 1 Whole methodology R . . . E performance estimation P Test 1 10 repeats average E ... A T Train 5 Performance estimation 10 (Repeat 10) . . . 5 folds average Test 5
  17. 17. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  18. 18. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures Introduction to metrics i—™h metri™ shows — dierent property of the model @rolt et —lFD PHHSY pern—ndes et —lFD PHIHA vow vs highX Lower is better (error) Higher is better (performance) foundsX Boundless Between 0 and 1 Between 0 and 100%
  19. 19. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  20. 20. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Numeric prediction metrics ‡here p —re predi™ted v—lues —nd a —re the —™tu—l v—luesF we—nEsqu—red errorX outliers → me—n —˜solute errorF ‚el—tive squ—red errorX rel—tive to the me—n of —™tu—l v—luesF gorrel—tion ™oe(™ientX ˜ounded ˜etween I —nd EIF ‚eprodu™ed from ‡itten —nd pr—nk @PHHSAF
  21. 21. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Root Mean Squared Error (RMSE) (p − a)2 RMSE = n qoodness of (t ˜etween model —nd o˜serv—tionsF „he ™loser to H the ˜etter is the (tF sf ‚wƒi gre—ter th—n v—ri—n™e of o˜serv—tionsX poor modelF ‚eprodu™ed from ellen @PHHWA
  22. 22. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Nash Sutclie Model Eciency) N 2 ME =I− n=1 (an − pn ) N 2 n=1 (an − a)) ‚—tio of the model error to d—t— v—ri—˜ilityF vevelsX bHFTS ex™ellentD bHFS very goodD bHFP goodD `HFP poor wáre™h—l @PHHRAF €roposed in x—sh —nd ƒut™lie @IWUHAD reprodu™ed from ellen @PHHWA
  23. 23. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Percentage Model Bias N Pbias = n=1 (an − pn ) ∗ IHH N n=1 (an ) ƒum of model error norm—lised ˜y the d—t—F we—sure of underestim—tion or overestim—tion of o˜serv—tionsF vevelsX `IH ex™ellentD `PH very goodD `RH goodD bRH poor wáre™h—l @PHHRAF ‚eprodu™ed from ellen @PHHWA
  24. 24. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Pearson correlation coecient (R) N R = n=1 (an − a)(pn − p ) ∗ IHH N 2 N 2 n=1 (an − a) n=1 (pn − p ) u—lity of (t of — model to o˜serv—tionsF ‚ a HD no rel—tionshipF ‚ a ID perfe™t (tF ƒqu—re of the ™orrel—tion ™oe0™ient @R2 AX per™ent—ge of the v—ri—˜ility in d—t— —™™ounted for ˜y the modelF ‚eprodu™ed from ellen @PHHWAF
  25. 25. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Reliability Index (RI) N I an RI = exp (log )2 n pn n =1 p—™tor of divergen™e ˜etween predi™tions —nd d—t—F ‚s a PD me—ns — divergen™e on —ver—ge within of — multipli™—tive f—™tor of PF ‚s the ™loser to I the ˜etterF ‚eprodu™ed from ellen @PHHWA
  26. 26. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Cost functions ho —ll errors h—ve the s—me weightD ™ost or impli™—tionsc ƒ™—ling of dieren™es ˜etween p —nd aF iFgF ‚wƒi s™—led ˜y the v—ri—n™e of d—t— @rolt et —lFD PHHSAF hierent ™ost v—lues depending on the type of errorF
  27. 27. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  28. 28. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Confusion matrix: accuracy and true positive Accuracy = TPcases # +TN True Positive Rate = TPTPFN + righer is ˜etter for ˜othF
  29. 29. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Confusion matrix: accuracy and true positive Accuracy = TPcases # +TN True Positive Rate = TPTPFN + righer is ˜etter for ˜othF
  30. 30. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Confusion matrix: accuracy and true positive Accuracy = TPcases # +TN True Positive Rate = TPTPFN + righer is ˜etter for ˜othF
  31. 31. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Confusion matrix: accuracy and true positive Accuracy = TPcases # +TN True Positive Rate = TPTPFN + righer is ˜etter for ˜othF
  32. 32. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Confusion matrix: accuracy and true positive Accuracy = TPcases # +TN True Positive Rate = TPTPFN + righer is ˜etter for ˜othF
  33. 33. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Confusion matrix: accuracy and true positive Accuracy = TPcases # +TN True Positive Rate = TPTPFN + righer is ˜etter for ˜othF
  34. 34. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l Actual High
  35. 35. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l yK = 0 l Actual Otherwise High Medium Low
  36. 36. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l yK = 0 l Actual Otherwise High Medium Low p1 0.7 0.2 0.1 (0.7-1)2 + (0.2-0)2 + (0.1-0)2 = 0.14
  37. 37. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l yK = 0 l Actual Otherwise High Medium Low p1 0.7 0.2 0.1 (0.7-1)2 + (0.2-0)2 + (0.1-0)2 = 0.14 p 2 0.8 0.1 0.1 (0.8-1)2 + (0.1-0)2 + (0.1-0)2 = 0.06
  38. 38. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l yK = 0 l Actual Otherwise High Medium Low p1 0.7 0.2 0.1 (0.7-1)2 + (0.2-0)2 + (0.1-0)2 = 0.14 p 2 0.8 0.1 0.1 (0.8-1)2 + (0.1-0)2 + (0.1-0)2 = 0.06 p3 0.1 0.5 0.4 (0.1-1)2 + (0.5-0)2 + (0.4-0)2 = 1.22
  39. 39. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l yK = 0 l Actual Otherwise High Medium Low p1 0.7 0.2 0.1 (0.7-1)2 + (0.2-0)2 + (0.1-0)2 = 0.14 p 2 0.8 0.1 0.1 (0.8-1)2 + (0.1-0)2 + (0.1-0)2 = 0.06 p3 0.1 0.5 0.4 (0.1-1)2 + (0.5-0)2 + (0.4-0)2 = 1.22 p 4 0.4 0.5 0.1 (0.4-1)2 + (0.5-0)2 + (0.1-0)2 = 0.62
  40. 40. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Brier Score @frierD IWSHY v—n der q——g et —lFD PHHPY ‰eung et —lFD PHHSA #cases #classes k Brier Score = 1 #cases k =1 l =1 (pl − ylk )2 vower is ˜etter @™ontr—ry to —™™ur—™y 8 true positiveA vevelsX `HFIH ex™ellentD `PH superiorD `HFQH —dequ—teD `HFQS —™™ept—˜leD bHFQS insu(™ient @pern—ndesD PHIIA yK = 1 l yK = 0 l Actual Otherwise High Medium Low p1 0.7 0.2 0.1 (0.7-1)2 + (0.2-0)2 + (0.1-0)2 = 0.14 p 2 0.8 0.1 0.1 (0.8-1)2 + (0.1-0)2 + (0.1-0)2 = 0.06 p3 0.1 0.5 0.4 (0.1-1)2 + (0.5-0)2 + (0.4-0)2 = 1.22 p 4 0.4 0.5 0.1 (0.4-1)2 + (0.5-0)2 + (0.1-0)2 = 0.62 Brier Score: (0.14 + 0.06 +1.22 + 0.62) / 4 = 0.51 Normalized Brier Score: 0.51 / 2 = 0.255
  41. 41. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Percent Reduction in Error (PRE) „he relev—n™e of — perform—n™e g—inF e P7 g—in of —n —lre—dy highly —™™ur—te ™l—ssi(er @WH7A FFF more relev—nt th—n with low st—rting —™™ur—™y @SH7A EB − EA PRE = IHH · EB if is the error in the (rst method @irror feforeA ie is in the se™ond method @irror efterA
  42. 42. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Accuracy paradox w—inly with un˜—l—n™ed d—t—sets @hu —nd h—vidsonD PHHUY e˜m—D PHHWAF ‚eprodu™ed from ‡ikipedi— @PHIIAF
  43. 43. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Minimum Description Length (MDL) principle uiss ruleX ueep st ƒimple FFF y™™—m9s ‚—zorX „he simplest expl—n—tion is the most likely to ˜e true FFF FFF —nd is more e—sily —™™epted ˜y others FFF FFF ˜utD it is not ne™ess—rily the truthF „he more — sequen™e of d—t— ™—n ˜e ™ompressedD FFF FFF the more regul—rity h—s ˜een dete™ted in the d—t—X whvX winimum hes™ription vength @‚iss—nenD IWUVA „r—deEo ˜etween perform—n™e —nd ™omplexityF ss whv f—lsec homingos @IWWWAY qrünw—ld et —lF @PHHSA „r—deEo ˜etween me™h—nism —nd ro˜ust p—r—metersF sf two models h—ve s—me perform—n™e then keep the simplestF
  44. 44. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Example complex vs simple
  45. 45. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development €erform—n™ewe—sures €erform—n™ewe—sures Lift chart, ROC curve, recall-precision curve
  46. 46. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—rison Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  47. 47. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—rison Corrected paired t-test ƒt—tisti™—l ™omp—risons of the perform—n™eF sde—lX test over sever—l d—t—sets of size NF xull hypothesis th—t the me—n dieren™e is zeroF irrorsX „ype sX pro˜F the test reje™ts the null hypothesis in™orre™tly „ype ssX pro˜F the null hypotF is not reje™ted with dieren™eF ‚e—lityX only one d—t—set of size N to get —ll estim—tesF €ro˜lemX „ype s errors ex™eed the signi(™—n™e level ƒolutionX heuristi™ versions of the t-testF @x—de—u —nd fengioD PHHQY w™gluskey —nd v—lkhenD PHHUY uotsi—ntisD PHHUY pern—ndesD PHIIA gomp—ring w…v„s€vi methods over yxi d—t—setsF gomp—ring yxi methods over w…v„s€vi d—t—setsF
  48. 48. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—rison Critical dierence diagrams § €roposed ˜y hems—r @PHHTA ‚evised priedm—n plus ƒh—er9s st—ti™ postEho™ test @q—r™í— —nd rerrer—D PHHVAF gomp—ring w…v„s€vi methods over w…v„s€vi d—t—setsF ƒhows —ver—ge r—nk of methods superiority in d—t—setsF xo signi(™—nt dieren™eX line ™onne™ting methodsF wore d—t—setsX more e—sy to (nd signi(™—nt diferen™esF
  49. 49. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—rison Taylor diagrams 2 E = σf + σr − Pσf σr R ; c 2 = a2 + b2 − Pab ™os ϕ 2 2 ƒimult—neouslyX ‚wƒ dieren™eD ™orrel—tion —nd stdF devF ‚X ™orrel—tion p aY E X ‚wƒ diFY σf σr X v—ri—n™es p aF 2 2 €roposed in „—ylor @PHHIAD reprodu™ed from ellen @PHHWAF
  50. 50. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—rison Target diagrams ‚wƒi in ˆE—xisY fi—s in ‰E—xisF p ƒtdF hevF l—rger @xbHA th—n aY fi—s positive @‰bHA or notF ‚eprodu™ed from tolli et —lF @PHHWA —nd ellen @PHHWAF
  51. 51. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development wodelgomp—rison Multivariate aproaches …niEv—ri—te multiEv—ri—te metri™s summ—rize model skillF wultiEv—ri—te —ppro—™hesX simult—neous ex—min—tion of sever—l v—ri—˜les v—ri—tion to e—™h other sp—ti—lly —nd tempor—llyF €rin™ip—l gomponet en—lysis @€geA @tollieD PHHPAF ƒhow the rel—tionship ˜etween sever—l v—ri—˜les in Ph sp—™eF wulti himension—l ƒ™—lling @whƒA @forg —nd qroenenD PHHSAF ixploring simil—rities or dissimil—rities in d—t— ƒelf org—nizing w—ps @ƒywA @uohonen —nd w—psD PHHIAF €rodu™e — lowEdimension—l dis™retized represent—tion of the o˜serv—tionsF
  52. 52. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  53. 53. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples Zooplankton biomass models ƒever—l models (ts with squ—red errorF ‚eprodu™ed from srigoien et —lF @PHHWAF
  54. 54. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples An example of anchovy recruitment €erform—n™e reported depending on v—lid—tion s™hem—F ‚eprodu™ed from pern—ndes et —lF @PHIHAF
  55. 55. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples Phytoplankton classication ‡ithout @„—˜le sssA —nd with @„—˜le ssA st—tisti™—l dieren™es @™orre™ted p—ired tEtestAF ‚eprodu™ed from —r—uz et —lF @PHHWA —nd —r—uz et —lF @PHHVAF
  56. 56. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ix—mples Zooplankton classication ‚eprodu™ed from pern—ndes et —lF @PHHWAF
  57. 57. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek— Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  58. 58. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek— Weka explorer
  59. 59. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek— Weka experimenter
  60. 60. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‡ek— Weka knowledge ow
  61. 61. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es Outline 1 wodel v—lid—tion 2 €erform—n™e me—sures or metri™s wetri™s in numeri™ predi™tion wetri™s in ™l—ssi(™—tion 3 gomp—ring methodologies —nd models 4 ix—mples 5 ‡ek—X open sour™e d—t— mining tool 6 ‚eferen™es
  62. 62. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es e˜m—D fF @PHHWAF Evaluation of requirements management tools with support for traceability-based change impact analysisF €hh thesisD …niversity of „wenteD ins™hedeD „he xetherl—ndsF ellenD tF @PHHWAF hPFU user guide —nd report outlining v—lid—tion methodologyF Deliverable in project Marine Ecosystem Evolution in a Changing Enviroment (MEECE). forgD sF —nd qroenenD €F @PHHSAF Modern multidimensional scaling: Theory and applicationsF ƒpringer †erl—gF fou™k—ertD ‚F ‚F —nd pr—nkD iF @PHHRAF iv—lu—ting the repli™—˜ility of signi(™—n™e tests for ™omp—ring le—rning —lgorithmsF Lect. Notes Artif. Int.D p—ges Q!IPF frierD qF ‡F @IWSHAF †eri(™—tion of fore™—sts expressed in terms of pro˜—˜ilityF Month. Weather Rev.D UV@IAXI!QF hems—rD tF @PHHTAF ƒt—tisti™—l ™omp—risons of ™l—ssi(ers over multiple d—t— setsF J. Mach. Learn. Res.D § UXI!QHF homingosD €F @IWWWAF „he role of y™™—m9s r—zor in knowledge dis™overyF Data Min. Knowl. DiscD Q@RAXRHW!RPSF ifronD fF @IWUWAF footstr—p methodsX —nother look —t the j—™kknifeF Ann. Stat.D U@IAXI!PTF pern—ndesD tF @PHIIAF Data analysis advances in marine science for sheries management: Supervised classication applicationsF €hh thesisD …niversity of the f—sque gountryD ƒ—n ƒe˜—sti—nD quipuzko—D ƒp—inF pern—ndesD tF eFD srigoienD ˆFD foyr—D qFD voz—noD tF eFD —nd snz—D sF @PHHWAF yptimizing the num˜er of ™l—sses in —utom—ted zoopl—nkton ™l—ssi(™—tionF J. Plankton Res.D QI@IAXIW!PWF pern—ndesD tF eFD srigoienD ˆFD qoikoetxe—D xFD voz—noD tF eFD snz—D sFD €érezD eFD —nd fodeD eF @PHIHAF pish re™ruitment predi™tionD using ro˜ust supervised ™l—ssi(™—tion methodsF Ecol. Model.D PPI@PAXQQV!QSPF pr—n™isD ‚F sF gF @PHHTAF we—suring the strength of environmentEre™ruitment rel—tionshipsX the import—n™e of in™luding predi™tor s™reening within ™rossEv—lid—tionsF ICES J. Mar. Sci.D TQ@RAXSWRF q—r™í—D ƒF —nd rerrer—D pF @PHHVAF en extension on 9st—tisti™—l ™omp—risons of ™l—ssi(ers over multiple d—t— sets9 for —ll p—irwise ™omp—risonsF J. Mach. Learn. Res.D WXPTUU!PTWRF
  63. 63. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es qrünw—ldD €FD wyungD sFD —nd €ittD wF @PHHSAF Advances in minimum description length: Theory and applicationsF „he ws„ €ressF roltD tFD ellenD tFD €ro™torD ‚FD —nd qil˜ertD pF @PHHSAF irror qu—nti(™—tion of — highEresolution ™oupled hydrodyn—mi™Ee™osystem ™o—st—lEo™e—n modelX €—rt I model overview —nd —ssessment of the hydrodyn—mi™sF Journal of Marine SystemsD SU@IEPAXITU!IVVF srigoienD ˆFD pern—ndesD tFD qrosje—nD €FD henisD uFD el˜—in—D eFD —nd ƒ—ntosD wF @PHHWAF ƒpring zoopl—nkton distri˜ution in the f—y of fis™—y from IWWV to PHHT in rel—tion with —n™hovy re™ruitmentF J. Plankton Res.D QI@IAXI!IUF tolliD tFD uindleD tFD ƒhulm—nD sFD €ent—D fFD priedri™hsD wFD rel˜erD ‚FD —nd ernoneD ‚F @PHHWAF ƒumm—ry di—gr—ms for ™oupled hydrodyn—mi™Ee™osystem model skill —ssessmentF Journal of Marine SystemsD UT@IEPAXTR!VPF tollieD sF @PHHPAF €rin™ip—l ™omponent —n—lysisF Encyclopedia of Statistics in Behavioral ScienceF uohonenD „F —nd w—psD ƒF @PHHIAF ƒpringer series in inform—tion s™ien™esF New York, New YorkF uotsi—ntisD ƒF @PHHUAF ƒupervised w—™hine ve—rningX e ‚eview of gl—ssi(™—tion „e™hniquesF Inform.D QIXPRW!PTVF v—™hen˜ru™hD €F —nd wi™keyD wF @IWTVAF istim—tion of error r—tes in dis™rimin—nt —n—lysisF TechnometricsD p—ges I!IIF v—rsonD ƒF gF @IWQIAF „he shrink—ge of the ™oe0™ient of multiple ™orrel—tionF J. Educ. Psychol.D PP@IAXRS!SSF wáre™h—lD hF @PHHRAF A soil-based approach to rainfall-runo modelling in ungauged catchments for England and WalesF €hh thesisD gr—n(eld …niversityD gr—n(eldD …uF w™gluskeyD eF —nd v—lkhenD eF qF @PHHUAF ƒt—tisti™s ivX snterpreting the results of st—tisti™—l testsF Continuing Education in Anaesthesia, Critical Care PainD U@TAXPHV!PIPF wostellerD pF —nd „ukeyD tF pF @IWTVAF Data Analysis, Including StatisticsF sn qF vindzey —nd iF eronsonD editorsF r—nd˜ook of ƒo™i—l €sy™hologyD †olF ssF eddisonE‡esleyD ‚e—dingD weD …ƒeF x—de—uD gF —nd fengioD ‰F @PHHQAF snferen™e for the gener—liz—tion errorF Mach. Learn.D SP@QAXPQW!PVIF
  64. 64. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es x—shD tF —nd ƒut™lieD tF @IWUHAF ‚iver )ow fore™—sting through ™on™eptu—l models p—rt i!— dis™ussion of prin™iplesF Journal of hydrologyD IH@QAXPVP!PWHF €érezD eFD v—rr—ñ—g—D €FD —nd sFD sF @PHHSAF istim—rD des™omponer y ™omp—r—r el error de m—l— ™l—si(™—™iónF sn Primer Congreso Español de InformáticaF ‚iss—nenD tF @IWUVAF wodeling ˜y the shortest d—t— des™riptionF AutomaticaD IRXRTS!RUIF ‚odríguezD tF hFD €érezD eFD —nd voz—noD tF eF @PHIHAF ƒensitivity —n—lysis of kEfold ™rossEv—lid—tion in predi™tion error estim—tionF IEEE Trans. Pattern Anal. Mach. Intell.D QP@QAXSTW!SUSF ƒ™hirrip—D wF tF —nd gol˜ertD tF tF @PHHTAF snter—nnu—l ™h—nges in s—˜le(sh @enoplopom— (m˜ri—A re™ruitment in rel—tion to o™e—nogr—phi™ ™onditions within the g—liforni— gurrent ƒystemF Fish. Oceanogr.D IS@IAXPS!QTF ƒtoneD wF @IWURAF grossEv—lid—tory ™hoi™e —nd —ssessment of st—tisti™—l predi™tionsF J. Roy. Statistical Society, Series BD QTF ƒtowD gFD tolliD tFD w™qilli™uddy trD hFD honeyD ƒFD ellenD tFD priedri™hsD wFD —nd ‚oseD uF @PHHWAF ƒkill —ssessment for ™oupled ˜iologi™—lGphysi™—l models of m—rine systemsF Journal of Marine SystemsD UT@IEPAXR!ISF „—ylorD uF @PHHIAF ƒumm—rizing multiple —spe™ts of model perform—n™e in — single di—gr—mF J. Geophys. ResD IHT@hUAXUIVQ!UIWPF v—n der q——gD vF gFD ‚enooijD ƒFD ‡ittem—nD gF vF wFD elem—nD fF wF €FD —nd „——lD fF qF @PHHPAF €ro˜—˜ilities for — pro˜—˜ilisti™ networkX — ™—se study in oesoph—ge—l ™—n™erF Artif. Intell. Med.D PS@PAXIPQ!IRVF ‡ikipedi— @PHIIAF e™™ur—™y p—r—doxF ‘ynlineY —™™essed ISEƒeptem˜erEPHII“F ‡ittenD sF rF —nd pr—nkD iF @PHHSAF Data mining: Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, San Francisco, CA, USAF ‰eungD uF ‰FD fumg—rnerD ‚F iFD —nd ‚—fteryD eF iF @PHHSAF f—yesi—n model —ver—gingX development of —n improved multiE™l—ssD gene sele™tion —nd ™l—ssi(™—tion tool for mi™ro—rr—y d—t—F BioinformaticsD PI@IHAXPQWR!PRHPF
  65. 65. i…‚yEfeƒsx „r—ining ‡orkshop on sntrodu™tion to st—tisti™—l modelling toolsD for h—˜it—t models development ‚eferen™es —r—uzD vFD srigoienD ˆFD —nd pern—ndesD tF @PHHVAF wodelling the in)uen™e of —˜ioti™ —nd ˜ioti™ f—™tors on pl—nkton distri˜ution in the f—y of fis™—yD during three ™onse™utive ye—rs @PHHREHTAF J. Plankton Res.D QH@VAXVSUF —r—uzD vFD srigoienD ˆFD —nd pern—ndesD tF @PHHWAF gh—nges in pl—nkton size stru™ture —nd ™ompositionD during the gener—tion of — phytopl—nkton ˜loomD in the ™entr—l g—nt—˜ri—n se—F J. Plankton Res.D QI@PAXIWQ!PHUF huD ˆF —nd h—vidsonD sF @PHHUAF Knowledge discovery and data mining: challenges and realitiesF sgi qlo˜—lF
  66. 66. EURO-­‐BASIN,  www.euro-­‐basin.eu   Introduc)on  to  Sta)s)cal  Modelling  Tools  for  Habitat  Models  Development,  26-­‐28th  Oct  2011  

×