SlideShare a Scribd company logo

Adequate and Precise Evaluation of Predictive Models in Software Engineering Studies

•Download as PPT, PDF•
•
Tim Menzies
Tim Menzies

Yan Ma and Bojan Cukic

Economy & FinanceTechnology
1 of 19
Yan Ma, Bojan Cukic Lane Department of Computer Science and Electrical Engineering West Virginia University May 2007 Adequate Evaluation of Quality Models in Software Engineering Studies CITeR The Center for Identification Technology Research www.citer.wvu.edu An NSF I/UCR Center advancing integrative research
Evaluating Defect Models ,[object Object],[object Object],[object Object],[object Object],[object Object]
Software defect data: Class Imbalance ,[object Object],[object Object]
Software Defect Data: Correlation MDP-PC1: Pearson correlation coefficients 1.000 0.473 0.982 0.987 0.996 0.924 N 0.473 1.000 0.468 0.468 0.464 0.545 LCC 0.982 0.468 1.000 0.995 0.971 0.931 B 0.987 0.468 0.995 1.000 0.976 0.937 V 0.996 0.464 0.971 0.976 1.000 0.908 TOpnd 0.924 0.545 0.931 0.937 0.908 1.000 LOC N LCC B V TOpnd LOC
Software Defect Data: Correlation (2) MDP-KC2: Pearson correlation coefficients 1.000 0.912 0.836 0.887 0.636 0.909 LOB 0.912 1.000 0.972 0.990 0.615 0.991 Top 0.836 0.972 1.000 0.970 0.577 0.968 IV.G 0.887 0.990 0.970 1.000 0.536 0.986 V 0.636 0.615 0.577 0.536 1.000 0.632 UOp 0.909 0.991 0.968 0.986 0.632 1.000 LOC LOB TOp IV.G V UOp LOC
Software Defect Data: Correlation (3) ,[object Object]
Software Defect Data: Module Size ,[object Object],[object Object],The 90 th percentile of LOC for the collection of defect modules and defect-free modules 131 165 114 167 99 Defect 55 72 47 55 42 Defect-free CM1 JM1 PC1 KC2 KC1
Software Defect Data: Close Neighbors ,[object Object],[object Object],97.96% 73.47% CM1 85.71% 75.32% PC1 75.46% 67.90% JM1 58.33% 58.33% KC2 73.62% 66.26% KC1 % of defect modules that has  2 among the three nearest neighbors in the majority class % of defect modules whose nearest neighbor is a majority class instance Project
Implications on Evaluation ,[object Object],[object Object],[object Object]
Classification Success Measures ,[object Object],[object Object],[object Object],[object Object],[object Object],Random Forests on PC1 (only 7% modules faulty )
Success Measures (2) ,[object Object],[object Object],The geometric mean of the two accuracies. Higher precision leads to “cheaper” V&V
Success Measures (3) ,[object Object],[object Object],[object Object],[object Object]
Comparing Models ,[object Object],[object Object],0.527 0.305 F-measure (  =2) 0.368 0.372 F-measure (  =1) 0.783 0.519 G-mean 2 0.426 0.399 G-mean 1 Figure 1. (b) Figure 1. (a) PC1: random forests at different voting cutoffs
Comparing Performance MDP-PC1 0.588 0.542 0.396 0.661 0.498 ED (  = 0.5) 0.200 0.264 0.436 0.077 0.290 F-measure (  = 2) 0.275 0.327 0.428 0.106 0.278 F-measure (  = 1) 0.410 0.480 0.649 0.253 0.529 G-mean 2 0.352 0.356 0.428 0.137 0.278 G-mean 1 0.938 0.933 0.918 0.924 0.892 Overall Accuracy 0.732 0.540 0.415 0.289 0.259 Precision 0.995 0.985 0.954 0.988 0.936 1 – PF 0.169 0.234 0.442 0.065 0.299 PD Bagging J48 IB1 Logistic Naïve Bayes Indices
Comparing Performance: KC-2 0.433 0.375 0.409 0.500 0.492 ED (  = 0.67) 0.377 0.329 0.361 0.435 0.427 ED (  = 0.5) 0.498 0.552 0.504 0.418 0.434 F-measure (  = 2) 0.543 0.562 0.496 0.472 0.501 F-measure (  = 1) 0.663 0.700 0.661 0.602 0.615 G-mean 2 0.549 0.562 0.496 0.483 0.518 G-mean 1 0.836 0.824 0.786 0.820 0.836 Overall Accuracy 0.639 0.578 0.483 0.599 0.674 Precision 0.931 0.896 0.858 0.932 0.950 1 - PF 0.472 0.546 0.509 0.389 0.398 PD Bagging J48 IB1 Logistic Naïve Bayes Indices
Visual Tools: Margin Plots
Visual Tools: ROC ,[object Object],[object Object],[object Object],Factor  depends of the misclassification cost.
Summary ,[object Object],[object Object],[object Object]
Current work ,[object Object],[object Object]

Recommended

COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING
COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING
COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING Pratyusha Mahavadi
 
Comparison of optimization technique of power system stabilizer by using gea
Comparison of optimization technique of power system stabilizer by using geaComparison of optimization technique of power system stabilizer by using gea
Comparison of optimization technique of power system stabilizer by using geaIAEME Publication
 
samir_jain_UROP_poster_spr14
samir_jain_UROP_poster_spr14samir_jain_UROP_poster_spr14
samir_jain_UROP_poster_spr14Samir Jain
 
Software engineering lecture notes
Software engineering lecture notesSoftware engineering lecture notes
Software engineering lecture notesSiva Ayyakutti
 
FallThesisFinal
FallThesisFinalFallThesisFinal
FallThesisFinalLeora Juster
 
Personality prediction from network structure
Personality prediction from network structurePersonality prediction from network structure
Personality prediction from network structuremrcmilano
 
Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...
Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...
Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...eHealth Africa
 
Comparing Machine Learning Algorithms in Text Mining
Comparing Machine Learning Algorithms in Text MiningComparing Machine Learning Algorithms in Text Mining
Comparing Machine Learning Algorithms in Text MiningAndrea Gigli
 

Recommended

COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING
COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING
COMPUTING THE GROWTH RATE OF STEM CELLS USING DIGITAL IMAGE PROCESSING Pratyusha Mahavadi
 
Comparison of optimization technique of power system stabilizer by using gea
Comparison of optimization technique of power system stabilizer by using geaComparison of optimization technique of power system stabilizer by using gea
Comparison of optimization technique of power system stabilizer by using geaIAEME Publication
 
samir_jain_UROP_poster_spr14
samir_jain_UROP_poster_spr14samir_jain_UROP_poster_spr14
samir_jain_UROP_poster_spr14Samir Jain
 
Software engineering lecture notes
Software engineering lecture notesSoftware engineering lecture notes
Software engineering lecture notesSiva Ayyakutti
 
FallThesisFinal
FallThesisFinalFallThesisFinal
FallThesisFinalLeora Juster
 
Personality prediction from network structure
Personality prediction from network structurePersonality prediction from network structure
Personality prediction from network structuremrcmilano
 
Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...
Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...
Kano GIS Day 2014 - The Application of Multivariate Geostatistical analyses i...eHealth Africa
 
Comparing Machine Learning Algorithms in Text Mining
Comparing Machine Learning Algorithms in Text MiningComparing Machine Learning Algorithms in Text Mining
Comparing Machine Learning Algorithms in Text MiningAndrea Gigli
 
Prediction-based Model Selection in PLS-PM
Prediction-based Model Selection in PLS-PMPrediction-based Model Selection in PLS-PM
Prediction-based Model Selection in PLS-PMGalit Shmueli
 
Improving Hardware Efficiency for DNN Applications
Improving Hardware Efficiency for DNN ApplicationsImproving Hardware Efficiency for DNN Applications
Improving Hardware Efficiency for DNN ApplicationsChester Chen
 
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...IRJET Journal
 
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...IRJET Journal
 
Machine-learning scoring functions for molecular docking
Machine-learning scoring functions for molecular dockingMachine-learning scoring functions for molecular docking
Machine-learning scoring functions for molecular dockingPedro Ballester
 
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Sung Kim
 
annInstance28Nov6pm
annInstance28Nov6pmannInstance28Nov6pm
annInstance28Nov6pmXunhu (Tiger) Sun
 
A Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality IndicatorsA Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality Indicatorsvie_dels
 
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO ecij
 
MultiModal Retrieval Image
MultiModal Retrieval ImageMultiModal Retrieval Image
MultiModal Retrieval ImageKonstantinos Zagoris
 
CDAC 2018 Pellegrini clustering ppi networks
CDAC 2018 Pellegrini clustering ppi networksCDAC 2018 Pellegrini clustering ppi networks
CDAC 2018 Pellegrini clustering ppi networksMarco Antoniotti
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)Fellowship at Vodafone FutureLab
 
Bayesian Autoencoders for anomaly detection in industrial environments
Bayesian Autoencoders for anomaly detection in industrial environmentsBayesian Autoencoders for anomaly detection in industrial environments
Bayesian Autoencoders for anomaly detection in industrial environmentsBang Xiang Yong
 
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...CS, NcState
 
Fuzzy Logic Based Decision System For PCB Defects Correction
Fuzzy Logic Based Decision System For PCB Defects CorrectionFuzzy Logic Based Decision System For PCB Defects Correction
Fuzzy Logic Based Decision System For PCB Defects CorrectionIJERA Editor
 
ADPD poster 2020 Cogstate in ADNI-3
ADPD poster 2020 Cogstate in ADNI-3ADPD poster 2020 Cogstate in ADNI-3
ADPD poster 2020 Cogstate in ADNI-3Chris Edgar
 
Pathogen Detection with Brewster's Angle Straddle Interferometry
Pathogen Detection with Brewster's Angle Straddle InterferometryPathogen Detection with Brewster's Angle Straddle Interferometry
Pathogen Detection with Brewster's Angle Straddle InterferometryLauren Brownlee
 
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDPAN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDPIRJET Journal
 
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...IJERD Editor
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)Fellowship at Vodafone FutureLab
 
Modeling the Effect of Size of Defect Proneness for Open-Source Software
Modeling the Effect of Size of Defect Proneness for Open-Source SoftwareModeling the Effect of Size of Defect Proneness for Open-Source Software
Modeling the Effect of Size of Defect Proneness for Open-Source SoftwareTim Menzies
 
Assessing the Reliability of a Human Estimator
Assessing the Reliability of a Human EstimatorAssessing the Reliability of a Human Estimator
Assessing the Reliability of a Human EstimatorTim Menzies
 

More Related Content

Similar to Adequate and Precise Evaluation of Predictive Models in Software Engineering Studies

Prediction-based Model Selection in PLS-PM
Prediction-based Model Selection in PLS-PMPrediction-based Model Selection in PLS-PM
Prediction-based Model Selection in PLS-PMGalit Shmueli
 
Improving Hardware Efficiency for DNN Applications
Improving Hardware Efficiency for DNN ApplicationsImproving Hardware Efficiency for DNN Applications
Improving Hardware Efficiency for DNN ApplicationsChester Chen
 
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...IRJET Journal
 
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...IRJET Journal
 
Machine-learning scoring functions for molecular docking
Machine-learning scoring functions for molecular dockingMachine-learning scoring functions for molecular docking
Machine-learning scoring functions for molecular dockingPedro Ballester
 
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Sung Kim
 
A Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality IndicatorsA Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality Indicatorsvie_dels
 
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO ecij
 
CDAC 2018 Pellegrini clustering ppi networks
CDAC 2018 Pellegrini clustering ppi networksCDAC 2018 Pellegrini clustering ppi networks
CDAC 2018 Pellegrini clustering ppi networksMarco Antoniotti
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)Fellowship at Vodafone FutureLab
 
Bayesian Autoencoders for anomaly detection in industrial environments
Bayesian Autoencoders for anomaly detection in industrial environmentsBayesian Autoencoders for anomaly detection in industrial environments
Bayesian Autoencoders for anomaly detection in industrial environmentsBang Xiang Yong
 
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...CS, NcState
 
Fuzzy Logic Based Decision System For PCB Defects Correction
Fuzzy Logic Based Decision System For PCB Defects CorrectionFuzzy Logic Based Decision System For PCB Defects Correction
Fuzzy Logic Based Decision System For PCB Defects CorrectionIJERA Editor
 
ADPD poster 2020 Cogstate in ADNI-3
ADPD poster 2020 Cogstate in ADNI-3ADPD poster 2020 Cogstate in ADNI-3
ADPD poster 2020 Cogstate in ADNI-3Chris Edgar
 
Pathogen Detection with Brewster's Angle Straddle Interferometry
Pathogen Detection with Brewster's Angle Straddle InterferometryPathogen Detection with Brewster's Angle Straddle Interferometry
Pathogen Detection with Brewster's Angle Straddle InterferometryLauren Brownlee
 
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDPAN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDPIRJET Journal
 
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...IJERD Editor
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)Fellowship at Vodafone FutureLab
 

Similar to Adequate and Precise Evaluation of Predictive Models in Software Engineering Studies (20)

Prediction-based Model Selection in PLS-PM
Prediction-based Model Selection in PLS-PMPrediction-based Model Selection in PLS-PM
Prediction-based Model Selection in PLS-PM
 
Improving Hardware Efficiency for DNN Applications
Improving Hardware Efficiency for DNN ApplicationsImproving Hardware Efficiency for DNN Applications
Improving Hardware Efficiency for DNN Applications
 
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
IRJET - Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence M...
 
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
IRJET- Plant Leaf Disease Diagnosis from Color Imagery using Co-Occurrence Ma...
 
Machine-learning scoring functions for molecular docking
Machine-learning scoring functions for molecular dockingMachine-learning scoring functions for molecular docking
Machine-learning scoring functions for molecular docking
 
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)Heterogeneous Defect Prediction (

ESEC/FSE 2015)
Heterogeneous Defect Prediction (

ESEC/FSE 2015)
 
annInstance28Nov6pm
annInstance28Nov6pmannInstance28Nov6pm
annInstance28Nov6pm
 
A Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality IndicatorsA Validation of Object-Oriented Design Metrics as Quality Indicators
A Validation of Object-Oriented Design Metrics as Quality Indicators
 
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
ASSESSMENT OF INTRICATE DG PLANNING WITH PRACTICAL LOAD MODELS BY USING PSO
 
MultiModal Retrieval Image
MultiModal Retrieval ImageMultiModal Retrieval Image
MultiModal Retrieval Image
 
CDAC 2018 Pellegrini clustering ppi networks
CDAC 2018 Pellegrini clustering ppi networksCDAC 2018 Pellegrini clustering ppi networks
CDAC 2018 Pellegrini clustering ppi networks
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
 
Bayesian Autoencoders for anomaly detection in industrial environments
Bayesian Autoencoders for anomaly detection in industrial environmentsBayesian Autoencoders for anomaly detection in industrial environments
Bayesian Autoencoders for anomaly detection in industrial environments
 
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
Promise 2011: "Local Bias and its Impacts on the Performance of Parametric Es...
 
Fuzzy Logic Based Decision System For PCB Defects Correction
Fuzzy Logic Based Decision System For PCB Defects CorrectionFuzzy Logic Based Decision System For PCB Defects Correction
Fuzzy Logic Based Decision System For PCB Defects Correction
 
ADPD poster 2020 Cogstate in ADNI-3
ADPD poster 2020 Cogstate in ADNI-3ADPD poster 2020 Cogstate in ADNI-3
ADPD poster 2020 Cogstate in ADNI-3
 
Pathogen Detection with Brewster's Angle Straddle Interferometry
Pathogen Detection with Brewster's Angle Straddle InterferometryPathogen Detection with Brewster's Angle Straddle Interferometry
Pathogen Detection with Brewster's Angle Straddle Interferometry
 
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDPAN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
AN EFFICIENT FACE RECOGNITION EMPLOYING SVM AND BU-LDP
 
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
IJERD (www.ijerd.com) International Journal of Engineering Research and Devel...
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
 

More from Tim Menzies

Modeling the Effect of Size of Defect Proneness for Open-Source Software
Modeling the Effect of Size of Defect Proneness for Open-Source SoftwareModeling the Effect of Size of Defect Proneness for Open-Source Software
Modeling the Effect of Size of Defect Proneness for Open-Source SoftwareTim Menzies
 
Assessing the Reliability of a Human Estimator
Assessing the Reliability of a Human EstimatorAssessing the Reliability of a Human Estimator
Assessing the Reliability of a Human EstimatorTim Menzies
 
Experiments on Design Pattern Discovery
Experiments on Design Pattern DiscoveryExperiments on Design Pattern Discovery
Experiments on Design Pattern DiscoveryTim Menzies
 
Using Developer Information as a Prediction Factor
Using Developer Information as a Prediction FactorUsing Developer Information as a Prediction Factor
Using Developer Information as a Prediction FactorTim Menzies
 
Project Data Incorporating Qualitative Factors for Improved Software Defect P...
Project Data Incorporating Qualitative Factors for Improved Software Defect P...Project Data Incorporating Qualitative Factors for Improved Software Defect P...
Project Data Incorporating Qualitative Factors for Improved Software Defect P...Tim Menzies
 
Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...
Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...
Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...Tim Menzies
 
Complexity Measures for Secure Service-Orieted Software Architectures
Complexity Measures for Secure Service-Orieted Software ArchitecturesComplexity Measures for Secure Service-Orieted Software Architectures
Complexity Measures for Secure Service-Orieted Software ArchitecturesTim Menzies
 
Decision Support Analyss for Software Effort Estimation by Analogy
Decision Support Analyss for Software Effort Estimation by AnalogyDecision Support Analyss for Software Effort Estimation by Analogy
Decision Support Analyss for Software Effort Estimation by AnalogyTim Menzies
 
Predicting Defects for Eclipse
Predicting Defects for EclipsePredicting Defects for Eclipse
Predicting Defects for EclipseTim Menzies
 

More from Tim Menzies (10)

Modeling the Effect of Size of Defect Proneness for Open-Source Software
Modeling the Effect of Size of Defect Proneness for Open-Source SoftwareModeling the Effect of Size of Defect Proneness for Open-Source Software
Modeling the Effect of Size of Defect Proneness for Open-Source Software
 
Assessing the Reliability of a Human Estimator
Assessing the Reliability of a Human EstimatorAssessing the Reliability of a Human Estimator
Assessing the Reliability of a Human Estimator
 
Experiments on Design Pattern Discovery
Experiments on Design Pattern DiscoveryExperiments on Design Pattern Discovery
Experiments on Design Pattern Discovery
 
Using Developer Information as a Prediction Factor
Using Developer Information as a Prediction FactorUsing Developer Information as a Prediction Factor
Using Developer Information as a Prediction Factor
 
Project Data Incorporating Qualitative Factors for Improved Software Defect P...
Project Data Incorporating Qualitative Factors for Improved Software Defect P...Project Data Incorporating Qualitative Factors for Improved Software Defect P...
Project Data Incorporating Qualitative Factors for Improved Software Defect P...
 
Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...
Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...
Make the Most of Your Time: How Should the Analyst Work with Automated Tracea...
 
Promise Keynote
Promise KeynotePromise Keynote
Promise Keynote
 
Complexity Measures for Secure Service-Orieted Software Architectures
Complexity Measures for Secure Service-Orieted Software ArchitecturesComplexity Measures for Secure Service-Orieted Software Architectures
Complexity Measures for Secure Service-Orieted Software Architectures
 
Decision Support Analyss for Software Effort Estimation by Analogy
Decision Support Analyss for Software Effort Estimation by AnalogyDecision Support Analyss for Software Effort Estimation by Analogy
Decision Support Analyss for Software Effort Estimation by Analogy
 
Predicting Defects for Eclipse
Predicting Defects for EclipsePredicting Defects for Eclipse
Predicting Defects for Eclipse
 

Recently uploaded

How Automation is Driving Efficiency Through the Last Mile of Reporting
How Automation is Driving Efficiency Through the Last Mile of ReportingHow Automation is Driving Efficiency Through the Last Mile of Reporting
How Automation is Driving Efficiency Through the Last Mile of ReportingAggregage
 
Call Girls In Yusuf Sarai Women Seeking Men 9654467111
Call Girls In Yusuf Sarai Women Seeking Men 9654467111Call Girls In Yusuf Sarai Women Seeking Men 9654467111
Call Girls In Yusuf Sarai Women Seeking Men 9654467111Sapana Sha
 
Bladex Earnings Call Presentation 1Q2024
Bladex Earnings Call Presentation 1Q2024Bladex Earnings Call Presentation 1Q2024
Bladex Earnings Call Presentation 1Q2024Bladex
 
VIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With Room
VIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With RoomVIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With Room
VIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With Roomdivyansh0kumar0
 
Classical Theory of Macroeconomics by Adam Smith
Classical Theory of Macroeconomics by Adam SmithClassical Theory of Macroeconomics by Adam Smith
Classical Theory of Macroeconomics by Adam SmithAdamYassin2
 
Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...
Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...
Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...shivangimorya083
 
Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...
Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...
Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...makika9823
 
government_intervention_in_business_ownership[1].pdf
government_intervention_in_business_ownership[1].pdfgovernment_intervention_in_business_ownership[1].pdf
government_intervention_in_business_ownership[1].pdfshaunmashale756
 
Lundin Gold April 2024 Corporate Presentation v4.pdf
Lundin Gold April 2024 Corporate Presentation v4.pdfLundin Gold April 2024 Corporate Presentation v4.pdf
Lundin Gold April 2024 Corporate Presentation v4.pdfAdnet Communications
 
OAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptx
OAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptxOAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptx
OAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptxhiddenlevers
 
AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...
AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...
AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...yordanosyohannes2
 
SBP-Market-Operations and market managment
SBP-Market-Operations and market managmentSBP-Market-Operations and market managment
SBP-Market-Operations and market managmentfactical
 
Log your LOA pain with Pension Lab's brilliant campaign
Log your LOA pain with Pension Lab's brilliant campaignLog your LOA pain with Pension Lab's brilliant campaign
Log your LOA pain with Pension Lab's brilliant campaignHenry Tapper
 
High Class Call Girls Nashik Maya 7001305949 Independent Escort Service Nashik
High Class Call Girls Nashik Maya 7001305949 Independent Escort Service NashikHigh Class Call Girls Nashik Maya 7001305949 Independent Escort Service Nashik
High Class Call Girls Nashik Maya 7001305949 Independent Escort Service NashikCall Girls in Nagpur High Profile
 
Chapter 2.ppt of macroeconomics by mankiw 9th edition
Chapter 2.ppt of macroeconomics by mankiw 9th editionChapter 2.ppt of macroeconomics by mankiw 9th edition
Chapter 2.ppt of macroeconomics by mankiw 9th editionMuhammadHusnain82237
 
VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...
VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...
VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...Suhani Kapoor
 
Call Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur Escorts
Call Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur EscortsCall Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur Escorts
Call Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur Escortsranjana rawat
 
Instant Issue Debit Cards - High School Spirit
Instant Issue Debit Cards - High School SpiritInstant Issue Debit Cards - High School Spirit
Instant Issue Debit Cards - High School Spiritegoetzinger
 

Recently uploaded (20)

How Automation is Driving Efficiency Through the Last Mile of Reporting
How Automation is Driving Efficiency Through the Last Mile of ReportingHow Automation is Driving Efficiency Through the Last Mile of Reporting
How Automation is Driving Efficiency Through the Last Mile of Reporting
 
Call Girls In Yusuf Sarai Women Seeking Men 9654467111
Call Girls In Yusuf Sarai Women Seeking Men 9654467111Call Girls In Yusuf Sarai Women Seeking Men 9654467111
Call Girls In Yusuf Sarai Women Seeking Men 9654467111
 
Bladex Earnings Call Presentation 1Q2024
Bladex Earnings Call Presentation 1Q2024Bladex Earnings Call Presentation 1Q2024
Bladex Earnings Call Presentation 1Q2024
 
🔝9953056974 🔝Call Girls In Dwarka Escort Service Delhi NCR
🔝9953056974 🔝Call Girls In Dwarka Escort Service Delhi NCR🔝9953056974 🔝Call Girls In Dwarka Escort Service Delhi NCR
🔝9953056974 🔝Call Girls In Dwarka Escort Service Delhi NCR
 
VIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With Room
VIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With RoomVIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With Room
VIP Kolkata Call Girl Jodhpur Park 👉 8250192130 Available With Room
 
Classical Theory of Macroeconomics by Adam Smith
Classical Theory of Macroeconomics by Adam SmithClassical Theory of Macroeconomics by Adam Smith
Classical Theory of Macroeconomics by Adam Smith
 
Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...
Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...
Russian Call Girls In Gtb Nagar (Delhi) 9711199012 💋✔💕😘 Naughty Call Girls Se...
 
Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...
Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...
Independent Lucknow Call Girls 8923113531WhatsApp Lucknow Call Girls make you...
 
government_intervention_in_business_ownership[1].pdf
government_intervention_in_business_ownership[1].pdfgovernment_intervention_in_business_ownership[1].pdf
government_intervention_in_business_ownership[1].pdf
 
Lundin Gold April 2024 Corporate Presentation v4.pdf
Lundin Gold April 2024 Corporate Presentation v4.pdfLundin Gold April 2024 Corporate Presentation v4.pdf
Lundin Gold April 2024 Corporate Presentation v4.pdf
 
OAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptx
OAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptxOAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptx
OAT_RI_Ep19 WeighingTheRisks_Apr24_TheYellowMetal.pptx
 
AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...
AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...
AfRESFullPaper22018EmpiricalPerformanceofRealEstateInvestmentTrustsandShareho...
 
SBP-Market-Operations and market managment
SBP-Market-Operations and market managmentSBP-Market-Operations and market managment
SBP-Market-Operations and market managment
 
Log your LOA pain with Pension Lab's brilliant campaign
Log your LOA pain with Pension Lab's brilliant campaignLog your LOA pain with Pension Lab's brilliant campaign
Log your LOA pain with Pension Lab's brilliant campaign
 
High Class Call Girls Nashik Maya 7001305949 Independent Escort Service Nashik
High Class Call Girls Nashik Maya 7001305949 Independent Escort Service NashikHigh Class Call Girls Nashik Maya 7001305949 Independent Escort Service Nashik
High Class Call Girls Nashik Maya 7001305949 Independent Escort Service Nashik
 
Chapter 2.ppt of macroeconomics by mankiw 9th edition
Chapter 2.ppt of macroeconomics by mankiw 9th editionChapter 2.ppt of macroeconomics by mankiw 9th edition
Chapter 2.ppt of macroeconomics by mankiw 9th edition
 
Commercial Bank Economic Capsule - April 2024
Commercial Bank Economic Capsule - April 2024Commercial Bank Economic Capsule - April 2024
Commercial Bank Economic Capsule - April 2024
 
VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...
VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...
VIP Call Girls LB Nagar ( Hyderabad ) Phone 8250192130 | ₹5k To 25k With Room...
 
Call Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur Escorts
Call Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur EscortsCall Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur Escorts
Call Girls Service Nagpur Maya Call 7001035870 Meet With Nagpur Escorts
 
Instant Issue Debit Cards - High School Spirit
Instant Issue Debit Cards - High School SpiritInstant Issue Debit Cards - High School Spirit
Instant Issue Debit Cards - High School Spirit
 

Adequate and Precise Evaluation of Predictive Models in Software Engineering Studies

  • 1. Yan Ma, Bojan Cukic Lane Department of Computer Science and Electrical Engineering West Virginia University May 2007 Adequate Evaluation of Quality Models in Software Engineering Studies CITeR The Center for Identification Technology Research www.citer.wvu.edu An NSF I/UCR Center advancing integrative research
  • 2.
  • 3.
  • 4. Software Defect Data: Correlation MDP-PC1: Pearson correlation coefficients 1.000 0.473 0.982 0.987 0.996 0.924 N 0.473 1.000 0.468 0.468 0.464 0.545 LCC 0.982 0.468 1.000 0.995 0.971 0.931 B 0.987 0.468 0.995 1.000 0.976 0.937 V 0.996 0.464 0.971 0.976 1.000 0.908 TOpnd 0.924 0.545 0.931 0.937 0.908 1.000 LOC N LCC B V TOpnd LOC
  • 5. Software Defect Data: Correlation (2) MDP-KC2: Pearson correlation coefficients 1.000 0.912 0.836 0.887 0.636 0.909 LOB 0.912 1.000 0.972 0.990 0.615 0.991 Top 0.836 0.972 1.000 0.970 0.577 0.968 IV.G 0.887 0.990 0.970 1.000 0.536 0.986 V 0.636 0.615 0.577 0.536 1.000 0.632 UOp 0.909 0.991 0.968 0.986 0.632 1.000 LOC LOB TOp IV.G V UOp LOC
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14. Comparing Performance MDP-PC1 0.588 0.542 0.396 0.661 0.498 ED (  = 0.5) 0.200 0.264 0.436 0.077 0.290 F-measure (  = 2) 0.275 0.327 0.428 0.106 0.278 F-measure (  = 1) 0.410 0.480 0.649 0.253 0.529 G-mean 2 0.352 0.356 0.428 0.137 0.278 G-mean 1 0.938 0.933 0.918 0.924 0.892 Overall Accuracy 0.732 0.540 0.415 0.289 0.259 Precision 0.995 0.985 0.954 0.988 0.936 1 – PF 0.169 0.234 0.442 0.065 0.299 PD Bagging J48 IB1 Logistic Naïve Bayes Indices
  • 15. Comparing Performance: KC-2 0.433 0.375 0.409 0.500 0.492 ED (  = 0.67) 0.377 0.329 0.361 0.435 0.427 ED (  = 0.5) 0.498 0.552 0.504 0.418 0.434 F-measure (  = 2) 0.543 0.562 0.496 0.472 0.501 F-measure (  = 1) 0.663 0.700 0.661 0.602 0.615 G-mean 2 0.549 0.562 0.496 0.483 0.518 G-mean 1 0.836 0.824 0.786 0.820 0.836 Overall Accuracy 0.639 0.578 0.483 0.599 0.674 Precision 0.931 0.896 0.858 0.932 0.950 1 - PF 0.472 0.546 0.509 0.389 0.398 PD Bagging J48 IB1 Logistic Naïve Bayes Indices
  • 17.
  • 18.
  • 19.