Upcoming SlideShare
×

# Network vs. Code Metrics to Predict Defects: A Replication Study

1,040

Published on

Published in: Technology
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

Views
Total Views
1,040
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
25
0
Likes
0
Embeds 0
No embeds

No notes for slide
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• \n
• The function computes a ROC curve by first applying a series of cutoffs for each metric and then computing the sensitivity and specificity for each cutoff point. The importance of the metric is then determined by computing the area under the ROC curve.\n
• The function computes a ROC curve by first applying a series of cutoffs for each metric and then computing the sensitivity and specificity for each cutoff point. The importance of the metric is then determined by computing the area under the ROC curve.\n
• The function computes a ROC curve by first applying a series of cutoffs for each metric and then computing the sensitivity and specificity for each cutoff point. The importance of the metric is then determined by computing the area under the ROC curve.\n
• The function computes a ROC curve by first applying a series of cutoffs for each metric and then computing the sensitivity and specificity for each cutoff point. The importance of the metric is then determined by computing the area under the ROC curve.\n
• ### Network vs. Code Metrics to Predict Defects: A Replication Study

1. 1. Network vs. Code Metricsto Predict Defects: A Replication Study Rahul Premraj Kim Herzig VU University Amsterdam Saarland University
2. 2. The Original StudyProceedings of the International Conference on Software Engineering (ICSE, May 2008).
3. 3. The Original StudyBug Database code qualityVersion Archive defect prediction model Source Code code metrics
4. 4. The Original Study modelcode metrics network metrics combined metrics
5. 5. The Original Study modelcode metrics network metrics combined metricsNetwork metrics outperformed code metrics!
7. 7. Contributionsnetwork vs. code metrics
8. 8. Contributions network vs. code metrics• using random sampling on same release (similar to Z&N)
9. 9. Contributions network vs. code metrics• using random sampling on same release (similar to Z&N)• across diﬀerent releases of same project (forward prediction)
10. 10. Contributions network vs. code metrics• using random sampling on same release (similar to Z&N)• across diﬀerent releases of same project (forward prediction)• across diﬀerent projects (cross project prediction)
11. 11. Contributions network vs. code metrics• using random sampling on same release (similar to Z&N)• across diﬀerent releases of same project (forward prediction) ati on• across diﬀerent projects cs itu is ti (cross project prediction) re al
12. 12. Data Collection gs bu se ea e el m t-r code metrics network metrics na os e #p ﬁl ●●● ●●●code ﬁles ●●● combined metrics ●●● ●●●
13. 13. Data Collection gs bu se ea e el m t-r code metrics network metrics na os e #p ﬁl ●●● ●●●code ﬁles ●●● combined metrics ●●● ●●● code metrics network metrics number 9 25 granularity class/method class [1] [2] tools Understand UCINET LoC, NumMethods, FanIn/ Ego-network, structural examples FanOut metrics, centrality [1] Understand, Scientific Toolworks Inc. (Version 2.0, Build 505, http://www.scitools.com/) [2] UCINET: Social Network Analysis Software, Analytic Technologies (http://www.analytictech.com/ucinet/)
14. 14. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
15. 15. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
16. 16. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
17. 17. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
18. 18. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
19. 19. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
20. 20. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
21. 21. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
22. 22. Data Collection: Diﬀerences our study Z&N Language Java C/C++ JRuby, ArgoUML, Projects Windows 2003 Server Eclipse Granularity source ﬁle binary 9 12 Code Metrics aggregated from class/ some speciﬁc for C/C method level ++ ... Tool Understand Microsoft in houseNetwork Metrics mostly the same (minor differences) ... Tool UCINET
23. 23. Experimental Setup
24. 24. One Release Classi cation Stratiﬁed repeated holdout setup • Randomly splitting data set • Preserving the proportion ofOne Release positive and negative instances Data Set • 300 independent training and testing sets • Repeat for code, network and training data (2/3) combined metrics (900 training and testing sets) testing data (1/3)
25. 25. Forward Prediction Closest to real world situation e.g. JRuby 1.0 e.g. JRuby 1.1 release N release N+1 time release N release N+1 training set testing set Repeated for all possible combinations.Releases must be from same project. Testing release must be later than testing release.
26. 26. Cross-Project Prediction Are defect predictions transferable? release M release NProject X Project Y e.g. ArgoUML e.g. Eclipse training set testing set Repeated for all combinations of projects (only one version per project)
27. 27. Results• Reporting prediction measures as box plot• Reporting results of best model• non-parametric statistical test (Kruskal-Wallis)
28. 28. One Release Classi cation JRuby 1.0 JRuby 1.1 ArgoUML 0.24 ArgoUML 0.26 Eclipse 2.1 Eclipse 3.01.00.8 Precision0.60.40.2 (svmRadial) (svmRadial) (svmRadial)0.0 (svmRadial) (svmRadial) (multinom) (svmRadial) (svmRadial) (treebag) (nb) (svmRadial) (treebag) (rpart) (multinom) (multinom) (rpart) (svmRadial) (rpart)1.00.80.6 Recall0.40.20.0 (rpart) (nb) (nb) (treebag) (treebag) (treebag) (nb) (treebag) (nb) (nb) (nb) (nb) (nb) (nb) (nb) (treebag) (treebag) (treebag)1.00.8 F-measure0.60.40.20.0 (rpart) (nb) (treebag) (treebag) (treebag) (treebag) (nb) (treebag) (treebag) (nb) (nb) (nb) (nb) (nb) (nb) (treebag) (treebag) (treebag) Code Network All Code Network All Code Network All Code Network All Code Network All Code Network All
29. 29. One Release Classi cation JRuby 1.0 JRuby 1.1 ArgoUML 0.24 ArgoUML 0.26 Eclipse 2.1 Eclipse 3.01.00.8 Precision0.60.40.2 (svmRadial) •Network metrics outperform code (svmRadial) (svmRadial)0.01.0 (svmRadial) (svmRadial) metrics! (multinom) (svmRadial) (svmRadial) (treebag) (nb) (svmRadial) (treebag) (rpart) (multinom) (multinom) (rpart) (svmRadial) (rpart)0.80.6 •Using all metrics together offers no Recall0.40.2 improvement!0.0 •Higher accuracy for the smaller (rpart) (nb) (nb) (treebag) (treebag) (treebag) (nb) (treebag) (nb) (nb) (nb) (nb) (nb) (nb) (nb) (treebag) (treebag) (treebag)1.00.8 projects in comparison to Eclipse! F-measure0.60.40.20.0 (rpart) (nb) (treebag) (treebag) (treebag) (treebag) (nb) (treebag) (treebag) (nb) (nb) (nb) (nb) (nb) (nb) (treebag) (treebag) (treebag) Code Network All Code Network All Code Network All Code Network All Code Network All Code Network All
30. 30. Forward Prediction Code Network All JRuby 1.0 to predict 1.1 ArgoUML 0.24 to predict 0.26 Eclipse 2.1 to predict 3.01.00.80.60.40.20.0 Precision Recall F-measure Precision Recall F-measure Precision Recall F-measure
31. 31. Forward Prediction Code Network All JRuby 1.0 to predict 1.1 ArgoUML 0.24 to predict 0.26 Eclipse 2.1 to predict 3.0 1.0 0.8 0.6 0.4 0.2 0.0 Precision Recall F-measure Precision Recall F-measure Precision Recall F-measure JRuby ArgoUML EclipseNetwork vs. better recall worse recall CodeAll vs. Code worse recall better recall worse recall & F- measure
32. 32. Forward Prediction Code Network All JRuby 1.0 to predict 1.1 ArgoUML 0.24 to predict 0.26 Eclipse 2.1 to predict 3.0 1.0 0.8 0.6 0.4 0.2 0.0 Precision Recall F-measure Precision Recall F-measure Precision Recall F-measure JRuby ArgoUML Eclipse All three metrics sets appear to haveNetwork vs. comparably prediction accuracy. better recall worse recall Code (no statistically signiﬁcant differences: ANOVA test)All vs. Code worse recall better recall worse recall & F- measure
33. 33. Cross-Project Prediction Code Network All Train: JRuby 1.1 Train: ArgoUML 0.26 Train: Eclipse 3.0 1.0 0.8 Test: JRuby 1.1 0.6 0.4 0.2 0.0 1.0 Test: ArgoUML 0.26 0.8 0.6 0.4 0.2 0.0 1.0 0.8 Test: Eclipse 3.0 0.6 0.4 0.2 0.0 Precision Recall F-measure Precision Recall F-measure Precision Recall F-measure
34. 34. Cross-Project Prediction Code Network All Train: JRuby 1.1 Train: ArgoUML 0.26 Train: Eclipse 3.0 1.0 0.8 Test: JRuby 1.1 0.6 0.4 0.2 •Combined metrics do not work well! 0.0 1.0 Test: ArgoUML 0.26 0.8 0.6 •Except Eclipse predicting Jruby no 0.4 0.2 statistical difference (ANOVA test) 0.0 1.0 0.8 Test: Eclipse 3.0 0.6 0.4 0.2 0.0 Precision Recall F-measure Precision Recall F-measure Precision Recall F-measure
35. 35. In uencial Metrics Measured by area under ROC curve using the combined metrics set• 4/6 cases, all top 10 metrics were network metrics ‣ Except: JRuby 1.0, ArgoUML 0.26, and Eclipse 2.1• No pattern with respect to presence or ranking
36. 36. [1] [2] Z&N our study Bird et al. Tosun et al. Language C/C++ Java C/C++, Java C/C++, Java Granularity Binary File Package File Network vs. Code metrics using ... one release prediction ▼ forward prediction ▼ cross-project prediction ▼ Network metrics performance with respect to ... project size ▼[1] Bird, C., Nagappan, N., Gall, H. and Murphy, B. 2009. Putting It All Together: Using Socio-technical Networks to Predict Failures. In Proceedings of the 20th International Symposium on Software Reliability Engineering (ISSRE 2009)[2] Tosun, A., Burak Turhan and Bener, A. 2009. Validation of network measures as indicators of defective modules in software systems. In Proceedings of the 5th International Conference on Predictor Models in Software Engineering (PROMISE 09)
37. 37. [1] [2] Z&N our study Bird et al. Tosun et al. Language C/C++ Java C/C++, Java C/C++, Java Granularity Binary File Package File Network vs. Code metrics using ... one release prediction Code metrics might be preferable because: ▼ • more easy to collect ▼ forward prediction • cross-projectfewer in numbers prediction ▼ • faster to train prediction model with respect to ... Network metrics performance project size ▼[1] Bird, C., Nagappan, N., Gall, H. and Murphy, B. 2009. Putting It All Together: Using Socio-technical Networks to Predict Failures. In Proceedings of the 20th International Symposium on Software Reliability Engineering (ISSRE 2009)[2] Tosun, A., Burak Turhan and Bener, A. 2009. Validation of network measures as indicators of defective modules in software systems. In Proceedings of the 5th International Conference on Predictor Models in Software Engineering (PROMISE 09)
1. #### A particular slide catching your eye?

Clipping is a handy way to collect important slides you want to go back to later.