Cross-Project Build Co-change Prediction
•
0 likes•1,795 views
Build systems orchestrate how human-readable source code is translated into executable programs. In a software project, source code changes can induce changes in the build system (aka. build co-changes). It is difficult for developers to identify when build co-changes are necessary due to the complexity of build systems. Prediction of build co-changes works well if there is a sufficient amount of training data to build a model. However, in practice, for new projects, there exists a limited number of changes. Using training data from other projects to predict the build co-changes in a new project can help improve the performance of the build co-change prediction. We refer to this problem as cross-project build co-change prediction. In this paper, we propose CroBuild, a novel cross-project build co-change prediction approach that iteratively learns new classifiers. CroBuild constructs an ensemble of classifiers by iteratively building classifiers and assigning them weights according to its prediction error rate. Given that only a small proportion of code changes are build co-changing, we also propose an imbalance-aware approach that learns a threshold boundary between those code changes that are build co-changing and those that are not in order to construct classifiers in each iteration. To examine the benefits of CroBuild, we perform experiments on 4 large datasets including Mozilla, Eclipse-core, Lucene, and Jazz, comprising a total of 50,884 changes. On average, across the 4 datasets, CroBuild achieves a F1-score of up to 0.408. We also compare CroBuild with other approaches such as a basic model, AdaBoost proposed by Freund et al., and TrAdaBoost proposed by Dai et al.. On average, across the 4 datasets, the CroBuild approach yields an improvement in F1-scores of 41.54%, 36.63%, and 36.97% over the basic model, AdaBoost, and TrAdaBoost, respectively.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Cross-Project Build Co-change Prediction
Similar to Cross-Project Build Co-change Prediction (20)
Recently uploaded
Recently uploaded (20)
Cross-Project Build Co-change Prediction
- 1. Cross-Project Build Co-change Prediction Shane McIntosh Ahmed E. Hassan shanemcintosh@acm.org @shane_mcintosh shanemcintosh.org Emad Shihab David Lo Xin Xia
- 2. What is a build system? Source code 2
- 3. What is a build system? Source code Deliverable 2
- 4. .tex .c .cc .o .o .dvi .a .exe .pdf .deb Build systems describe how sources are translated into deliverables 3
- 5. The build system is at the heart of techniques like Continuous Integration (CI) 4 .c .mk
- 6. The build system is at the heart of techniques like Continuous Integration (CI) Commit 4 Commit 9719cf0 .c .mk
- 7. The build system is at the heart of techniques like Continuous Integration (CI) Commit 4 Build Commit 9719cf0 .c .mk
- 8. The build system is at the heart of techniques like Continuous Integration (CI) Commit 4 Build Test Commit 9719cf0 .c .mk
- 9. The build system is at the heart of techniques like Continuous Integration (CI) Commit 4 Build Test Report Commit 9719cf0 wassuccessfullyintegrated Commit 9719cf0 .c .mk
- 10. The build system is at the heart of techniques like Continuous Integration (CI) Commit 4 Build Test Report Commit 9719cf0 wassuccessfullyintegrated Commit 9719cf0 .c .mk
- 11. “...nothing can be said to be certain, except death and taxes” - Benjamin Franklin The Build “Tax” An Empirical Study of Build Maintenance Effort S. McIntosh, B. Adams, T. H. D. Nguyen, Y. Kamei, A. E. Hassan [ICSE 2011] Up to 27% of source changes require build changes, too! 5
- 12. Neglected build maintenance is a frequent cause of build breakage 6 .c .mk
- 13. Neglected build maintenance is a frequent cause of build breakage Commit 6 Commit aedd38 .c .mk
- 14. Neglected build maintenance is a frequent cause of build breakage Commit 6 Commit aedd38 .c .mk
- 15. Neglected build maintenance is a frequent cause of build breakage Commit 6 Build Commit aedd38 .c .mk
- 16. Neglected build maintenance is a frequent cause of build breakage Commit 6 Build Test Commit aedd38 .c .mk
- 17. Neglected build maintenance is a frequent cause of build breakage Commit 6 Build Test Commit aedd38 .c .mk
- 18. Neglected build maintenance is a frequent cause of build breakage Commit 6 Build Test Report Commit aedd38 .c .mk Commit aedd38 broke the build!
- 19. Neglected build maintenance can even impact end users 7
- 20. Neglected build maintenance can even impact end users 7 Not working due to linking of incorrect SQLite library version
- 21. Neglected build maintenance can even impact end users 7 Not working due to linking of incorrect SQLite library version When are build changes necessary?
- 22. 8 Overview of the studied systems
- 23. 8 Overview of the studied systems 29 years of historical data
- 24. 8 Overview of the studied systems 29 years of historical data Proprietary and opensource systems
- 25. Grouping related changes according to the work items that they address 9
- 26. Grouping related changes according to the work items that they address .c .c .c Changes .mk 9
- 27. Missed code in #2121 Add feature #2121 Fix for bug #1234 Grouping related changes according to the work items that they address .c .c .c Transactions Changes .mk 9
- 28. 2121 Missed code in #2121 Add feature #2121 1234 Fix for bug #1234 Grouping related changes according to the work items that they address .c .c .c Transactions Work items Changes .mk 9
- 29. 1 2 .mk 10 We train classifiers to identify code changes that require build co-changes Work items .c.c .c Classification model Build change necessary No build change necessary
- 30. 1 2 .mk 10 We train classifiers to identify code changes that require build co-changes Work items .c .c .cClassification model Build change necessary No build change necessary
- 31. 1 2 .mk 11 Work items .c Build change necessary No build change necessary Classification model We train classifiers to identify code changes that require build co-changes
- 32. 12 Prior work shows that within-project build co-change prediction can be accurate Mining Co-Change Information to Understand when Build Changes are Necessary S. McIntosh, B. Adams, M. Nagappan, A. E. Hassan [ICSME 2014] Build co-change classifiers can achieve an AUC of 0.60-0.88
- 33. However, a large amount of historical data was used to train the classifiers 13
- 34. However, a large amount of historical data was used to train the classifiers 13
- 35. However, a large amount of historical data was used to train the classifiers 13 What about new projects?
- 36. However, a large amount of historical data was used to train the classifiers 13 What about new projects? …or projects withpoorly-recordedhistorical data?
- 37. However, a large amount of historical data was used to train the classifiers 13 What about new projects? …or projects withpoorly-recordedhistorical data? Can we leverage these large corpora for the small ones?
- 38. 14
- 39. 14 How well do build co- change prediction models perform on sparse data? Precision Recall F1-score AUC 0 0.25 0.5 0.75 1 5% 50% 90%
- 40. 14 How well do build co- change prediction models perform on sparse data? Precision Recall F1-score AUC 0 0.25 0.5 0.75 1 5% 50% 90% Challenge 1: Very small datasets tend to yield models that under-perform
- 41. 14 How well do build co- change prediction models perform on sparse data? Precision Recall F1-score AUC 0 0.25 0.5 0.75 1 5% 50% 90% How well do build co- change prediction models perform on other datasets? Precision Recall F1-score AUC 0 0.25 0.5 0.75 1 Eclipse => Mozilla Jazz => Mozilla Lucene => Mozilla Challenge 1: Very small datasets tend to yield models that under-perform
- 42. 14 How well do build co- change prediction models perform on sparse data? Precision Recall F1-score AUC 0 0.25 0.5 0.75 1 5% 50% 90% How well do build co- change prediction models perform on other datasets? Precision Recall F1-score AUC 0 0.25 0.5 0.75 1 Eclipse => Mozilla Jazz => Mozilla Lucene => Mozilla Challenge 1: Very small datasets tend to yield models that under-perform Challenge 2: Cross-project build co- change models tend to under-perform
- 43. 15 Domain-specific project characteristics may limit the applicability of cross-project models Training corpus Testing corpus
- 44. Training corpus 16 Classification model Testing corpus Domain-specific project characteristics may limit the applicability of cross-project models
- 45. Training corpus 16 Classification model Testing corpus ? Domain-specific project characteristics may limit the applicability of cross-project models
- 46. 17 Using transfer learning to provide some domain knowledge to the training corpus Training corpus Testing corpus
- 47. Move some training data from target system to the training corpus 17 Using transfer learning to provide some domain knowledge to the training corpus Training corpus Testing corpus
- 48. 18 Training corpus Testing corpus Using transfer learning to provide some domain knowledge to the training corpus
- 49. 19 Training corpus Testing corpus Classification model Using transfer learning to provide some domain knowledge to the training corpus
- 50. 19 Training corpus Testing corpus Classification model ? Using transfer learning to provide some domain knowledge to the training corpus
- 51. 20 Challenge 3: Build co-changes are the minority
- 52. 20 Challenge 3: Build co-changes are the minority Only 8%-17% of changesare build co-changing
- 53. 21 Training corpus Testing corpus Use training corpus to find an appropriate threshold
- 54. 22 Training corpus Testing corpus Classification model Use training corpus to find an appropriate threshold
- 55. Set aside the testing corpus 22 Training corpus Testing corpus Classification model Use training corpus to find an appropriate threshold
- 56. 23 Training corpus Classification model Use training corpus to find an appropriate threshold Training corpus
- 57. Incorrectly classified! 23 Training corpus Classification model Use training corpus to find an appropriate threshold Training corpus
- 58. 24 Use training corpus to find an appropriate threshold Training corpus Classification model
- 59. 24 Use training corpus to find an appropriate threshold Training corpus Classification model 1
- 60. 25 Use training corpus to find an appropriate threshold Training corpus Classification model Classification model 1 2
- 61. 25 Use training corpus to find an appropriate threshold Training corpus Classification model Classification model 1 2
- 62. 26 Use training corpus to find an appropriate threshold Classification model Classification model 1 2 … Classification model N
- 63. Ensemble of models used on the testing corpus 26 Use training corpus to find an appropriate threshold Classification model Classification model 1 2 … Classification model N
- 65. 27 Evaluating our approach Relative performance Training configuration sensitivity Source Target
- 66. 28 Evaluating our approach Relative performance Source Target Training configuration sensitivity
- 67. 29 Our approach outperforms baseline cross-project approaches Eclipse Jazz Lucene Mozilla Average 0 0.25 0.5 0.75 1 Our approach Ordinary cross-project AdaBoost TrAdaBoost Worstmeasured F-score
- 68. 29 Our approach outperforms baseline cross-project approaches Eclipse Jazz Lucene Mozilla Average 0 0.25 0.5 0.75 1 Our approach Ordinary cross-project AdaBoost TrAdaBoost Worstmeasured F-score 37%-42% improvement
- 69. 30 Our approach achieves similar results to within-project models Eclipse Jazz Lucene Mozilla Average 0 0.25 0.5 0.75 1 Our approach Within-project Worstmeasured F-score
- 70. 30 Our approach achieves similar results to within-project models Eclipse Jazz Lucene Mozilla Average 0 0.25 0.5 0.75 1 Our approach Within-project Only a 7% drop in performance Worstmeasured F-score
- 71. 31 Evaluating our approach Relative performance Source Target Training configuration sensitivity
- 72. 31 Evaluating our approach Relative performance 37%-42% improvement over baseline Source Target Training configuration sensitivity
- 73. 31 Evaluating our approach Relative performance 37%-42% improvement over baseline Only 7% drop of within-project F-measure Source Target Training configuration sensitivity
- 74. 32 Evaluating our approach Relative performance Source Target 37%-42% improvement over baseline Only 7% drop of within-project F-measure Training configuration sensitivity
- 75. 33 Additional data from the target system slowly improves classifier performance Source Target F-score
- 76. 34 Evaluating our approach Relative performance Source Target 37%-42% improvement over baseline Only 7% drop of within-project F-measure Training configuration sensitivity
- 77. 34 Evaluating our approach Relative performance Source Target 37%-42% improvement over baseline Only 7% drop of within-project F-measure Training configuration sensitivity F-score tends to improve as more target system data becomes available