Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Impact of Code Review Coverage and Participation on Software Quality

1,845 views

Published on

Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.

Published in: Software, Technology, Business
  • Be the first to comment

The Impact of Code Review Coverage and Participation on Software Quality

  1. 1. The Impact of Code Review Coverage and Participation on Software Quality Shane McIntosh Ahmed E. Hassan Bram Adams Yasutaka Kamei @shane_mcintosh shanemcintosh@acm.org
  2. 2. Patch + + - - Patch + + - - 2 The modern, tool-supported code review process Gerrit Shane Bram Yasu
  3. 3. Patch + + - - Patch + + - - 2 The modern, tool-supported code review process Gerrit Shane Bram Yasu
  4. 4. Patch + + - - Patch + + - - 2 The modern, tool-supported code review process Gerrit Request review Shane Bram Yasu
  5. 5. Patch + + - - Patch + + - - 2 The modern, tool-supported code review process Gerrit Watch about NULL values at line 3! Shane Bram Yasu
  6. 6. Patch + + - - Patch + + - - 2 The modern, tool-supported code review process Gerrit Your code sucks! Watch about NULL values at line 3! Shane Bram Yasu
  7. 7. Patch + + - - Patch + + - - 2 The modern, tool-supported code review process Gerrit Your code sucks! Watch about NULL values at line 3! Shane Bram Yasu
  8. 8. The modern, tool-supported code review process 3 Gerrit Shane Bram Yasu
  9. 9. The modern, tool-supported code review process 3 Gerrit Shane Bram Yasu
  10. 10. The modern, tool-supported code review process 3 Upstream Gerrit Shane Bram Yasu
  11. 11. The modern, tool-supported code review process 3 Upstream Gerrit Shane Bram Yasu Changes are automatically integrated after review criteria are satisfied
  12. 12. Traditional! Code Inspections Mandated reviewer checklists and in-person meetings
  13. 13. Traditional! Code Inspections Mandated reviewer checklists and in-person meetings Measurably! improves! software! quality
  14. 14. Traditional! Code Inspections Mandated reviewer checklists and in-person meetings Modern! Code Review Lightweight, tool-supported Measurably! improves! software! quality
  15. 15. Traditional! Code Inspections Mandated reviewer checklists and in-person meetings Modern! Code Review Lightweight, tool-supported Measurably! improves! software! quality ???
  16. 16. Do lax reviewing practices have an impact on software quality? Expert code! reviewer
  17. 17. 6 Upstream Development Feature development Defect repairing Commit types Reviews can be linked to commits in version control systems
  18. 18. 6 Upstream Development Feature development Defect repairing Commit types Reviews can be linked to commits in version control systems
  19. 19. 7 commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <martin.jones@jollamobile.com> Gerrit Upstream Reviews can be linked to commits in version control systems
  20. 20. 7 commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <martin.jones@jollamobile.com> Gerrit Upstream Reviews can be linked to commits in version control systems
  21. 21. 7 commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <martin.jones@jollamobile.com> Gerrit Upstream Reviews can be linked to commits in version control systems
  22. 22. 7 commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <alan.alpert@nokia.com> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <martin.jones@jollamobile.com> Gerrit Upstream Reviews can be linked to commits in version control systems
  23. 23. 8 The Baseline
  24. 24. 8 Product Metrics The Baseline
  25. 25. 8 Product Metrics e.g., SLOC, complexity The Baseline
  26. 26. 8 Product Metrics Process Metrics e.g., SLOC, complexity The Baseline
  27. 27. 8 Product Metrics Process Metrics e.g., SLOC, complexity e.g., Prior defects, churn The Baseline
  28. 28. 8 Product Metrics Process Metrics Human Factor Metrics e.g., SLOC, complexity e.g., Prior defects, churn The Baseline
  29. 29. 8 Product Metrics Process Metrics Human Factor Metrics e.g., SLOC, complexity e.g., Prior defects, churn e.g., # Minor contributors The Baseline
  30. 30. 9 The Context
  31. 31. The Context 1,339 components 254 defect-prone v5.0.0 218 components 24 defect-prone v4.3.0 170 components 15 defect-prone v5.10.0 1,337 components 187 defect-prone v5.1.0 10
  32. 32. ParticipationCoverage Do lax reviewing practices have an impact on software quality?
  33. 33. ParticipationCoverage Do lax reviewing practices have an impact on software quality?
  34. 34. 13 Measuring review coverage using the proportion of reviewed changes 13 Upstream Development
  35. 35. 13 Measuring review coverage using the proportion of reviewed changes 13 Upstream Development Gerrit
  36. 36. 13 Measuring review coverage using the proportion of reviewed changes 13 Upstream Development Gerrit 7 out of 10 changes are! linked to code reviews
  37. 37. 14 Concrete Predicted Defect Count Defect Model Create artificial component setting explanatory variables to median values Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes 100% Churn 150
  38. 38. 15 Defect Model Calculate number of predicted defects Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes Churn 150 Reviewed changes PredictedDefectCount Concrete Predicted Defect Count 100%
  39. 39. 15 Defect Model Calculate number of predicted defects Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes Churn 150 Reviewed changes 0 PredictedDefectCount Concrete Predicted Defect Count 100%
  40. 40. 15 Defect Model Calculate number of predicted defects Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes Churn 150 Reviewed changes 0 PredictedDefectCount Concrete Predicted Defect Count 100%
  41. 41. 16 Defect Model Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes 90% Churn 150 Modify variable under test while keeping other explanatory variables constant Reviewed changes PredictedDefectCount Concrete Predicted Defect Count
  42. 42. 16 Defect Model Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes 90% Churn 150 Modify variable under test while keeping other explanatory variables constant Reviewed changes PredictedDefectCount Concrete Predicted Defect Count
  43. 43. 16 Defect Model Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes 90% Churn 150 1 Modify variable under test while keeping other explanatory variables constant Reviewed changes PredictedDefectCount Concrete Predicted Defect Count
  44. 44. 16 Defect Model Artificial Comp. 500SLOC Explanatory! Variable Value Reviewed changes 90% Churn 150 1 Modify variable under test while keeping other explanatory variables constant Reviewed changes PredictedDefectCount Concrete Predicted Defect Count
  45. 45. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●0 1 2 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Proportion of Reviewed Changes ConcretePredictedDefectCount ● Qt 5.0.0 VTK 5.10.0 17 Review coverage doesn’t tell the whole story
  46. 46. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●0 1 2 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Proportion of Reviewed Changes ConcretePredictedDefectCount ● Qt 5.0.0 VTK 5.10.0 17 Review coverage doesn’t tell the whole story Not a significant factor for Qt 5.1.0 or ITK
  47. 47. ParticipationCoverage Do lax reviewing practices have an impact on software quality?
  48. 48. ParticipationCoverage Do lax reviewing practices have an impact on software quality? ! Coverage is linked to quality in two of the four studied systems, yet it doesn’t tell the whole story
  49. 49. ParticipationCoverage Do lax reviewing practices have an impact on software quality? ! Coverage is linked to quality in two of the four studied systems, yet it doesn’t tell the whole story
  50. 50. 20 Measuring review participation using proportions of lax reviews I❤️ Me Self-approved changes
  51. 51. 20 Measuring review participation using proportions of lax reviews I❤️ Me Self-approved changes Hastily-reviewed changes
  52. 52. 20 Measuring review participation using proportions of lax reviews I❤️ Me Self-approved changes Hastily-reviewed changes Changes without discussion
  53. 53. 21 Measuring review participation using proportions of lax reviews I❤️ Me Self-approved changes Hastily-reviewed changes Changes without discussion
  54. 54. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 1 2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Proportion of Self−Approved Changes ConcretePredictedDefectCount 22 Lax participation leads to many post-release defects in our models
  55. 55. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 1 2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Proportion of Self−Approved Changes ConcretePredictedDefectCount 22 5 additional post-release defects Lax participation leads to many post-release defects in our models
  56. 56. ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 1 2 3 4 5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Proportion of Self−Approved Changes ConcretePredictedDefectCount 22 5 additional post-release defects Small increase leads to defects Lax participation leads to many post-release defects in our models
  57. 57. ParticipationCoverage Do lax reviewing practices have an impact on software quality? ! Coverage is linked to quality in two of the four studied systems, yet it doesn’t tell the whole story
  58. 58. ParticipationCoverage Do lax reviewing practices have an impact on software quality? ! Coverage is linked to quality in two of the four studied systems, yet it doesn’t tell the whole story ! Components w/ lax review participation are estimated to have several additional post-release defects

×