Download to read offline
Uploaded byDelft University of Technology
When Testing Meets Code Review: Why and How Developers Review Tests
The document discusses a study on how developers review test code during code reviews. The study examined code reviews from three open source projects (Eclipse, Qt, OpenStack) and interviewed 12 developers. The main findings were: 1) Test code is reviewed less rigorously than production code - production code is 1.9x more likely to receive review comments. However, test-only files receive more attention than production-only files. 2) Reviewers discuss testing practices, code improvements, and knowledge transfer like documentation links when reviewing tests. Defects are not a major discussion topic. 3) Interviewed developers said they follow practices like running tests and checking test coverage when reviewing tests. 4)
More Related Content
Similar to When Testing Meets Code Review: Why and How Developers Review Tests
When Testing Meets Code Review: Why and How Developers Review Tests
- 1. When testing meetsCode Review: Why and How Developers Review Tests Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli
- 2. When testing meetsCode Review: Why and How Developers Review Tests Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli @DavideSpadini ishepard
- 4.
- 5. Research Questions RQ1: Howrigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
- 6. Research Questions RQ1: Howrigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
- 7. RQ1: How rigorouslyis test code reviewed? — Method # of prod. files # of test files # of code reviews # of reviewers # of comments Eclipse 215k 19k 60k 1k 95k Openstack 75k 48k 199k 9k 894k Qt 158k 8k 114k 1k 19k Total 450k 77k 374k 12k 1,010k
- 8. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java
- 9. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java
- 10. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java ATest.java
- 11. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java
- 12. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Test alone A.java ATest.java Production alone A.java
- 13. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java
- 14. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments
- 15. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments
- 16. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments
- 17. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments When together, production code is 1.9 more likely do be discussed
- 18. RQ1: How rigorouslyis test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments When together, production code is 1.9 more likely do be discussed When taken separately, test code is 1.16 more likely to be discussed
- 19. Avg # ofcomments per file Avg # of reviewers Avg length of comments Together Production Test 3.00 1.27 5.49 19.09 15.32 Production alone 1.64 3.95 18.13 Test alone 2.30 5.15 17.01 RQ1: How rigorously is test code reviewed? — Results
- 20. # of comments, avglength, avg # reviewers Prod. files are 2 times more likely to be discussed than test files Test files are discussed more when alone RQ1: How rigorously is test code reviewed? — Summary
- 21. Research Questions RQ1: Howrigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
- 22. RQ2: What doreviewers discuss in test code reviews? — Method > 1,000,000 comments 600 comments 1st round: 6 categories* 2nd round: for each category, more fine-grained *Bacchelli & Bird. Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013
- 23. RQ2: What doreviewers discuss in test code reviews? — Results 0% 10% 20% 30% 40% production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study) Code improvements Understanding Social communication Defects Knowledge transfer
- 24. 0 10 20 30 40 Testing practices Codestyling Un/Tested paths Better naming Unnecessary code Assertion handling Code improvements RQ2: What do reviewers discuss in test code reviews? — Results
- 25. RQ2: What doreviewers discuss in test code reviews? — Results 0% 10% 20% 30% 40% production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study) Understanding Social communication Defects Knowledge transfer Code improvements
- 26. 0 10 20 30 40 50 Severe issues Lesssevere issues Wrong assertions Defects RQ2: What do reviewers discuss in test code reviews? — Results “You need to instantiate LttngKernelTrace here otherwise you will get a class cast exception” — Severe issue “This isnt being used, hence pep8 error.” — Less severe issue
- 27. RQ2: What doreviewers discuss in test code reviews? — Results 0% 10% 20% 30% 40% production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study) Code improvements Understanding Social communication Knowledge transfer Defects
- 28. 0 15 30 45 60 Link to externaldocumentation How-to Knowledge transfer RQ2: What do reviewers discuss in test code reviews? — Results
- 29. RQ2: What doreviewers discuss in test code reviews? — Summary Testing practices, tested paths and assertions Defects are not among the most discussed topics Developers often comment with how-to or ext. documentation
- 30. Research Questions RQ1: Howrigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
- 31. RQ3: Which practicesdo reviewers follow for test files? — Method Interviews!
- 32. 1 Openstack 1 Qt 1Eclipse 1 Microsoft 1 Ericsson 2 Alura 5 OSS 12 interviews RQ3: Which practices do reviewers follow for test files? — Method
- 33. RQ3: Which practicesdo reviewers follow for test files? — Summary Checking out the change and run the tests Understanding if all paths are covered Test Driven Review
- 34. Research questions RQ1: Howrigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
- 35. Management policies prioritize strongly production code Novicedevelopers do not know the effect of poor testing Lack of navigation support and in- depth code coverage info Reviewing tests requires context on both tests and production RQ4: What problems and challenges do developers face when reviewing tests?
- 36. Management policies prioritize strongly production code Novicedevelopers do not know the effect of poor testing Lack of navigation support and in- depth code coverage info Reviewing tests requires context on both tests and production RQ4: What problems and challenges do developers face when reviewing tests? Provide the context for reviewing test Plan enough time for test review Educate on test reviewing and writing Detailed code coverage information Implications and recommendations
- 38. Code Review •Developers havelimited time to do code review •Should test files be reviewed? •If production files are more prone to contain bugs…reviewer should focus on production!
- 39. Should test filesbe reviewed? • We conducted a preliminary investigation to see whether test and production code files are equally associated with defects • We built a statistical model and used as an explanatory variable “being a test”
- 40. Should test filesbe reviewed? • 3 OSS projects: Qt, Eclipse and Openstack • Dependent variable: post-release defects • Explanatory variables: metrics well related to defect proneness: product metrics, process metrics and human factors. ‣ Plus, our variable: isTest
- 41. Metrics in orderof importance Attribute Average merit Average Rank Churn 0.753 1 Author ownership 0.599 2.2 ± 0.4 Cumulative churn 0.588 2.8 ± 0.4 Total authors 0.521 4 Major authors 0.506 5 Size 0.411 6 Prior defects 0.293 7 Minor authors 0.149 8 Is test 0.085 9 Should test files be reviewed? • Yes! • The decision to review a file should not be based on whether the file contains production or test code, as this has no association with defects.