Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
When testing meets Code Review:
Why and How Developers Review Tests
Davide Spadini, Mauricio Aniche,
Margaret-Anne Storey,...
When testing meets Code Review:
Why and How Developers Review Tests
Davide Spadini, Mauricio Aniche,
Margaret-Anne Storey,...
Code review
Research Questions
RQ1: How rigorously is test code reviewed?
RQ2: What do reviewers discuss in test
code reviews?
RQ3: Wh...
Research Questions
RQ1: How rigorously is test code reviewed?
RQ2: What do reviewers discuss in test
code reviews?
RQ3: Wh...
RQ1: How rigorously is test code reviewed? — Method
# of
prod. files
# of
test files
# of code
reviews
# of
reviewers
# of
c...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java ATest.java
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Test alone
A.java ATest.java
Production al...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
Test alone
ATest.j...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
Test alone
ATest.j...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
Test alone
ATest.j...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
Test alone
ATest.j...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
Test alone
ATest.j...
RQ1: How rigorously is test code reviewed? — Results
Together
A.java ATest.java
Production alone
A.java
Test alone
ATest.j...
Avg # of comments
per file
Avg # of reviewers
Avg length of
comments
Together
Production

Test
3.00

1.27
5.49
19.09

15.32...
# of comments,
avg length,
avg # reviewers
Prod. files are
2 times more likely
to be discussed
than test files
Test files are...
Research Questions
RQ1: How rigorously is test code reviewed?
RQ2: What do reviewers discuss in test
code reviews?
RQ3: Wh...
RQ2: What do reviewers discuss in test code reviews? — Method
> 1,000,000
comments
600
comments
1st round:
6 categories*
2...
RQ2: What do reviewers discuss in test code reviews? — Results
0% 10% 20% 30% 40%
production reviews (Bacchelli & Bird, IC...
0
10
20
30
40
Testing practices Code styling Un/Tested paths Better naming Unnecessary code Assertion handling
Code improv...
RQ2: What do reviewers discuss in test code reviews? — Results
0% 10% 20% 30% 40%
production reviews (Bacchelli & Bird, IC...
0
10
20
30
40
50
Severe issues Less severe issues Wrong assertions
Defects
RQ2: What do reviewers discuss in test code rev...
RQ2: What do reviewers discuss in test code reviews? — Results
0% 10% 20% 30% 40%
production reviews (Bacchelli & Bird, IC...
0
15
30
45
60
Link to external documentation How-to
Knowledge transfer
RQ2: What do reviewers discuss in test code reviews...
RQ2: What do reviewers discuss in test code reviews? — Summary
Testing practices,
tested paths and
assertions
Defects are ...
Research Questions
RQ1: How rigorously is test code reviewed?
RQ2: What do reviewers discuss in test
code reviews?
RQ3: Wh...
RQ3: Which practices do reviewers follow for test files? — Method
Interviews!
1 Openstack
1 Qt
1 Eclipse
1 Microsoft
1 Ericsson
2 Alura
5 OSS
12 interviews
RQ3: Which practices do reviewers follow for...
RQ3: Which practices do reviewers follow for test files? — Summary
Checking out the
change and run
the tests
Understanding ...
Research questions
RQ1: How rigorously is test code reviewed?
RQ2: What do reviewers discuss in test
code reviews?
RQ3: Wh...
Management
policies prioritize
strongly
production code
Novice developers
do not know the
effect of poor
testing
Lack of n...
Management
policies prioritize
strongly
production code
Novice developers
do not know the
effect of poor
testing
Lack of n...
Code Review
•Developers have limited time to do
code review
•Should test files be reviewed?
•If production files are more pr...
Should test files be reviewed?
• We conducted a preliminary investigation to see whether test and production
code files are ...
Should test files be reviewed?
• 3 OSS projects: Qt, Eclipse and Openstack
• Dependent variable: post-release defects
• Exp...
Metrics in order of importance
Attribute Average merit Average Rank
Churn 0.753 1
Author ownership 0.599 2.2 ± 0.4
Cumulat...
When Testing Meets Code Review: Why and How Developers Review Tests
When Testing Meets Code Review: Why and How Developers Review Tests
Upcoming SlideShare
Loading in …5
×

When Testing Meets Code Review: Why and How Developers Review Tests

24 views

Published on

ICSE 2018

Davide Spadini

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

When Testing Meets Code Review: Why and How Developers Review Tests

  1. 1. When testing meets Code Review: Why and How Developers Review Tests Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli
  2. 2. When testing meets Code Review: Why and How Developers Review Tests Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli @DavideSpadini ishepard
  3. 3. Code review
  4. 4. Research Questions RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
  5. 5. Research Questions RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
  6. 6. RQ1: How rigorously is test code reviewed? — Method # of prod. files # of test files # of code reviews # of reviewers # of comments Eclipse 215k 19k 60k 1k 95k Openstack 75k 48k 199k 9k 894k Qt 158k 8k 114k 1k 19k Total 450k 77k 374k 12k 1,010k
  7. 7. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java
  8. 8. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java
  9. 9. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java ATest.java
  10. 10. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java
  11. 11. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Test alone A.java ATest.java Production alone A.java
  12. 12. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java
  13. 13. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments
  14. 14. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments
  15. 15. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments
  16. 16. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments When together, production code is 1.9 more likely do be discussed
  17. 17. RQ1: How rigorously is test code reviewed? — Results Together A.java ATest.java Production alone A.java Test alone ATest.java 0 15 30 45 60 75 % of files with review comments When together, production code is 1.9 more likely do be discussed When taken separately, test code is 1.16 more likely to be discussed
  18. 18. Avg # of comments per file Avg # of reviewers Avg length of comments Together Production Test 3.00 1.27 5.49 19.09 15.32 Production alone 1.64 3.95 18.13 Test alone 2.30 5.15 17.01 RQ1: How rigorously is test code reviewed? — Results
  19. 19. # of comments, avg length, avg # reviewers Prod. files are 2 times more likely to be discussed than test files Test files are discussed more when alone RQ1: How rigorously is test code reviewed? — Summary
  20. 20. Research Questions RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
  21. 21. RQ2: What do reviewers discuss in test code reviews? — Method > 1,000,000 comments 600 comments 1st round: 6 categories* 2nd round: for each category, more fine-grained *Bacchelli & Bird. Expectations, Outcomes, and Challenges of Modern Code Review. ICSE 2013
  22. 22. RQ2: What do reviewers discuss in test code reviews? — Results 0% 10% 20% 30% 40% production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study) Code improvements Understanding Social communication Defects Knowledge transfer
  23. 23. 0 10 20 30 40 Testing practices Code styling Un/Tested paths Better naming Unnecessary code Assertion handling Code improvements RQ2: What do reviewers discuss in test code reviews? — Results
  24. 24. RQ2: What do reviewers discuss in test code reviews? — Results 0% 10% 20% 30% 40% production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study) Understanding Social communication Defects Knowledge transfer Code improvements
  25. 25. 0 10 20 30 40 50 Severe issues Less severe issues Wrong assertions Defects RQ2: What do reviewers discuss in test code reviews? — Results “You need to instantiate LttngKernelTrace here otherwise you will get a class cast exception” — Severe issue “This isnt being used, hence pep8 error.” — Less severe issue
  26. 26. RQ2: What do reviewers discuss in test code reviews? — Results 0% 10% 20% 30% 40% production reviews (Bacchelli & Bird, ICSE 2013) test reviews (this study) Code improvements Understanding Social communication Knowledge transfer Defects
  27. 27. 0 15 30 45 60 Link to external documentation How-to Knowledge transfer RQ2: What do reviewers discuss in test code reviews? — Results
  28. 28. RQ2: What do reviewers discuss in test code reviews? — Summary Testing practices, tested paths and assertions Defects are not among the most discussed topics Developers often comment with how-to or ext. documentation
  29. 29. Research Questions RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
  30. 30. RQ3: Which practices do reviewers follow for test files? — Method Interviews!
  31. 31. 1 Openstack 1 Qt 1 Eclipse 1 Microsoft 1 Ericsson 2 Alura 5 OSS 12 interviews RQ3: Which practices do reviewers follow for test files? — Method
  32. 32. RQ3: Which practices do reviewers follow for test files? — Summary Checking out the change and run the tests Understanding if all paths are covered Test Driven Review
  33. 33. Research questions RQ1: How rigorously is test code reviewed? RQ2: What do reviewers discuss in test code reviews? RQ3: Which practices do reviewers follow for test files? RQ4: What problems and challenges do developers face when reviewing tests? We collected Code Reviews from Gerrit 3 OSS: Eclipse, Qt, OpenStack 12 interviews
  34. 34. Management policies prioritize strongly production code Novice developers do not know the effect of poor testing Lack of navigation support and in- depth code coverage info Reviewing tests requires context on both tests and production RQ4: What problems and challenges do developers face when reviewing tests?
  35. 35. Management policies prioritize strongly production code Novice developers do not know the effect of poor testing Lack of navigation support and in- depth code coverage info Reviewing tests requires context on both tests and production RQ4: What problems and challenges do developers face when reviewing tests? Provide the context for reviewing test Plan enough time for test review Educate on test reviewing and writing Detailed code coverage information Implications and recommendations
  36. 36. Code Review •Developers have limited time to do code review •Should test files be reviewed? •If production files are more prone to contain bugs…reviewer should focus on production!
  37. 37. Should test files be reviewed? • We conducted a preliminary investigation to see whether test and production code files are equally associated with defects • We built a statistical model and used as an explanatory variable “being a test”
  38. 38. Should test files be reviewed? • 3 OSS projects: Qt, Eclipse and Openstack • Dependent variable: post-release defects • Explanatory variables: metrics well related to defect proneness: product metrics, process metrics and human factors. ‣ Plus, our variable: isTest
  39. 39. Metrics in order of importance Attribute Average merit Average Rank Churn 0.753 1 Author ownership 0.599 2.2 ± 0.4 Cumulative churn 0.588 2.8 ± 0.4 Total authors 0.521 4 Major authors 0.506 5 Size 0.411 6 Prior defects 0.293 7 Minor authors 0.149 8 Is test 0.085 9 Should test files be reviewed? • Yes! • The decision to review a file should not be based on whether the file contains production or test code, as this has no association with defects.

×