Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Writing Acceptable Patches: An Empirical 
Study of Open Source Project Patches 
Yida Tao*, DongGyun Han§ and Sunghun Kim* ...
Developers submit many patches 
…but not all of them can be accepted 
Code Reviewer 
2
Developers submit many patches 
…but not all of them can be accepted 
Patch Reviewer 
“IT’S A NO FROM ME!” 
3
70% 
60% 
50% 
40% 
30% 
20% 
10% 
0% 
Firefox 
(Baysal et al. 
WCRE’12) 
Apache 
(Rigby and 
German, 2006) 
FLAC,OpenAFS ...
Patch rejection & resubmission = ? 
increased interruption and workload 
5
Patch rejection & resubmission = 
increased interruption and workload 
6
We hope our patches to be 
accepted right after the first or 
early rounds of code review 
7
How to write 
acceptable 
patches? 
8
How to write 
acceptable 
patches? 
• Learn from “past failures” 
• Q1: Why are patches rejected? 
• Understand the priori...
How to write 
acceptable 
patches? 
• Learn from “past failures” 
• Q1: Why are patches rejected? 
• Understand the priori...
How to write 
acceptable 
patches? 
• Learn from “past failures” 
• Q1: Why are patches rejected? 
• Understand the priori...
• Learn from “past failures” 
• Q1: Why are patches rejected? 
• Understand the priority 
• Q2: Which reasons are more dec...
Patch Extraction & Inspection 
13
Extracting Rejected Patches 
• Projects 
• Eclipse and Mozilla 
• Bug tracking system 
• Bugzilla 
• Criterion for rejecte...
Patch Inspection 
• Open coding 
• Review comments of 300 rejected patches 
• # open codes: 637 
“The patch fixes CCE but ...
Patch Inspection 
Open code: 
“malformed patch” 
Open code: 
“wrong indentation” 
Theme: 
Violating coding styles 
• Open ...
Patch Inspection 
Open code: 
“malformed patch” 
Open code: 
“wrong indentation” 
Theme: 
Violating coding styles 
• Open ...
Theme Frequency 
24.6% 
Suboptimal solution 
18
Theme Frequency 
24.6% 
19 
Incomplete fix 
22.3%
Theme Frequency 
24.6% 
20 
22.3% 
11.9% Including unnecessary changes
Theme Frequency 
Violating coding styles 
9.7% 
24.6% 
21 
22.3% 
11.9%
Bad Naming 
Inconsistent, misleading 
documentation 
Missing documentation 
Test Failure 
Introducing New Bugs 
Compilatio...
Suboptimal 
solution, 24.6% 
Incomplete fix, 
22.3% 
Including 
unnecessary 
Introducing new 
bugs, 4.1% 
Test failures, 
...
Q2: How decisive each rejection reason is? 
24
Patch reviewers have very low tolerance for patches 
that fail compilation or tests, or introduce new bugs 
25
Inconsistent or misleading documentation is highly 
unacceptable and way worse than missing documentation 
26
Inconsistent or misleading documentation is highly 
unacceptable and way worse than missing documentation 
27
Reviewers reject a large patch not solely because of its size, 
but mainly because the underlying reasons that induce its ...
Reviewers reject a large patch not solely because of its size, 
but mainly because the underlying reasons that induce its ...
Q3: Which patch-rejection reasons 
are more difficult to judge? 
30
Very easy Very difficult to judge 
Not decisive Very decisive to reject a patch 31
Inconsistent, misleading 
documentation 
Test failures 
Compile errors 
Very easy Very difficult to judge 
Not decisive Ve...
Very easy Very difficult to judge 
Duplication 
Unnecessary 
changes 
Not decisive Very decisive to reject a patch 
33
Very easy Very difficult to judge 
Incomplete fix 
Suboptimal solution 
Introduce 
new bugs 
Not decisive Very decisive to...
Q4: Do patch reviewers and 
writers have unanimous patch 
evaluation criteria? 
35
Distinguish patch reviewers and patch writers 
•Method 
• Patch writer: write ≥ 1 patch 
• Patch reviewer: review ≥ 1 patc...
Avg. Decisive Score 
2 3 4 5 
37 
Compilation errors 
Test Failures 
Introducing new bugs 
Inconsistent Documentation 
Sub...
Avg. Decisive Score 
2 3 4 5 
38 
Compilation errors 
Test Failures 
Introducing new bugs 
Inconsistent Documentation 
Sub...
Avg. Decisive Score 
2 3 4 5 
39 
Compilation errors 
Test Failures 
Introducing new bugs 
Inconsistent Documentation 
Sub...
Why are patches rejected? 
40
Why are patches rejected? 
Patch inspection 
41
Why are patches rejected? 
+ 
Patch inspection Developer survey 
42
Why are patches rejected? 
+ + 
Patch inspection Developer survey Literature 
43
Why are patches rejected? 
+ + 
Patch inspection Developer survey Literature 
30 patch rejection reasons in 5 categories 
...
1. Problematic implementation or solution 
2. Difficult to read or maintain 
3. Lack of communication or trust 
4. Deviati...
1. Problematic implementation or solution 
• Compilation & test failures 
• Incomplete 
• Suboptimal 
• Solution too aggre...
2. Difficult to read or maintain 
• Coding style violation 
• Inconsistent documentation 
• Using deprecated API 
• No acc...
3. Lack of communication or trust 
• No discussion before patch submission 
• Patch writers are not responsive 
4. Deviati...
Please check our paper for more 
details on patch rejection reasons 
49
Summary 
Writing acceptable patches 
50
Checklist 
* Compile error 
* Test failure 
* Documentation 
* Duplication 
* Unnecessary changes 
51
Checklist 
* Compile error 
* Test failure 
* Documentation 
* Duplication 
* Unnecessary changes 
52 
Communication 
* Di...
Checklist 
* Compile error 
* Test failure 
* Documentation 
* Duplication 
* Unnecessary changes 
53 
Communication 
* Di...
Upcoming SlideShare
Loading in …5
×

1

Share

Download to read offline

Writing acceptable patches: an empirical study of open source project patches

Download to read offline

Presented at ICSME2014 http://www.icsme.org/

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Writing acceptable patches: an empirical study of open source project patches

  1. 1. Writing Acceptable Patches: An Empirical Study of Open Source Project Patches Yida Tao*, DongGyun Han§ and Sunghun Kim* *The Hong Kong University of Science and Technology §KAIST Institute for IT Convergence, Korea Advanced Institute of Science and Technology
  2. 2. Developers submit many patches …but not all of them can be accepted Code Reviewer 2
  3. 3. Developers submit many patches …but not all of them can be accepted Patch Reviewer “IT’S A NO FROM ME!” 3
  4. 4. 70% 60% 50% 40% 30% 20% 10% 0% Firefox (Baysal et al. WCRE’12) Apache (Rigby and German, 2006) FLAC,OpenAFS (Weißgerber et al. MSR’08) Patch rejection & resubmission 56% 40% 60% 4
  5. 5. Patch rejection & resubmission = ? increased interruption and workload 5
  6. 6. Patch rejection & resubmission = increased interruption and workload 6
  7. 7. We hope our patches to be accepted right after the first or early rounds of code review 7
  8. 8. How to write acceptable patches? 8
  9. 9. How to write acceptable patches? • Learn from “past failures” • Q1: Why are patches rejected? • Understand the priority • Q2: Which reasons are more decisive to reject a patch? • Q3: Which reasons are more difficult to judge? • Analyze primary stakeholders • Q4: Do patch reviewers and writers have unanimous patch evaluation criteria? 9
  10. 10. How to write acceptable patches? • Learn from “past failures” • Q1: Why are patches rejected? • Understand the priority • Q2: Which reasons are more decisive to reject a patch? • Q3: Which reasons are more difficult to judge? • Analyze primary stakeholders • Q4: Do patch reviewers and writers have unanimous patch evaluation criteria? 10
  11. 11. How to write acceptable patches? • Learn from “past failures” • Q1: Why are patches rejected? • Understand the priority • Q2: Which reasons are more decisive to reject a patch? • Q3: Which reasons are more difficult to judge? • Analyze primary stakeholders • Q4: Do patch reviewers and writers have unanimous patch evaluation criteria? 11
  12. 12. • Learn from “past failures” • Q1: Why are patches rejected? • Understand the priority • Q2: Which reasons are more decisive to reject a patch? • Q3: Which reasons are more difficult to judge? • Analyze primary stakeholders • Q4: Do patch reviewers and writers have unanimous patch evaluation criteria? Patch extraction & inspection Developer survey Literature 12
  13. 13. Patch Extraction & Inspection 13
  14. 14. Extracting Rejected Patches • Projects • Eclipse and Mozilla • Bug tracking system • Bugzilla • Criterion for rejected patch • Patches with explicit review- flag • Number of patches • 300 randomly inspected • 162 Eclipse + 138 Mozilla Patch attachments for Eclipse bug#233156 spell checking performance test (patch) 2009-11-23 no flags performance test (5.69 KB, patch) 2009-11-30 no flags performance test + fix (12.95 KB, patch) 2009-12-01 review-fix (7.63 KB, patch) 2010-01-04 review-performance test (6.53KB, patch) 2010-01-04 review+ fix (10.08 KB, patch) 2010-01-14 review+ 14
  15. 15. Patch Inspection • Open coding • Review comments of 300 rejected patches • # open codes: 637 “The patch fixes CCE but introduces a new bug: … Also the parameter should be renamed to …“ Open code: • new bug introduced • bad parameter name 15
  16. 16. Patch Inspection Open code: “malformed patch” Open code: “wrong indentation” Theme: Violating coding styles • Open coding 16 • Review comments of 300 rejected patches • # open codes: 637 • Axial coding
  17. 17. Patch Inspection Open code: “malformed patch” Open code: “wrong indentation” Theme: Violating coding styles • Open coding 17 • Review comments of 300 rejected patches • # open codes: 637 • Axial coding • 12 themes why patches are rejected
  18. 18. Theme Frequency 24.6% Suboptimal solution 18
  19. 19. Theme Frequency 24.6% 19 Incomplete fix 22.3%
  20. 20. Theme Frequency 24.6% 20 22.3% 11.9% Including unnecessary changes
  21. 21. Theme Frequency Violating coding styles 9.7% 24.6% 21 22.3% 11.9%
  22. 22. Bad Naming Inconsistent, misleading documentation Missing documentation Test Failure Introducing New Bugs Compilation errors Duplicate code 5.5% Theme Frequency 5.7% 6.1% 6.3% 22 24.6% 22.3% 11.9% 9.7%
  23. 23. Suboptimal solution, 24.6% Incomplete fix, 22.3% Including unnecessary Introducing new bugs, 4.1% Test failures, 5.5% Missing documentation, 5.7% Inconsistent, misleading documentation, 6.1% Bad naming, 6.3% Violating coding change, 11.9% style guidelines, 9.7% Compilation errors, 1.9%Duplication, 1.6% Patch size too large, 0.3% 12 patch rejection reasons Developer survey from manual inspection • Decisive score of each reason 246 / 1785 (14%) • Difficulty to judge each reason Eclipse and Mozilla developers 23
  24. 24. Q2: How decisive each rejection reason is? 24
  25. 25. Patch reviewers have very low tolerance for patches that fail compilation or tests, or introduce new bugs 25
  26. 26. Inconsistent or misleading documentation is highly unacceptable and way worse than missing documentation 26
  27. 27. Inconsistent or misleading documentation is highly unacceptable and way worse than missing documentation 27
  28. 28. Reviewers reject a large patch not solely because of its size, but mainly because the underlying reasons that induce its large size 28
  29. 29. Reviewers reject a large patch not solely because of its size, but mainly because the underlying reasons that induce its large size 29
  30. 30. Q3: Which patch-rejection reasons are more difficult to judge? 30
  31. 31. Very easy Very difficult to judge Not decisive Very decisive to reject a patch 31
  32. 32. Inconsistent, misleading documentation Test failures Compile errors Very easy Very difficult to judge Not decisive Very decisive to reject a patch 32
  33. 33. Very easy Very difficult to judge Duplication Unnecessary changes Not decisive Very decisive to reject a patch 33
  34. 34. Very easy Very difficult to judge Incomplete fix Suboptimal solution Introduce new bugs Not decisive Very decisive to reject a patch 34
  35. 35. Q4: Do patch reviewers and writers have unanimous patch evaluation criteria? 35
  36. 36. Distinguish patch reviewers and patch writers •Method • Patch writer: write ≥ 1 patch • Patch reviewer: review ≥ 1 patch or both • Responses • 246 respondents • 131 patch writers, 115 patch reviewers 36
  37. 37. Avg. Decisive Score 2 3 4 5 37 Compilation errors Test Failures Introducing new bugs Inconsistent Documentation Suboptimal solution Incomplete fix Duplication Unnecessary changes Violating coding styles Bad naming Missing documentation Patch size too large Patch reviewer Patch writer
  38. 38. Avg. Decisive Score 2 3 4 5 38 Compilation errors Test Failures Introducing new bugs Inconsistent Documentation Suboptimal solution Incomplete fix Duplication Unnecessary changes Violating coding styles Bad naming Missing documentation Patch size too large Patch reviewer Patch writer
  39. 39. Avg. Decisive Score 2 3 4 5 39 Compilation errors Test Failures Introducing new bugs Inconsistent Documentation Suboptimal solution Incomplete fix Duplication Unnecessary changes Violating coding styles Patch reviewers tend to have stricter criteria on patch quality Bad naming compared to patch writers Missing documentation Patch size too large Patch reviewer Patch writer
  40. 40. Why are patches rejected? 40
  41. 41. Why are patches rejected? Patch inspection 41
  42. 42. Why are patches rejected? + Patch inspection Developer survey 42
  43. 43. Why are patches rejected? + + Patch inspection Developer survey Literature 43
  44. 44. Why are patches rejected? + + Patch inspection Developer survey Literature 30 patch rejection reasons in 5 categories 44
  45. 45. 1. Problematic implementation or solution 2. Difficult to read or maintain 3. Lack of communication or trust 4. Deviating from the project focus or scope 5. Affecting the development schedule 30 patch rejection reasons in 5 categories 45
  46. 46. 1. Problematic implementation or solution • Compilation & test failures • Incomplete • Suboptimal • Solution too aggressive or hostile for end users 46
  47. 47. 2. Difficult to read or maintain • Coding style violation • Inconsistent documentation • Using deprecated API • No accompanied test cases 47
  48. 48. 3. Lack of communication or trust • No discussion before patch submission • Patch writers are not responsive 4. Deviating from the project scope or focus • Irrelevant or obsolete 5. Affecting the development schedule • Freeze the progress of other features 48
  49. 49. Please check our paper for more details on patch rejection reasons 49
  50. 50. Summary Writing acceptable patches 50
  51. 51. Checklist * Compile error * Test failure * Documentation * Duplication * Unnecessary changes 51
  52. 52. Checklist * Compile error * Test failure * Documentation * Duplication * Unnecessary changes 52 Communication * Discussion prior to patch submission * Timely response to review comments
  53. 53. Checklist * Compile error * Test failure * Documentation * Duplication * Unnecessary changes 53 Communication * Discussion prior to patch submission * Timely response to review comments Call for industrial & research efforts * Introduce new bugs * Suboptimal solutions * Incomplete fix
  • xrodrigorgs

    Feb. 13, 2015

Presented at ICSME2014 http://www.icsme.org/

Views

Total views

683

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

12

Shares

0

Comments

0

Likes

1

×