Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Fairness and Code Reviews Slide 1 Fairness and Code Reviews Slide 2 Fairness and Code Reviews Slide 3 Fairness and Code Reviews Slide 4 Fairness and Code Reviews Slide 5 Fairness and Code Reviews Slide 6 Fairness and Code Reviews Slide 7 Fairness and Code Reviews Slide 8 Fairness and Code Reviews Slide 9 Fairness and Code Reviews Slide 10 Fairness and Code Reviews Slide 11 Fairness and Code Reviews Slide 12 Fairness and Code Reviews Slide 13 Fairness and Code Reviews Slide 14 Fairness and Code Reviews Slide 15 Fairness and Code Reviews Slide 16 Fairness and Code Reviews Slide 17 Fairness and Code Reviews Slide 18 Fairness and Code Reviews Slide 19 Fairness and Code Reviews Slide 20 Fairness and Code Reviews Slide 21 Fairness and Code Reviews Slide 22 Fairness and Code Reviews Slide 23 Fairness and Code Reviews Slide 24 Fairness and Code Reviews Slide 25 Fairness and Code Reviews Slide 26 Fairness and Code Reviews Slide 27 Fairness and Code Reviews Slide 28 Fairness and Code Reviews Slide 29 Fairness and Code Reviews Slide 30 Fairness and Code Reviews Slide 31 Fairness and Code Reviews Slide 32 Fairness and Code Reviews Slide 33 Fairness and Code Reviews Slide 34 Fairness and Code Reviews Slide 35 Fairness and Code Reviews Slide 36 Fairness and Code Reviews Slide 37 Fairness and Code Reviews Slide 38 Fairness and Code Reviews Slide 39 Fairness and Code Reviews Slide 40 Fairness and Code Reviews Slide 41 Fairness and Code Reviews Slide 42 Fairness and Code Reviews Slide 43 Fairness and Code Reviews Slide 44 Fairness and Code Reviews Slide 45 Fairness and Code Reviews Slide 46 Fairness and Code Reviews Slide 47 Fairness and Code Reviews Slide 48 Fairness and Code Reviews Slide 49 Fairness and Code Reviews Slide 50 Fairness and Code Reviews Slide 51 Fairness and Code Reviews Slide 52 Fairness and Code Reviews Slide 53 Fairness and Code Reviews Slide 54 Fairness and Code Reviews Slide 55 Fairness and Code Reviews Slide 56 Fairness and Code Reviews Slide 57 Fairness and Code Reviews Slide 58 Fairness and Code Reviews Slide 59 Fairness and Code Reviews Slide 60 Fairness and Code Reviews Slide 61 Fairness and Code Reviews Slide 62 Fairness and Code Reviews Slide 63 Fairness and Code Reviews Slide 64 Fairness and Code Reviews Slide 65 Fairness and Code Reviews Slide 66 Fairness and Code Reviews Slide 67 Fairness and Code Reviews Slide 68 Fairness and Code Reviews Slide 69 Fairness and Code Reviews Slide 70 Fairness and Code Reviews Slide 71 Fairness and Code Reviews Slide 72 Fairness and Code Reviews Slide 73 Fairness and Code Reviews Slide 74 Fairness and Code Reviews Slide 75 Fairness and Code Reviews Slide 76 Fairness and Code Reviews Slide 77 Fairness and Code Reviews Slide 78 Fairness and Code Reviews Slide 79 Fairness and Code Reviews Slide 80 Fairness and Code Reviews Slide 81 Fairness and Code Reviews Slide 82 Fairness and Code Reviews Slide 83 Fairness and Code Reviews Slide 84 Fairness and Code Reviews Slide 85 Fairness and Code Reviews Slide 86 Fairness and Code Reviews Slide 87 Fairness and Code Reviews Slide 88 Fairness and Code Reviews Slide 89 Fairness and Code Reviews Slide 90 Fairness and Code Reviews Slide 91 Fairness and Code Reviews Slide 92 Fairness and Code Reviews Slide 93 Fairness and Code Reviews Slide 94 Fairness and Code Reviews Slide 95 Fairness and Code Reviews Slide 96 Fairness and Code Reviews Slide 97 Fairness and Code Reviews Slide 98 Fairness and Code Reviews Slide 99 Fairness and Code Reviews Slide 100 Fairness and Code Reviews Slide 101 Fairness and Code Reviews Slide 102 Fairness and Code Reviews Slide 103 Fairness and Code Reviews Slide 104 Fairness and Code Reviews Slide 105 Fairness and Code Reviews Slide 106 Fairness and Code Reviews Slide 107 Fairness and Code Reviews Slide 108 Fairness and Code Reviews Slide 109 Fairness and Code Reviews Slide 110 Fairness and Code Reviews Slide 111 Fairness and Code Reviews Slide 112 Fairness and Code Reviews Slide 113 Fairness and Code Reviews Slide 114 Fairness and Code Reviews Slide 115 Fairness and Code Reviews Slide 116 Fairness and Code Reviews Slide 117 Fairness and Code Reviews Slide 118 Fairness and Code Reviews Slide 119 Fairness and Code Reviews Slide 120 Fairness and Code Reviews Slide 121 Fairness and Code Reviews Slide 122 Fairness and Code Reviews Slide 123 Fairness and Code Reviews Slide 124 Fairness and Code Reviews Slide 125 Fairness and Code Reviews Slide 126 Fairness and Code Reviews Slide 127 Fairness and Code Reviews Slide 128 Fairness and Code Reviews Slide 129 Fairness and Code Reviews Slide 130 Fairness and Code Reviews Slide 131 Fairness and Code Reviews Slide 132 Fairness and Code Reviews Slide 133 Fairness and Code Reviews Slide 134 Fairness and Code Reviews Slide 135 Fairness and Code Reviews Slide 136 Fairness and Code Reviews Slide 137 Fairness and Code Reviews Slide 138 Fairness and Code Reviews Slide 139 Fairness and Code Reviews Slide 140 Fairness and Code Reviews Slide 141 Fairness and Code Reviews Slide 142 Fairness and Code Reviews Slide 143 Fairness and Code Reviews Slide 144 Fairness and Code Reviews Slide 145 Fairness and Code Reviews Slide 146 Fairness and Code Reviews Slide 147 Fairness and Code Reviews Slide 148 Fairness and Code Reviews Slide 149 Fairness and Code Reviews Slide 150 Fairness and Code Reviews Slide 151 Fairness and Code Reviews Slide 152 Fairness and Code Reviews Slide 153 Fairness and Code Reviews Slide 154 Fairness and Code Reviews Slide 155 Fairness and Code Reviews Slide 156 Fairness and Code Reviews Slide 157 Fairness and Code Reviews Slide 158 Fairness and Code Reviews Slide 159 Fairness and Code Reviews Slide 160 Fairness and Code Reviews Slide 161 Fairness and Code Reviews Slide 162 Fairness and Code Reviews Slide 163 Fairness and Code Reviews Slide 164 Fairness and Code Reviews Slide 165 Fairness and Code Reviews Slide 166 Fairness and Code Reviews Slide 167 Fairness and Code Reviews Slide 168 Fairness and Code Reviews Slide 169 Fairness and Code Reviews Slide 170 Fairness and Code Reviews Slide 171 Fairness and Code Reviews Slide 172 Fairness and Code Reviews Slide 173 Fairness and Code Reviews Slide 174 Fairness and Code Reviews Slide 175 Fairness and Code Reviews Slide 176 Fairness and Code Reviews Slide 177 Fairness and Code Reviews Slide 178
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

Fairness and Code Reviews

Download to read offline

This presentation provides an overview of fairness in general, and specifically, how it can be applied to code reviews. It also describes an empirical study of openstack that shows that fairness in their code review process requires some attention as some participants feel that their contributions are being unfairly reviewed.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Fairness and Code Reviews

  1. 1. “Was my contribution fairly reviewed” A framework to study the perception of fairness in modern code reviews Daniel M German University of Victoria dmg@uvic.ca
  2. 2. http://github.com/dmgerman/papers
  3. 3. What does it mean to be fair?
  4. 4. Is ICSE fair?
  5. 5. 62% of reviews were considered to be fair (2017) Is ICSE fair?
  6. 6. 62% of reviews were considered to be fair (2017) 48% (for rejected papers) Is ICSE fair?
  7. 7. ICSE experiments
  8. 8. ICSE experiments • 2016 and before:
  9. 9. ICSE experiments • 2016 and before: • Introduction of a Program Board
  10. 10. ICSE experiments • 2016 and before: • Introduction of a Program Board • Rebuttal period
  11. 11. ICSE experiments:
  12. 12. ICSE experiments: • 2017
  13. 13. ICSE experiments: • 2017 • Structured reviews
  14. 14. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC
  15. 15. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC • Public “Reviewer guidelines”
  16. 16. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC • Public “Reviewer guidelines” • No “subreviewer” (e.g reviews done by students)
  17. 17. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC • Public “Reviewer guidelines” • No “subreviewer” (e.g reviews done by students) • No list of research topics
  18. 18. ICSE experiments
  19. 19. ICSE experiments • 2018
  20. 20. ICSE experiments • 2018 • Gender balance of PC
  21. 21. ICSE experiments • 2018 • Gender balance of PC • Double blind
  22. 22. ICSE experiments • 2018 • Gender balance of PC • Double blind • Authors received numerical scores
  23. 23. What is ICSE trying to solve?
  24. 24. What is ICSE trying to solve? • Inconsistency of reviewing
  25. 25. What is ICSE trying to solve? • Inconsistency of reviewing • Bias
  26. 26. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic
  27. 27. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal
  28. 28. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal • Lack of transparency
  29. 29. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal • Lack of transparency The ICSE reviewer lottery
  30. 30. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal • Lack of transparency The ICSE reviewer lottery Some participants consider the process unfair
  31. 31. Participants are likely to leave a system if they perceive it to be unfair
  32. 32. What does it mean to be fair?
  33. 33. What does it mean to be treated fairly?
  34. 34. Fairness
  35. 35. Fairness • Also called “Justice”
  36. 36. Fairness • Also called “Justice” • Concept from: • Philosophy/Law/Psychology/ Sociology/Management/ Economics…
  37. 37. Fair (Merriam Webster):
  38. 38. Fair (Merriam Webster): 1.
  39. 39. Fair (Merriam Webster): 1. a. Marked with impartiality and honesty: free from self-interest, prejudice, or favouritism
  40. 40. Fair (Merriam Webster): 1. a. Marked with impartiality and honesty: free from self-interest, prejudice, or favouritism b. Conforming with the established rules
  41. 41. Justice (Merriam-Webster):
  42. 42. Justice (Merriam-Webster): 1.
  43. 43. Justice (Merriam-Webster): 1. a. The maintenance or administration of what is just, specially by the impartial adjustment of conflicting claims or the assignment of merited rewards or punishments
  44. 44. Justice (Merriam-Webster):
  45. 45. Justice (Merriam-Webster): 2.
  46. 46. Justice (Merriam-Webster): 2. a. The quality of being just, impartial or fair
  47. 47. Organizational Fairness/Justice:
  48. 48. Organizational Fairness/Justice: ‣ how to decide and allocate
  49. 49. Organizational Fairness/Justice: ‣ how to decide and allocate ‣ resources and decisions
  50. 50. Organizational Fairness/Justice: ‣ how to decide and allocate ‣ resources and decisions ‣ in a way that is fair and just
  51. 51. Organizational Fairness/Justice: ‣ how to decide and allocate ‣ resources and decisions ‣ in a way that is fair and just ‣ to the individuals and the group as a whole.
  52. 52. In general, we care about fairness when somebody makes a decision that affects us.
  53. 53. Fairness theory “focuses on explaining when authorities should be held accountable for unfavourable events”
  54. 54. How to increase the perception of fairness?
  55. 55. Distributive Fairness
  56. 56. Distributive Fairness Outcomes are allotted
  57. 57. Distributive Fairness Outcomes are allotted to persons
  58. 58. Distributive Fairness Outcomes are allotted to persons whose relative shares
  59. 59. Distributive Fairness Outcomes are allotted to persons whose relative shares can be described by some functional rule
  60. 60. Distributive Fairness Outcomes are allotted to persons whose relative shares can be described by some functional rule and some standard
  61. 61. Distributive Fairness
  62. 62. Distributive Fairness • Should those that contribute more receive more?
  63. 63. Distributive Fairness • Should those that contribute more receive more? • How do you measure contribution?
  64. 64. Distributive Fairness • Should those that contribute more receive more? • How do you measure contribution? • Is there a minimal standard defined in the system?
  65. 65. Equity vs Equality
  66. 66. Equity vs Equality • Equity: • Those that contribute more should receive more
  67. 67. Equity vs Equality • Equity: • Those that contribute more should receive more • Equality: • Everybody deserves the same
  68. 68. Organizations favour equity
  69. 69. Organizations favour equity • Rewards those that contribute more • Improves performance

  70. 70. Newcomers are special case
  71. 71. Newcomers are special case • Ok to have a positive bias to newcomers • They have not yet contributed

  72. 72. Procedural Fairness
  73. 73. Procedural Fairness • Concerned with the rules/process of making a decision:
  74. 74. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time
  75. 75. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers
  76. 76. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions
  77. 77. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions • Control: those affected should have a voice
  78. 78. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions • Control: those affected should have a voice • Correctability: there should exist an appeal process
  79. 79. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions • Control: those affected should have a voice • Correctability: there should exist an appeal process • Ethicality: the process upholds personal and organizational standards
  80. 80. Interactional fairness
  81. 81. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated
  82. 82. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another
  83. 83. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another • No rudeness/berating/sexism/racism
  84. 84. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another • No rudeness/berating/sexism/racism • Invasion of privacy: no disclosure of secrets/improper questions
  85. 85. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another • No rudeness/berating/sexism/racism • Invasion of privacy: no disclosure of secrets/improper questions • Psychological safety: no increase of stress
  86. 86. Informational fairness
  87. 87. Informational fairness • What information is used/exchanged is important
  88. 88. Informational fairness • What information is used/exchanged is important • Truthfulness:
  89. 89. Informational fairness • What information is used/exchanged is important • Truthfulness: • Avoid deception: people don’t like to be “lied to”
  90. 90. Informational fairness • What information is used/exchanged is important • Truthfulness: • Avoid deception: people don’t like to be “lied to” • Adequacy of explanations:
  91. 91. Informational fairness • What information is used/exchanged is important • Truthfulness: • Avoid deception: people don’t like to be “lied to” • Adequacy of explanations: • Reasonable explanations help accepting an outcome
  92. 92. Code Reviews
  93. 93. Code Reviews • A decision process that involves individuals
  94. 94. Code Reviews • A decision process that involves individuals • Prone to fairness concerns
  95. 95. Distributive Fairness
  96. 96. Distributive Fairness • Two concerns:
  97. 97. Distributive Fairness • Two concerns: • Patch level
  98. 98. Distributive Fairness • Two concerns: • Patch level • Individual level
  99. 99. Patches
  100. 100. Patches • Equity: Some patches
  101. 101. Patches • Equity: Some patches • are more important than others
  102. 102. Patches • Equity: Some patches • are more important than others • require more than others
  103. 103. Patches • Equity: Some patches • are more important than others • require more than others • Need: what is the minimal treatment that a patch should receive?
  104. 104. Distributive Fairness: individuals
  105. 105. Distributive Fairness: individuals • Equity vs Equality
  106. 106. Distributive Fairness: individuals • Equity vs Equality • Should core developers receive preferential treatment?
  107. 107. Distributive Fairness: individuals • Equity vs Equality • Should core developers receive preferential treatment? • Newcomers:
  108. 108. Distributive Fairness: individuals • Equity vs Equality • Should core developers receive preferential treatment? • Newcomers: • What preferential treatment should they get?
  109. 109. Procedural Fairness
  110. 110. Consistency and bias suppression
  111. 111. Consistency and bias suppression • Define:
  112. 112. Consistency and bias suppression • Define: • standards of quality and minimal requirements for approved patches
  113. 113. Consistency and bias suppression • Define: • standards of quality and minimal requirements for approved patches • prioritization strategies
  114. 114. Control and Inf. Accuracy
  115. 115. Control and Inf. Accuracy • Allow authors and non-reviewers to comment on patches under review

  116. 116. Correctability
  117. 117. Correctability • Create an appeal process for rejected patches

  118. 118. Ethicallity
  119. 119. Ethicallity • Define ethical standards on how code reviews should be done
  120. 120. Ethicallity • Define ethical standards on how code reviews should be done • What is Conflict of Interest?
  121. 121. Ethicallity • Define ethical standards on how code reviews should be done • What is Conflict of Interest? • Is self-reviewing allowed?
  122. 122. Interactional Fairness
  123. 123. Interactional Fairness • Define a code of conduct

  124. 124. Equity vs Equality
  125. 125. Equity vs Equality • A code of conduct seems to imply that everybody is equal
  126. 126. Equity vs Equality • A code of conduct seems to imply that everybody is equal • But in reality, some people are more important than others
  127. 127. Informational Fairness
  128. 128. Truthfulness
  129. 129. Truthfulness • Do code reviews with full ongoing transparency

  130. 130. Adequacy
  131. 131. Adequacy • Reviewers should:
  132. 132. Adequacy • Reviewers should: • provide good feedback
  133. 133. Adequacy • Reviewers should: • provide good feedback • explain delays, and
  134. 134. Adequacy • Reviewers should: • provide good feedback • explain delays, and • unexpected treatments.
  135. 135. Empirical Study
  136. 136. OpenStack
  137. 137. OpenStack • Ecosystem of 30-something projects
  138. 138. OpenStack • Ecosystem of 30-something projects • Industrial
  139. 139. OpenStack • Ecosystem of 30-something projects • Industrial • Most contributions are by paid developers
  140. 140. OpenStack • Ecosystem of 30-something projects • Industrial • Most contributions are by paid developers • 0.5 M SLOCs by 60k contributors
  141. 141. OpenStack • Ecosystem of 30-something projects • Industrial • Most contributions are by paid developers • 0.5 M SLOCs by 60k contributors • Using code reviews
  142. 142. Survey
  143. 143. Survey • 2870 invitations
  144. 144. Survey • 2870 invitations • 7.4% response rate (213)
  145. 145. Have your contributions being treated unfairly? 0 10 20 30 40 Never Rarely Occassionally Often Always
  146. 146. According to your experience as a reviewer, do you perform code reviews unfairly? 0 15 30 45 60 Never Rarely Occassionally Often Always
  147. 147. How do you prioritize reviews?
  148. 148. Open ended answers
  149. 149. Open ended answers
  150. 150. Open ended answers
  151. 151. Some open-ended answers
  152. 152. Equity vs Equality “A submitter is presented with the illusion that all submissions are equal, when in reality the community prioritizes [code reviews] from known submitters”
  153. 153. Newcomers “I have only tried to make one contribution. I was so discouraged by the response from the reviewer I have not tried again.”
  154. 154. Consistency “I have contributed and reviewed in 3 projects: [A, B] and [C]. I found that [A] and [B] to be quite reasonable and fair. As stated above, [C] is a disaster.”
  155. 155. Control and Correctability “[A] core reviewer gave me a -1 which was not understandable. I asked 4 time on IRC to discuss it and he ignored me”
  156. 156. Ethicallity “People look out for friends instead of looking out for their projects”
  157. 157. Interactional Fairness “Contributions are ignored unless I beg for attention. I might have to contribute a minor improvement. I never have time to beg for attention. If you don’t want my help, I got the message loud and clear”
  158. 158. OpenStack: Summary
  159. 159. OpenStack: Summary • Overall, it seems to be perceived as fair
  160. 160. OpenStack: Summary • Overall, it seems to be perceived as fair • There are signs of discontent, specially in some sub-projects
  161. 161. Takeaway message
  162. 162. Takeaway message Fairness is an issue that needs attention in code reviews
  163. 163. Takeaway message Fairness is an issue that needs attention in code reviews This framework helps understand what are the issues and how to address them
  164. 164. To summarize
  165. 165. http://turingmachine.org http://github.com/dmgerman/papers
  • NatalieReyna1

    Nov. 23, 2021

This presentation provides an overview of fairness in general, and specifically, how it can be applied to code reviews. It also describes an empirical study of openstack that shows that fairness in their code review process requires some attention as some participants feel that their contributions are being unfairly reviewed.

Views

Total views

1,097

On Slideshare

0

From embeds

0

Number of embeds

21

Actions

Downloads

9

Shares

0

Comments

0

Likes

1

×