Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fairness and Code Reviews

429 views

Published on

This presentation provides an overview of fairness in general, and specifically, how it can be applied to code reviews. It also describes an empirical study of openstack that shows that fairness in their code review process requires some attention as some participants feel that their contributions are being unfairly reviewed.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Fairness and Code Reviews

  1. 1. “Was my contribution fairly reviewed” A framework to study the perception of fairness in modern code reviews Daniel M German University of Victoria dmg@uvic.ca
  2. 2. http://github.com/dmgerman/papers
  3. 3. What does it mean to be fair?
  4. 4. Is ICSE fair?
  5. 5. 62% of reviews were considered to be fair (2017) Is ICSE fair?
  6. 6. 62% of reviews were considered to be fair (2017) 48% (for rejected papers) Is ICSE fair?
  7. 7. ICSE experiments
  8. 8. ICSE experiments • 2016 and before:
  9. 9. ICSE experiments • 2016 and before: • Introduction of a Program Board
  10. 10. ICSE experiments • 2016 and before: • Introduction of a Program Board • Rebuttal period
  11. 11. ICSE experiments:
  12. 12. ICSE experiments: • 2017
  13. 13. ICSE experiments: • 2017 • Structured reviews
  14. 14. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC
  15. 15. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC • Public “Reviewer guidelines”
  16. 16. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC • Public “Reviewer guidelines” • No “subreviewer” (e.g reviews done by students)
  17. 17. ICSE experiments: • 2017 • Structured reviews • Reviewers were anonymous among PC • Public “Reviewer guidelines” • No “subreviewer” (e.g reviews done by students) • No list of research topics
  18. 18. ICSE experiments
  19. 19. ICSE experiments • 2018
  20. 20. ICSE experiments • 2018 • Gender balance of PC
  21. 21. ICSE experiments • 2018 • Gender balance of PC • Double blind
  22. 22. ICSE experiments • 2018 • Gender balance of PC • Double blind • Authors received numerical scores
  23. 23. What is ICSE trying to solve?
  24. 24. What is ICSE trying to solve? • Inconsistency of reviewing
  25. 25. What is ICSE trying to solve? • Inconsistency of reviewing • Bias
  26. 26. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic
  27. 27. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal
  28. 28. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal • Lack of transparency
  29. 29. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal • Lack of transparency The ICSE reviewer lottery
  30. 30. What is ICSE trying to solve? • Inconsistency of reviewing • Bias • Topic • Interpersonal • Lack of transparency The ICSE reviewer lottery Some participants consider the process unfair
  31. 31. Participants are likely to leave a system if they perceive it to be unfair
  32. 32. What does it mean to be fair?
  33. 33. What does it mean to be treated fairly?
  34. 34. Fairness
  35. 35. Fairness • Also called “Justice”
  36. 36. Fairness • Also called “Justice” • Concept from: • Philosophy/Law/Psychology/ Sociology/Management/ Economics…
  37. 37. Fair (Merriam Webster):
  38. 38. Fair (Merriam Webster): 1.
  39. 39. Fair (Merriam Webster): 1. a. Marked with impartiality and honesty: free from self-interest, prejudice, or favouritism
  40. 40. Fair (Merriam Webster): 1. a. Marked with impartiality and honesty: free from self-interest, prejudice, or favouritism b. Conforming with the established rules
  41. 41. Justice (Merriam-Webster):
  42. 42. Justice (Merriam-Webster): 1.
  43. 43. Justice (Merriam-Webster): 1. a. The maintenance or administration of what is just, specially by the impartial adjustment of conflicting claims or the assignment of merited rewards or punishments
  44. 44. Justice (Merriam-Webster):
  45. 45. Justice (Merriam-Webster): 2.
  46. 46. Justice (Merriam-Webster): 2. a. The quality of being just, impartial or fair
  47. 47. Organizational Fairness/Justice:
  48. 48. Organizational Fairness/Justice: ‣ how to decide and allocate
  49. 49. Organizational Fairness/Justice: ‣ how to decide and allocate ‣ resources and decisions
  50. 50. Organizational Fairness/Justice: ‣ how to decide and allocate ‣ resources and decisions ‣ in a way that is fair and just
  51. 51. Organizational Fairness/Justice: ‣ how to decide and allocate ‣ resources and decisions ‣ in a way that is fair and just ‣ to the individuals and the group as a whole.
  52. 52. In general, we care about fairness when somebody makes a decision that affects us.
  53. 53. Fairness theory “focuses on explaining when authorities should be held accountable for unfavourable events”
  54. 54. How to increase the perception of fairness?
  55. 55. Distributive Fairness
  56. 56. Distributive Fairness Outcomes are allotted
  57. 57. Distributive Fairness Outcomes are allotted to persons
  58. 58. Distributive Fairness Outcomes are allotted to persons whose relative shares
  59. 59. Distributive Fairness Outcomes are allotted to persons whose relative shares can be described by some functional rule
  60. 60. Distributive Fairness Outcomes are allotted to persons whose relative shares can be described by some functional rule and some standard
  61. 61. Distributive Fairness
  62. 62. Distributive Fairness • Should those that contribute more receive more?
  63. 63. Distributive Fairness • Should those that contribute more receive more? • How do you measure contribution?
  64. 64. Distributive Fairness • Should those that contribute more receive more? • How do you measure contribution? • Is there a minimal standard defined in the system?
  65. 65. Equity vs Equality
  66. 66. Equity vs Equality • Equity: • Those that contribute more should receive more
  67. 67. Equity vs Equality • Equity: • Those that contribute more should receive more • Equality: • Everybody deserves the same
  68. 68. Organizations favour equity
  69. 69. Organizations favour equity • Rewards those that contribute more • Improves performance

  70. 70. Newcomers are special case
  71. 71. Newcomers are special case • Ok to have a positive bias to newcomers • They have not yet contributed

  72. 72. Procedural Fairness
  73. 73. Procedural Fairness • Concerned with the rules/process of making a decision:
  74. 74. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time
  75. 75. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers
  76. 76. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions
  77. 77. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions • Control: those affected should have a voice
  78. 78. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions • Control: those affected should have a voice • Correctability: there should exist an appeal process
  79. 79. Procedural Fairness • Concerned with the rules/process of making a decision: • Consistency of decisions across time • Bias suppression of decision makers • Accuracy of information used in decisions • Control: those affected should have a voice • Correctability: there should exist an appeal process • Ethicality: the process upholds personal and organizational standards
  80. 80. Interactional fairness
  81. 81. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated
  82. 82. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another
  83. 83. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another • No rudeness/berating/sexism/racism
  84. 84. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another • No rudeness/berating/sexism/racism • Invasion of privacy: no disclosure of secrets/improper questions
  85. 85. Interactional fairness • Individuals are not only concerned with outcomes but how they are treated • Respect: actions and signals that reflect the intrinsic value that a person has for another • No rudeness/berating/sexism/racism • Invasion of privacy: no disclosure of secrets/improper questions • Psychological safety: no increase of stress
  86. 86. Informational fairness
  87. 87. Informational fairness • What information is used/exchanged is important
  88. 88. Informational fairness • What information is used/exchanged is important • Truthfulness:
  89. 89. Informational fairness • What information is used/exchanged is important • Truthfulness: • Avoid deception: people don’t like to be “lied to”
  90. 90. Informational fairness • What information is used/exchanged is important • Truthfulness: • Avoid deception: people don’t like to be “lied to” • Adequacy of explanations:
  91. 91. Informational fairness • What information is used/exchanged is important • Truthfulness: • Avoid deception: people don’t like to be “lied to” • Adequacy of explanations: • Reasonable explanations help accepting an outcome
  92. 92. Code Reviews
  93. 93. Code Reviews • A decision process that involves individuals
  94. 94. Code Reviews • A decision process that involves individuals • Prone to fairness concerns
  95. 95. Distributive Fairness
  96. 96. Distributive Fairness • Two concerns:
  97. 97. Distributive Fairness • Two concerns: • Patch level
  98. 98. Distributive Fairness • Two concerns: • Patch level • Individual level
  99. 99. Patches
  100. 100. Patches • Equity: Some patches
  101. 101. Patches • Equity: Some patches • are more important than others
  102. 102. Patches • Equity: Some patches • are more important than others • require more than others
  103. 103. Patches • Equity: Some patches • are more important than others • require more than others • Need: what is the minimal treatment that a patch should receive?
  104. 104. Distributive Fairness: individuals
  105. 105. Distributive Fairness: individuals • Equity vs Equality
  106. 106. Distributive Fairness: individuals • Equity vs Equality • Should core developers receive preferential treatment?
  107. 107. Distributive Fairness: individuals • Equity vs Equality • Should core developers receive preferential treatment? • Newcomers:
  108. 108. Distributive Fairness: individuals • Equity vs Equality • Should core developers receive preferential treatment? • Newcomers: • What preferential treatment should they get?
  109. 109. Procedural Fairness
  110. 110. Consistency and bias suppression
  111. 111. Consistency and bias suppression • Define:
  112. 112. Consistency and bias suppression • Define: • standards of quality and minimal requirements for approved patches
  113. 113. Consistency and bias suppression • Define: • standards of quality and minimal requirements for approved patches • prioritization strategies
  114. 114. Control and Inf. Accuracy
  115. 115. Control and Inf. Accuracy • Allow authors and non-reviewers to comment on patches under review

  116. 116. Correctability
  117. 117. Correctability • Create an appeal process for rejected patches

  118. 118. Ethicallity
  119. 119. Ethicallity • Define ethical standards on how code reviews should be done
  120. 120. Ethicallity • Define ethical standards on how code reviews should be done • What is Conflict of Interest?
  121. 121. Ethicallity • Define ethical standards on how code reviews should be done • What is Conflict of Interest? • Is self-reviewing allowed?
  122. 122. Interactional Fairness
  123. 123. Interactional Fairness • Define a code of conduct

  124. 124. Equity vs Equality
  125. 125. Equity vs Equality • A code of conduct seems to imply that everybody is equal
  126. 126. Equity vs Equality • A code of conduct seems to imply that everybody is equal • But in reality, some people are more important than others
  127. 127. Informational Fairness
  128. 128. Truthfulness
  129. 129. Truthfulness • Do code reviews with full ongoing transparency

  130. 130. Adequacy
  131. 131. Adequacy • Reviewers should:
  132. 132. Adequacy • Reviewers should: • provide good feedback
  133. 133. Adequacy • Reviewers should: • provide good feedback • explain delays, and
  134. 134. Adequacy • Reviewers should: • provide good feedback • explain delays, and • unexpected treatments.
  135. 135. Empirical Study
  136. 136. OpenStack
  137. 137. OpenStack • Ecosystem of 30-something projects
  138. 138. OpenStack • Ecosystem of 30-something projects • Industrial
  139. 139. OpenStack • Ecosystem of 30-something projects • Industrial • Most contributions are by paid developers
  140. 140. OpenStack • Ecosystem of 30-something projects • Industrial • Most contributions are by paid developers • 0.5 M SLOCs by 60k contributors
  141. 141. OpenStack • Ecosystem of 30-something projects • Industrial • Most contributions are by paid developers • 0.5 M SLOCs by 60k contributors • Using code reviews
  142. 142. Survey
  143. 143. Survey • 2870 invitations
  144. 144. Survey • 2870 invitations • 7.4% response rate (213)
  145. 145. Have your contributions being treated unfairly? 0 10 20 30 40 Never Rarely Occassionally Often Always
  146. 146. According to your experience as a reviewer, do you perform code reviews unfairly? 0 15 30 45 60 Never Rarely Occassionally Often Always
  147. 147. How do you prioritize reviews?
  148. 148. Open ended answers
  149. 149. Open ended answers
  150. 150. Open ended answers
  151. 151. Some open-ended answers
  152. 152. Equity vs Equality “A submitter is presented with the illusion that all submissions are equal, when in reality the community prioritizes [code reviews] from known submitters”
  153. 153. Newcomers “I have only tried to make one contribution. I was so discouraged by the response from the reviewer I have not tried again.”
  154. 154. Consistency “I have contributed and reviewed in 3 projects: [A, B] and [C]. I found that [A] and [B] to be quite reasonable and fair. As stated above, [C] is a disaster.”
  155. 155. Control and Correctability “[A] core reviewer gave me a -1 which was not understandable. I asked 4 time on IRC to discuss it and he ignored me”
  156. 156. Ethicallity “People look out for friends instead of looking out for their projects”
  157. 157. Interactional Fairness “Contributions are ignored unless I beg for attention. I might have to contribute a minor improvement. I never have time to beg for attention. If you don’t want my help, I got the message loud and clear”
  158. 158. OpenStack: Summary
  159. 159. OpenStack: Summary • Overall, it seems to be perceived as fair
  160. 160. OpenStack: Summary • Overall, it seems to be perceived as fair • There are signs of discontent, specially in some sub-projects
  161. 161. Takeaway message
  162. 162. Takeaway message Fairness is an issue that needs attention in code reviews
  163. 163. Takeaway message Fairness is an issue that needs attention in code reviews This framework helps understand what are the issues and how to address them
  164. 164. To summarize
  165. 165. http://turingmachine.org http://github.com/dmgerman/papers

×