Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Beyond the Code Itself: How Programmers Really Look at Pull Requests

38 views

Published on

Developers in open source projects must make decisions on contributions from other community members, such as whether or not to accept a pull request. However, secondary factors—beyond the code itself—can influence those decisions. For example, signals from GitHub profiles, such as a number of followers, activity, names, or gender can also be considered when developers make decisions. In this paper, we examine how developers use these signals (or not) when making decisions about code contributions. To evaluate this question, we evaluate how signals related to perceived gender identity and code quality influenced decisions on accepting pull requests. Unlike previous work, we analyze this decision process with data collected from an eye-tracker. We analyzed differences in what signals developers said are important for themselves versus what signals they actually used to make decisions about others. We found that after the code snippet (x=57%), the second place programmers spent their time fixating on supplemental technical signals(x=32%), such as previous contributions and popular repositories. Diverging from what participants reported themselves, we also found that programmers fixated on social signals more than recalled.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Beyond the Code Itself: How Programmers Really Look at Pull Requests

  1. 1. Beyond the Code Itself: How Programmers Really Look at Pull Requests Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, Chris ParninDenaeFordRobin North Carolina State University + Eindhoven University Of Technology
  2. 2. PULL REQUEST: SUBMITTING CODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull Request
  3. 3. PULL REQUEST: SUBMITTING CODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull RequestPull Request Is Merged
  4. 4. PULL REQUEST: SUBMITTING CODE ONLINE 2 https://guides.github.com/introduction/flow/ Open a Pull RequestPull Request Is MergedDiscuss + Review
  5. 5. CODE CONTRIBUTIONS BASED ON “MERITS”? 3 https://help.github.com/en/articles/about-your-profile
  6. 6. CODE CONTRIBUTIONS BASED ON “MERITS”? 3 https://help.github.com/en/articles/about-your-profile
  7. 7. ABBY’S GITHUB PROFILE PAGE 4
  8. 8. ABBY’S GITHUB PROFILE PAGE 4
  9. 9. 5ABBY’S GITHUB PROFILE PAGE
  10. 10. 6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  11. 11. 6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  12. 12. 6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  13. 13. 6BUILDING ON PRIOR WORK Previous studies were done post factum Further insight into the decision making process Eye tracking offers a holistic perspective to the story [Dabbish, 2012] [Tsay, 2014] [Marlow 2013]
  14. 14. How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 7 RQ2 RQ1 RQ3 RESEARCH QUESTIONS
  15. 15. 8METHODOLOGY PROTOCOL Pre Experiment Survey Have you ever accepted or rejected or submitted a pull request? * Please rate your programming experience on a scale from 1-10. * How much experience do you have performing integration tasks? * Do you have a GitHub account? *
  16. 16. 9 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session 9 METHODOLOGY
  17. 17. 10 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request METHODOLOGY
  18. 18. 10 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request METHODOLOGY
  19. 19. 10 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request METHODOLOGY
  20. 20. 11 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Review Profile Page + Pull Request 11 Reasonable Code Snippet: Unreasonable Code Snippet: METHODOLOGY
  21. 21. 12 PROTOCOL Pre Experiment Survey Tic Tac Toe Training Session Post Experiment Survey Review Profile Page + Pull Request What elements of the displayed profile or pull request did you consider when making your decision? * What strategies do you use to decide to share this content? Does it vary across each community? * METHODOLOGY
  22. 22. 13 PARTICIPANTS METHODOLOGY
  23. 23. Recruited 42 Participants 13 PARTICIPANTS METHODOLOGY
  24. 24. Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY
  25. 25. All familiar with submitting and reviewing pull requests Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY
  26. 26. All familiar with submitting and reviewing pull requests Recruited 42 Participants Advanced Topics Computer Science Course 13 PARTICIPANTS METHODOLOGY Gender Quantity Age Range Country of Origin Minority Not Minority Men 30 22-33 24 India, 1 Nepal, 1 USA 2 23 Women 12 23-29 2 China, 5 India, 1 Iran, 3 USA 1 10
  27. 27. How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS
  28. 28. How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience
  29. 29. How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience Mapped Responses
  30. 30. How do programmers review pull requests? Where do programmers think they look vs. where they really look? What strategies do people use to manage signals for their personal identity? 14 RQ2 RQ1 RQ3 ANALYSIS Fixation Duration # Of Fixations Reported Experience Mapped Responses Platform Specific Strategies
  31. 31. Social Signals Code Signals Technical Signals
  32. 32. (AI) (DN) (FF) (RE) (HM) (CA) (RS) (BC) (AC) (UD) (PT) (SD) (RP) (CD) (TM) Social Signals Code Signals Technical Signals
  33. 33. RESULTS
  34. 34. TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision
  35. 35. TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision 12 correct decisions
  36. 36. TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision 12 correct decisions
  37. 37. TABLE II: Participant Fixations on Areas of Interest Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% T = true accept F = false accept F = false reject T = true reject — = no decision
  38. 38. Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject
  39. 39. Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Code Signals
  40. 40. Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals
  41. 41. Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals Participants spent a considerable amount of time on social signals ( x̅ = 10.43%,  x̃ = 7.38%) Social Signals
  42. 42. Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject 5 out of 20 participants spent 48% to 62% of their time fixating on technical signals and an above average time on social signals (17% to 31%) Participants spent a majority of their time fixating on code ( x̅  = 57.15%,  x̃ = 64.23%) Participants spent a considerable amount of time on tech signals ( x̅ = 32.42%, x̃ = 28.45%) Code Signals Technical Signals Participants spent a considerable amount of time on social signals ( x̅ = 10.43%,  x̃ = 7.38%) Social Signals
  43. 43. Participants M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 Experience H H H L H H H H H H L L H L H L L L H L PR Reviewed P P P A A A A T T T P P P A A A A T T T Decision Evaluation T – F – T✗ F T✗ T✗ – T✗ T T F F✗ T F T✗ T T T✗ Overview Code Signals 67% 66% 66% 21% 59% 25% 70% 83% 27% 73% 60% 70% 25% 69% 52% 75% 86% 63% 58% 27% Technical Signals 26% 30% 28% 48% 29% 49% 22% 11% 57% 17% 31% 24% 62% 25% 42% 18% 7% 28% 38% 56% Social Signals 7% 4% 6% 31% 12% 26% 8% 6% 16% 10% 8% 5% 13% 7% 6% 7% 7% 9% 3% 17% Code Signals After Code Snippet (AC) 97% 90% 88% 80% 98% 80% 89% 96% 71% 74% 93% 100% 28% 94% 97% 86% 89% 82% 99% 54% Before Code Snippet (BC) 3% 10% 12% 20% 2% 20% 11% 4% 29% 26% 7% – 72% 6% 3% 14% 11% 18% 1% 46% Technical Signals Contribution Activity (CA) 47% 65% 48% 36% – 18% 11% – 35% 20% 19% 5% 18% 24% 43% 12% 11% 28% 5% 11% Commit Details (CD) 7% 1% – 2% – 2% 19% – – – – 1% 9% 3% – 3% – 9% 3% 3% Contribution Heat Map (HM) 14% 16% 17% 12% – 8% 11% 25% 13% 8% 23% 28% 10% 14% 14% 13% 26% 19% 44% 18% Pull Request Title (PT) 2% 3% 3% 9% 63% 23% 20% 18% 17% 28% 33% 24% 18% 7% 8% 6% 25% 15% 13% 5% Popular Repositories (RE) 23% 15% 17% 28% 12% 26% 11% 46% 17% 13% 14% 11% 13% 23% 23% 32% 2% 16% 30% 45% Submission Details (SD) 6% – 15% 13% 25% 24% 28% 10% 18% 31% 12% 32% 32% 28% 13% 35% 36% 13% 6% 18% Social Signals Avatar Image (AI) 25% 20% 51% 28% 13% 16% 7% 64% 26% 52% 35% 33% 7% 24% 50% 21% 74% 42% 35% 46% Display Name (DN) 16% 24% 4% 8% – 3% 5% 12% 11% – – – 5% 8% 14% 5% 14% 11% – 11% Followers/Following (FF) 6% 19% 19% 6% – 11% – 3% – – – – – 2% – 2% – – – – Repository Popularity (RE) – – – – – – – 5% – – – – – – – – – 1% – – Repository Stars (RS) 45% 21% 17% – 12% 32% – 3% 14% – 3% – – 8% – 22% – 5% – 13% To Merge (TM) 6% 4% – 42% 70% 29% 39% – 28% 13% 58% 57% 63% 44% 20% 24% 3% 11% 65% 21% User Details (UD) 2% 13% 9% 16% 5% 9% 49% 14% 20% 35% 5% 10% 25% 13% 16% 26% 10% 29% – 9% Specifically, participants mentioned the correctness of the pull request, code complexity, and beautification such as style and Next, we consider the elements our participants reported considering compared to what they fixated on during the T = true accept F = false accept F = false reject T = true reject — = no decision More details in paper: http://bit.ly/BeyondCode
  44. 44. FIXATION + EXPERIENCE = RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
  45. 45. FIXATION + EXPERIENCE = RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
  46. 46. FIXATION + EXPERIENCE = RESULTING GROUPS RQ1: HOW DO PROGRAMMERS REVIEW PULL REQUESTS? 20 M1, M7, M5, M8 M4, W2, W4, W6, W8, W5, W9 M2, M3, M6, M9, M10, W1, W7, W10 High-Experienced Thinkers High-Experienced Glancers Low-Experienced Foragers Low-Experienced Thinkers
  47. 47. RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
  48. 48. 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
  49. 49. 45% reported using information related to previous contributions 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest
  50. 50. 45% reported using information related to previous contributions 73% reported using the code snippet to make a decision RQ2: WHERE DO PROGRAMMERS THINK THEY LOOK VS. WHERE THEY REALLY LOOK? PROGRAMMERS REVIEWED MORE SOCIAL SIGNALS THAN THEY REPORTED 21 Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest Post Survey Eye-Tracker Study Phase 0 5 10 15 20 No. of Participants w/ Overlap Between Phases AC AC- BC CA CD CD- HM CDHM- PT D- RE D- SD E- AI E- DN FF RP RS TM UD AreasofInterest 1 participant reported using the submitter’s profile image
  51. 51. RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy
  52. 52. RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy “I’ve different accounts for the different kinds of work. [...] It maybe because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26)
  53. 53. RQ3: WHAT STRATEGIES DO PEOPLE USE TO MANAGE SIGNALS FOR THEIR PERSONAL IDENTITY? PLATFORM STRATEGIES VARY TO PROTECT THEIR IDENTITY & CONTRIBUTIONS 22 Nameless code should stand alone Full Profile —>Trustworthy “I’ve different accounts for the different kinds of work. [...] It maybe because my code has nothing to do with my name or my image, the code needs to talk for itself.” (S26) “such familiarity gives a feeling of trust.” (S15)
  54. 54. DISCUSSION + IMPLICATIONS 23 All participants reviewed at least one social signal. Does viewing content necessarily mean that the information seen will influence the decision about a pull request?
  55. 55. DISCUSSION + IMPLICATIONS 23 GitHub profiles make social + human capital explicit. How does giving users control over their identity influence participation? All participants reviewed at least one social signal. Does viewing content necessarily mean that the information seen will influence the decision about a pull request?
  56. 56. Summary Paper: http://bit.ly/BeyondCode DenaeFordRobin (BC) (AC) (UD) (PT) (SD) (RP) (CD) (TM) dford3@ncsu.edu We observe that both social and technical aspects are being taken into consideration when deciding upon pull request acceptance. Moreover, many more social aspects are being considered during the experiment than reported during the post-experiment survey Future work will study how the execution of concealing or amplifying these signals affect developers across the identity spectrum and development experiences at scale. Artifact: http://bit.ly/BeyondCodeArtifacts

×