Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
“We Don’t Do That Here”:
How Collaborative Editing with Mentors Improves
Engagement in Social Q&A Communities
Denae Ford
N...
2
source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif
source: https://media.tumblr.com/3d150f8da60feb45ed32...
3
source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif
source: https://media.tumblr.com/3d150f8da60feb45ed32...
4
[Burnett et al., 2016]
5
What to ask?
Question may be a duplicate since list of
existing questions appear
6
How to ask?
Unfamiliar with community expectations and feel like question is unfit
7
Where to get that info?
Don’t see other users with a similar new experience
8
[Stack Overflow 2018 Developer Survey]
Even though, about 50 million people visit Stack Overflow monthly…
9
[Stack Overflow 2017 Developer Survey]
…not everyone contributes.
[Stack Overflow 2018 Developer Survey]
7.6% v.s. 88.6%...
Collaborative Editing
Two core components:
Collaborative question draft
Help Room to discuss and edit the
draft
10
2
1
Design Principles
Provide formative and timely feedback
Allow mistakes in private space
Do not answer questions, help othe...
Just-In-Time Mentorship
Study
12
Study Design Goals
To observe and measure changes in question quality
To understand feedback provided by mentors
To improv...
How it works
14
Novice writes
question
Novice writes
question
Novice selects to
post their
question
Novice writes
question
Novice selects to
post their
question
Novice 

selects to join 

Help Room
Novice writes
question
Novice selects to
post their
question
Novice 

selects to join 

Help Room
Novice receives
feedback on
question
Novice writes
question
Novice edits
question draft
Novice selects to
post their
quest...
Novice receives
feedback on
question
Novice writes
question
Novice selects to
post their
question
Novice 

selects to join...
Novice receives
feedback on
question
Novice writes
question
Novice selects to
post their
question
Novice 

selects to join...
Novice receives
feedback on
question
Novice writes
question
Novice edits
question draft
Novice selects to
post their
quest...
23
[Burnett et al., 2016]
343
conversations
Data Collected
24
33 day study
71,068
Eligible Novices
271 Novices
343
conversations
Transcripts + Quest...
Findings
25
Mentored questions have higher quality.
MENTORED QUESTIONS HAD A
50% SCORE INCREASE
26
GOOD NEUTRAL BAD
mentored 25% 49% 2...
Mentors suggest high-fidelity improvements.
Paraphrasing title and content (Question Phrasing)
Formatting code to make que...
Mentors suggest high-fidelity improvements.
Paraphrasing title and content (Question Phrasing)
Formatting code to make que...
Community Triage
29
Community Triage
30
“ok the fact is that on SO you can’t ask for libs you would need to do that on
another site [link] (wi...
Community Triage
31
Community Triage
32
“Your question is off topic here....We’re not a hacking service.” (C72 -Mentor)
"ok the fact is that o...
Community Culture of Asking
33
Community Culture of Asking
34
“You can probably remove ‘Hello’ and ‘Problem’ from the top of the question.
While it’s goo...
Community Culture of Asking
35
“You can probably remove ‘Hello’ and ‘Problem’ from the top of the question.
While it’s goo...
Participants are satisfied with their interactions.
36
Likert Statement Median
I feel that I am a part of the
Stack Overfl...
Discussion + Implications
37
Having a human mentor surprised novices and made them feel
welcomed
Interpret dialogue for non-native English speaker
Orga...
Identity-Based Mentoring
39
Novice: Yeah,
actually I am!
I thought I picked up on
your name too!
Mentor: Hey! are you
from...
40
Implications for community-based mentorship
Asking community members about their interest
Mentor many-to-many, one-to-m...
41
Implications for community-based mentorship
http://community.rstudio.com/
https://www.askquestions.tech/faq
http://pyth...
Online Q&A communities are widely used, but unfamiliar norms
can leave novices discouraged from participating.
Just-in-tim...
http://bit.ly/StackMentorPodcast
http://bit.ly/StackMentorPaper
dford3@ncsu.edu
@DenaeFord @KristinaLustig @JeremyBanks @C...
++
44
Feature Implementation
45
Expanded existing Stack Overflow chat
rooms
Novices: eligible novice drafting a
question
Mentors...
Upcoming SlideShare
Loading in …5
×

“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities

87 views

Published on

Slides from CHI 2018 paper presentation describing the pilot Stack Overflow just-in-time mentorship program.

Abstract:
Online question-and-answer (Q&A) communities like Stack
Overflow have norms that are not obvious to novice users.
Novices create and post programming questions without feedback, and the community enforces site norms through public
downvoting and commenting. This can leave novices discouraged
from further participation. We deployed a month long, just-in-time mentorship program to Stack Overflow in which we redirected novices in the process of asking a question to an on-site Help Room. There, novices received feedback on their question drafts from experienced Stack Overflow mentors. We present examples and discussion of various question improvements including: question context, code formatting, and wording that adheres to on-site cultural norms. We find that mentored questions are substantially improved over non-mentored questions, with average scores increasing by 50%. We provide design implications that challenge how socio-technical communities onboard novices across domains.

Published in: Software
  • Be the first to comment

  • Be the first to like this

“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities

  1. 1. “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities Denae Ford North Carolina State University Kristina Lustig Stack Exchange, Inc. Jeremy Banks Stack Exchange, Inc. Chris Parnin North Carolina State University @DenaeFord
  2. 2. 2 source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif source: https://media.tumblr.com/3d150f8da60feb45ed3254b5721e5c44/tumblr_inline_mjw0l2etMI1qz4rgp.gif
  3. 3. 3 source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif source: https://media.tumblr.com/3d150f8da60feb45ed3254b5721e5c44/tumblr_inline_mjw0l2etMI1qz4rgp.gif What to ask? How to ask? Where to get that info?
  4. 4. 4 [Burnett et al., 2016]
  5. 5. 5 What to ask? Question may be a duplicate since list of existing questions appear
  6. 6. 6 How to ask? Unfamiliar with community expectations and feel like question is unfit
  7. 7. 7 Where to get that info? Don’t see other users with a similar new experience
  8. 8. 8 [Stack Overflow 2018 Developer Survey] Even though, about 50 million people visit Stack Overflow monthly…
  9. 9. 9 [Stack Overflow 2017 Developer Survey] …not everyone contributes. [Stack Overflow 2018 Developer Survey] 7.6% v.s. 88.6% ~87% have accounts—but 20% have never posted
  10. 10. Collaborative Editing Two core components: Collaborative question draft Help Room to discuss and edit the draft 10 2 1
  11. 11. Design Principles Provide formative and timely feedback Allow mistakes in private space Do not answer questions, help others ask better questions 11 [Epstein et al., 2002] [Baron 1993] [Wenger, 1998] [Kraut et al., 2012] [Kraut et al., 2012][Fedor et al., 1993] [Ford et al., 2016] [Kraut et al., 2012]
  12. 12. Just-In-Time Mentorship Study 12
  13. 13. Study Design Goals To observe and measure changes in question quality To understand feedback provided by mentors To improve experience for new question askers To learn how to scale these benefits to the whole community 13
  14. 14. How it works 14
  15. 15. Novice writes question
  16. 16. Novice writes question Novice selects to post their question
  17. 17. Novice writes question Novice selects to post their question Novice 
 selects to join 
 Help Room
  18. 18. Novice writes question Novice selects to post their question Novice 
 selects to join 
 Help Room
  19. 19. Novice receives feedback on question Novice writes question Novice edits question draft Novice selects to post their question Novice 
 selects to join 
 Help Room
  20. 20. Novice receives feedback on question Novice writes question Novice selects to post their question Novice 
 selects to join 
 Help Room
  21. 21. Novice receives feedback on question Novice writes question Novice selects to post their question Novice 
 selects to join 
 Help Room Petter Friber   2 days later… Mason   3 hours later… Jerem y   The last message was posted 5 days ago.             };  edit copy draft back to Stack Overflow Hey @KristinaLustig, thanks for coming. :) This is the Stack Overflow chat room we're using to help people improve their questions. This message is automated but everything else is real! Before we start: do you have any particular concerns, or are you just double­checking before you post? Welcome @KristinaLustig I'm here to help you with the question, are you in chat? edited post draft. room mode changed to Private: only approved users may enter this room (this should only be used for moderation purposes) Wed 15:19 chat ­ they will b leave send help | f
  22. 22. Novice receives feedback on question Novice writes question Novice edits question draft Novice selects to post their question Novice 
 selects to join 
 Help Room Novice posts question
  23. 23. 23 [Burnett et al., 2016]
  24. 24. 343 conversations Data Collected 24 33 day study 71,068 Eligible Novices 271 Novices 343 conversations Transcripts + Question Scores 63 Mentors Interview Mentors Survey Novices 4 Help Rooms 1 Private Mentor Room 520 Entered Room
  25. 25. Findings 25
  26. 26. Mentored questions have higher quality. MENTORED QUESTIONS HAD A 50% SCORE INCREASE 26 GOOD NEUTRAL BAD mentored 25% 49% 25% not mentored (control) 18% 51% 30% 6 -30 p <0.05 p <0.05
  27. 27. Mentors suggest high-fidelity improvements. Paraphrasing title and content (Question Phrasing) Formatting code to make question scannable Community triage Including all the pieces of a question (Question Framing) Community culture of asking 27
  28. 28. Mentors suggest high-fidelity improvements. Paraphrasing title and content (Question Phrasing) Formatting code to make question scannable Community triage Including all the pieces of a question (Question Framing) Community culture of asking 28
  29. 29. Community Triage 29
  30. 30. Community Triage 30 “ok the fact is that on SO you can’t ask for libs you would need to do that on another site [link] (with some [rules] that we can check if you like), instead if you like some code it would be really great if you tried something, do you have some code, do you have some post that you already checked?” (A30 -Mentor)
  31. 31. Community Triage 31
  32. 32. Community Triage 32 “Your question is off topic here....We’re not a hacking service.” (C72 -Mentor) "ok the fact is that on SO you can’t ask for libs you would need to do that on another site [link] (with some [rules] that we can check if you like), instead if you like some code it would be really great if you tried something, do you have some code, do you have some post that you already checked?” (A30 -Mentor)
  33. 33. Community Culture of Asking 33
  34. 34. Community Culture of Asking 34 “You can probably remove ‘Hello’ and ‘Problem’ from the top of the question. While it’s good to be social, it’s kind of just fluff on a Q&A site.” (D1 -Mentor)
  35. 35. Community Culture of Asking 35 “You can probably remove ‘Hello’ and ‘Problem’ from the top of the question. While it’s good to be social, it’s kind of just fluff on a Q&A site.” (D1 -Mentor) “You also might want to edit out the “Thank you!” at the end. I know it seems polite, but people object to it on Stack Overflow.” (D5 -Mentor) “Ok thanks for you help. I hope this time people won’t attack me.” (D62 -Novice)
  36. 36. Participants are satisfied with their interactions. 36 Likert Statement Median I feel that I am a part of the Stack Overflow community. 4 I feel more comfortable posting on Stack Overflow. 4 The help that I received from this program was useful to me. 5 I would recommend this program to other SO users. 5 I would like to participate in this program again. 5 NOVICES SATISFIEDWITH THEIR EXPERIENCE “If we can get the [original poster] through the first question with a positive experience and they can see how this site really works, then we should get more good questions which feeds in to having more good answers.” (M5 -Mentor) MENTORS INTERESTEDIN OFFERING GUIDANCE
  37. 37. Discussion + Implications 37
  38. 38. Having a human mentor surprised novices and made them feel welcomed Interpret dialogue for non-native English speaker Organic conversations to work through questions Advantages of humans helping humans 38 Source: https://www.businessinsider.com.au/why-the-iraq-war-cost-2-trillion-2013-3
  39. 39. Identity-Based Mentoring 39 Novice: Yeah, actually I am! I thought I picked up on your name too! Mentor: Hey! are you from Egypt by chance? Identity-Based Mentoring [Stack Overflow 2018 Developer Survey] Egypt Yossef
  40. 40. 40 Implications for community-based mentorship Asking community members about their interest Mentor many-to-many, one-to-many, and one-to-one Integrate mentorship functionality precisely when needed Source: http://humansofthearsenal.com/post/143654160037/im-taking-part-because-we-want-things-to-change
  41. 41. 41 Implications for community-based mentorship http://community.rstudio.com/ https://www.askquestions.tech/faq http://pythontutor.com/visualize.html
  42. 42. Online Q&A communities are widely used, but unfamiliar norms can leave novices discouraged from participating. Just-in-time mentoring provided novices with help to ask a well- received question, precisely when they needed it. Identifying novice-mentor interactions can define new onboarding and feedback experiences in other online communities. http://bit.ly/StackMentorPodcast http://bit.ly/StackMentorPaper dford3@ncsu.edu Summary
  43. 43. http://bit.ly/StackMentorPodcast http://bit.ly/StackMentorPaper dford3@ncsu.edu @DenaeFord @KristinaLustig @JeremyBanks @ChrisParnin Thank You! Online Q&A communities are widely used, but unfamiliar norms can leave novices discouraged from participating. Just-in-time mentoring provided novices with help to ask a well- received question, precisely when they needed it. Identifying novice-mentor interactions can define new onboarding and feedback experiences in other online communities.
  44. 44. ++ 44
  45. 45. Feature Implementation 45 Expanded existing Stack Overflow chat rooms Novices: eligible novice drafting a question Mentors: Experienced users

×