Presentation at the Tilburg School of Economics and Management on May 8, and at IÉSEG School of Management, Paris on May 3, 2024 on how Stack Overflow community question answering service tweaked its rejection notices to improve the retention of new contributors whose initial question is rejected (closed). The presentation is based on a paper coauthored with Sunil Wattal.

1. May 8, 2024
Tilburg School of Economics and Management
Not Good Enough, But Try Again!
The Impact of Improved Rejection Communications on Contributor
Retention and Performance in Open Knowledge Collaboration
Aleksi Aaltonen
Temple University
Sunil Wattal
Temple University

2. PhD 2012
Moves, acquired
by Facebook 2014
Assistant Professor
2014–2018
Assistant Professor
2018–

3. Open
knowledge
collaboration

4. High-quality production in open knowledge collaboration
requires often rejecting contribution made in good faith.
Rejections demotivate especially new contributors from
attempting further contributions.

5. New Contributor Retention Is a Major Issue!
1. Systems must keep converting users into regular contributors to replace
those who stop contributing (Halavais and Lackaff 2008, Faraj et al. 2011,
Liu and Ram 2011).
2. Most new contributors never return after making their first contribution
(Arazy et al. 2016, Bayus 2013, Panciera et al. 2009, Piezunka and
Dahlander 2019).
3. The contributors are even less likely to make further contributions if their
initial contribution is rejected (Halfaker et al. 2013, Musicant et al. 2011).

6. Literature shows that there is a trade-off between quality
control and new contributor retention in open knowledge
collaboration.
We do not know how an open knowledge collaboration
system can mitigate the trade-off.

7. Literature

8. The regulation of behavior in online communities
A system should provide cues to help users to become better contributors as they
interact with the system (Ren et al. 2018).
Content removal notices are an under-utilized moderation practice in many systems
(Ahn et al. 2013, Jhaver et al. 2019, Piezunka and Dahlander 2019).
More informative rejection notices could encourage new contributors to try again
by reducing uncertainty about future interactions with the system (Fang and
Neufeld 2009, Pavlou et al. 2007, Shah 2006, White et al. 2007).

9. Organizational selection
The ways in which rejected applicants are treated can have important
implications on:
• Employee turnover (Dlugos and Keller 2021)
• Diversity (Bapna et al. 2022)
• Job performance post-hire (Konradt et al. 2017)
• The intentions of rejected applicants to reapply (Gilliland et al. 2001)
Positive outcomes are explained by the perceived fairness of
organizational selection.

10. Research Design

11. Data
Stack Overflow, the biggest community
question answering service for
programmers changed the wording of its
rejection notices on June 25, 2013 to
inform contributors better about the
reasons for their rejection.
We analyze 11,035 new contributors
who submitted their initial question with
±84 days sample window around the
treatment date (169 daily observations).
Authors’ names blinded for peer revi
12 Article submitted to Management Science; manuscript
0
500
1,000
1,500
2,000
Apr May Jun Jul Aug Sep
2013
Initial Questions per Day
Figure 2 The number of initial questions asked per day in the sample window. The dashed vertical line mark
the treatment date.

12. OLD NOTICE
Not a real question. It's difficult to tell
what is being asked here. This question is
ambiguous, vague, incomplete, overly
broad, or rhetorical and cannot be
reasonably answered in its current form.
For help clarifying this question so that it
can be reopened, see the FAQ.
NEW NOTICE
Unclear what you are asking. Please
clarify your specific problem or add
additional details to highlight exactly what
you need. As it’s currently written, it’s
hard to tell exactly what you’re asking.

13. Manipulation check
We use a survey instrument with validated items: “Not informative at all...Very informative” (Sen and Lerman
2007) and ”I feel the decision to close the question was fair: Strongly disagree...Strongly agree” (Gilliland 1994).
104 valid mTurk subjects (programmers but not Stack Overflow users) scored five rejection notices randomly
drawn from the old and new notices and answered background questions including an attention check.
Robustness checks: i) including current Stack Overflow users, ii) old rejection notices shortened to a similar
length than the new ones, iii) test for a difference in the sentiment between the old and new notices.
experience, but answers to this question will tend
to be almost entirely based on opinions, rather than
facts, references, or specific expertise.
Figure A.3 Old and new rejection notices.
Table A.5 The evaluation of the informativeness of the old and new rejection notices on a seven-point scale by Amazon
Mechanical Turk subjects.
Old rejection notices New rejection notices Welch t-test
Dimension Mean SD n Mean SD n Di↵erence in means p-value
Informativeness 5.143 (1.477) 105 5.568 (1.200) 95 0.425* 0.026
Fairness 5.349 (1.363) 109 5.336 (1.345) 107 -0.013 0.947
Notes. *** p < 0.001, ** p < 0.01, * p < 0.05; SD, standard deviation

14. Contributor Retention (part 1)
• Regression discontinuity in time design (causal identification)
estimated using local polynomial regression in rdrobust R package
with data-driven bandwidths
• Survival model to assess the persistence of the treatment effect
Contributor Performance (part 2)
Mediation model to distinguish different mechanism by which the
increased informativeness may affect contribution quality and
productivity (estimated using PROCESS macro)

15. Dependent variables
RETENTION_RATE
The proportion of initially rejected contributors (whose first question is rejected) that submit another question
within 84 days from the rejection. Measured for each day.
MEAN_SCORE
The mean score for all questions submitted by contributors who submitted their initial question on the same
day (after the initial question). Measured for each day.
QUANTITY
The mean number of questions submitted by contributors who submitted their initial question on the same
day (after the initial question). Measured for each day.

16. Results and contributions

17. Contributor retention
Authors’ names blinded for peer review
Article submitted to Management Science; manuscript no. 49
0%
25%
50%
75%
100%
−84 −56 −28 0 28 56 84
Days from the treatment
Retention Rate After an Initial Rejection
Figure A.5 The figure follows a common practice in regression discontinuity design studies to plot a
Kernel function uniform | triangular
Registration delay Max. 1 day
Closure delay Max. 14 days
Sample window ±84 days
Notes. 1
We use separate, data-driven MSE optimal bandwidths before and after the treatment date.
0,U
1,U 0,T
1,T
−0.25
0.00
0.25
0.50
0.75
1.00
1 2 3 4
Estimation ID
Base scenario
Average Treatment Effect (ATE)
Figure 3 Average treatment e↵ects in base scenario estimations including Bonferroni-corrected 95%

18. Contributor retention
More informative rejection notices improve the retention of initially
rejected contributors by 21.7 percentage points (mean ATE).
Authors’ names blinded for peer review
Article submitted to Management Science; manuscript no. 19
Table 3 Estimation Results for the Base Scenario.
Estim. Tuning parameters
ID ATE SE 95% CI Poly. order Kernel Bandwidth before Bandwidth after
1 0.166 (0.033) 0.084 — 0.249 0 Uniform 13.7 5.3
2 0.219 (0.048) 0.101 — 0.338 1 Uniform 17.4 14.5
3 0.223 (0.032) 0.143 — 0.303 0 Triangular 11.8 7.4
4 0.261 (0.042) 0.156 — 0.366 1 Triangular 21.1 18.9
Notes. ATE, average treatment e↵ect; SE, standard error; CI, robust Bonferroni-corrected confidence interval (↵ = 0.0125)
We find that the average treatment e↵ect is statistically significant, positive, and of similar size in
all estimations that make up the base scenario. This suggests that more informative rejection notices
improve the retention of initially rejected contributors. Table 3 shows the average treatment e↵ect
(ATE), its standard error and robust, Bonferroni-corrected 95% confidence interval, and tuning

19. Contributor retention
The difference in the proportion of
initially rejected contributors who
have submitted a second question
holds steady still one year after the
rejection (p=0.04).
The treatment effect does not
taper off over time.
Authors’ names blinded for peer review
20 Article submitted to Management Science; manuscript no.
+
+
0%
10%
20%
30%
40%
50%
0 100 200 300
Days after the initial rejection
Has
submitted
a
second
question
+
+
After treatment
Before treatment
Contributor Retention After the Initial Question Is Rejected
Figure 4 The proportion of initially rejected contributors who have asked a second question until 365 days
after the rejection. The dashed vertical line marks the 84-days continuance threshold used in the base scenario.

20. Contributor retention
Authors’ names blinded for peer review
Article submitted to Management Science; manuscript no. 21
Table 4 Robustness Checks.
Check Outcome
Sorting around the treatment date Contributors do not self-select into a treatment or control group
Covariate smoothness Covariates remain smooth over the treatment date
Bandwidth sensitivity
Seven-day bandwidth Treatment e↵ect remains significant and nearly unchanged
Double bandwidth Treatment e↵ect remains significant but is slightly reduced
Registration and closure delay sensitivity Treatment e↵ect does not change at di↵erent parameter values
Falsification tests
E↵ect on non-treated questions No statistically significant e↵ect on non-treated questions
Change in the rejection rate No statistically significant e↵ect on rejection rate
Capacity to detect a true discontinuity The discontinuity stands out from noise in the data

21. Contributor performance Authors’ names blinded for peer review
26 Article submitted to Management Science; manuscript no.
Improved
Informativeness
Retention
Performance
a b
c’
H1
H2a, H3a
Mediated effect: H2b, H3b
Figure 6 Theoretical Model.
onward to 1. Retention is measured as the daily retention rate of initially rejected contributors as
Retention
analysis
Selection mechanism
(mediated effect)
Performance improvement
mechanism (direct effect)
Measures:
MEAN_SCORE,
QUANTITY

22. Contributor performance Authors’ names blinded for peer review
28 Article submitted to Management Science; manuscript no.
Table 6 Mediation Analysis Results.
Response Direct e↵ect (c’) Indirect e↵ect (ab)
variable Coefficient SE 95% CI E↵ect SE 95% bootstrap CI
MEAN SCORE -23.703 (22.319) -67.777 — 20.370 (H2a) 5.331° (4.082) -0.567 — 15.121 (H2b)
QUANTITY -8.384 (12.881) -33.822 — 17.053 (H3a) 4.705* (2.712) 0.287 — 10.823 (H3b)
Notes. Indicative p-value equivalents: *** p < 0.001, ** p < 0.01, * p < 0.05, ° p < 0.1; SE, standard error; CI, confidence interval
6.2.1. Robustness Check. The literature on the regulation of behavior in online communities
suggests that contributors can be nudged to make better contributions (Piezunka and Dahlander
2019, Ren et al. 2018, Srinivasan et al. 2019), yet we find no evidence of quality improvement
triggered by more informative rejection notices. To ensure that we are not missing such an e↵ect,
we devise a further check to test if we can detect any quality improvement. To do this, we now
focus on the quality improvement from the initial question to the second question instead of the
mean quality of all questions over the contributor’s entire tenure in the system. While it is intuitive
More informative rejection notices improve the retention of
contributors (selection mechanism) who are more productive (submit
more questions over their entire tenure in the system) and maybe also
higher quality questions.

23. Summary of results
1. A minor change to the wording of the rejection notices that reduced the
uncertainty of outcome if the rejected contributor tries again resulted in a
substantial increase in the retention of initially rejected contributors.
2. The impact on retention is long-lasting; it does not taper off as time passes
from the initial rejection.
3. The newly retained contributors are more productive as they submit more and
possibly better-quality questions.

24. Contributions
1. We extend earlier results on contributor retention by theoretically explaining
the improved retention with the reduced uncertainty about future interactions
(Jhaver et al. 2019, Piezunka and Dahlander 2019, Srinivasan et al. 2019).
2. We identify a selection mechanism by which the more informative rejection
notices improve also contributor performance.
3. We offer a template for studying the impact of rejection communications in
other open knowledge collaboration systems.

25. Thank You!

26. Base scenario tuning parameters
Authors’ names blinded for peer review
18 Article submitted to Management Science; manuscript no
Table 2 Tuning Parameter Values in the Base Scenario.
Tuning parameter Value(s)
Polynomial order 0 | 1
Bandwidth(s) variable1
Kernel function uniform | triangular
Registration delay Max. 1 day
Closure delay Max. 14 days
Sample window ±84 days
Notes. 1
We use separate, data-driven MSE optimal bandwidths before and after the treatment date.

27. The Data Studies Bibliography is a curated, searchable bibliography of
papers that focus on data as an object of research. The bibliography is
available at https://DataStudiesBibliography.org and maintained by
Aleksi Aaltonen (Temple University) and Marta Stelmaszak (Portland
State University).
Just added!

28. Presentation history
Date Institution / event Title
May 8, 2024 Tilburg School of Economics and Management Not Good Enough, But Try Again! The Impact of Improved Rejection
Communications on Contributor Retention and Performance in Open
Knowledge Collaboration
May 3, 2024 ÍESEG School of Management, Paris Not Good Enough, But Try Again! The Impact of Improved Rejection
Communications on Contributor Retention and Performance in Open
Knowledge Collaboration
February 11, 2022 Fox School of Business (MIS Distinguished
Speaker Series)
Not Good Enough But Try Again! Mitigating the Impact of Rejections on
New Contributor Retention in Open Knowledge Collaboration
December 3, 2021 University of Miami Not Good Enough But Try Again! Mitigating the Impact of Rejections on
New Contributor Retention in Open Knowledge Collaboration
October 24, 2021 CIST, Los Angeles Rejecting and Retaining New Contributors in Open Knowledge
Collaboration: A Natural Experiment in Stack Overflow
June 16, 2020 European Conference on Information Systems
(ECIS), online presentation
Rejecting and Retaining New Contributors in Open Knowledge
Collaboration: A Natural Experiment in Stack Overflow Q&A Service