This document summarizes research on the role of emotions in questions asked on Stack Overflow. Through logistic regression of over 87,000 questions, the study found that expressing positive or negative emotions decreases the chance a question will be successfully answered, while factors like reputation, code snippets, and body length impact success. Qualitative analysis revealed misclassifications of sentiment and differences in emotional direction based on reputation. User surveys confirmed quantitative findings but also identified perceptions of rudeness. Guidelines combining evidence and user perspectives were proposed to improve question writing. Ongoing work includes refining sentiment analysis tools and building mentoring plugins to mitigate issues.

1. Building evidence-based guidelines:
the role of emotions in Stack Overflow
Filippo Lanubile
University of Bari
15th International Advanced School on Empirical Software Engineering (IASESE 2018)
October 10, 2018 - Oulu, Finland

3. Community-based
Q&A: Stack Overflow
• Largest SE Q&A
community
• SO Annual Developer Survey 2018
insights.stackoverflow.com/survey/2018
– ~50M visits per month
• ~21M are professional developers and university-level
students (25%)
– > 80% rely on SO when learning
something new
• Developers read manuals less and less,
they rather “search”
(Mary Shaw’s keynote at ICSE 2016)

5. Research question 1
• How an information seeker
can increase the chance of
eliciting an answer?
Or better
• How an information seeker
can increase the chance of
eliciting a good answer?
But how do we know whether
an answer is actually good?
• How an information seeker
can increase the chance of
eliciting a useful answer?
One of the answers is
marked as accepted by the
question asker
 successful (or resolved)
question

6. Research question 2
What is the role of emotions in Stack Overflow?

7. Complaints about harsh
interactions, downvotes, and
handling of duplicate questions

8. https://stackoverflow.blog/2018/08/07/get-to-know-our-new-code-of-conduct/

9. Refining RQ 2
We focus on question-writing
What is effect of expressing emotions when
asking a question in Stack Overflow?
“I have very simple and stupid trouble […]. I'm
pretty confused, explain please, what is wrong?”
“[…] Any help would be really great! :-)”
“I want them to resize based on the length of the
data they’re showing”

10. Theoretical framework:
critical factors of successful questions
Actionable
• Affect
• Presentation
quality
• Time
Non Actionable
• Reputation
Knowledge sources
Prior research findings about Q&A sites
– technical, linguistic, and human aspects

11. Mixed-methods approach
Sequential explanatory strategy Steps
1. Operationalization of success
factors
2. Data preparation
3. Logistic regression
4. Qualitative analysis of
emotions
5. User survey
Collection/analysis of
quantitative data
Collection/analysis
of qualitative data
Interpretation

12. Operationalization of
success factors

13. Reputation factor
Model
Based on SO privileges
• New → Q&A
• Low → vote, comment
• Established → edit
• Trusted → delete Q&A
Measurement
Based on points received
for acknowledged help
• New: <10
• Low: [10, 1K[
• Established: [1K, 20K[
• Trusted: ≥ 20K

14. Modeling the Affect factor
Russel, 1980 - A circumplex model of affect. J. Personality and Social Psychology, 39, 1161-1178.
Polarity
(valence)
ActivationRussel’s
Circumplex Model
of Affect

15. Measuring the Affect factor
Sentiment analysis
• Classification of a text according to its positive,
negative or neutral semantic orientation
How? SentiStrength
• Estimates the
strength of both
positive and
negative polarity in a text
Excerpts from the Stack Overflow Sentiment Strength
Scores
Discretized Sentiment
Scores
Positive Negative Positive Negative Neutral
“I have very simple and stupid trouble […].
I'm pretty confused, explain please, what is
wrong?”
+1 -2 False True False
“[…] Any help would be really great! :-)” +5 -1 True False False
“I want them to resize based on the length
of the data they’re showing”
+1 -1 False False True

16. Data preparation
from July 2008
to Sept 2014
13M questions
SO dump
Only in the last month
but at least 3 days old
No self-answers
No closed or removed
No edits
87K questions
Filtering
• Discard code
snippets
• Remove HTML
tags
Cleaning

17. Logistic regression
• Dependent variable
– success of a question: yes (35%) vs no (65%)
• Independent variables

18. Logistic regression:
results about Reputation
Predictor Odds
Ratio
‘New’ as
default value
Low [10, 1K[ 2.39
Established
[1K, 20K[
2.67
Trusted
[20K,+∞[
3.24
Many ‘New’ users
forget to accept useful
answers!

19. Logistic regression:
results about actionable factors
• Affect
– Positive Sentiment (OR=0.94) down
– Negative Sentiment (OR=0.81) down
• Presentation quality
– Few Uppercase chars (OR=1.27) up
– Body Length > 199 words (OR=0.65) down
– Code snippets (OR=2.04) up
• Time
– GMT evening hours (OR=1.16) up

20. Simulating the improvement
of a bad question

21. Qualitative analysis of
emotions
Subset of questions across 4 reputation categories
• 120 questions with positive sentiment
• 120 questions with negative sentiment
as measured by SentiStrength
Content analysis according to
• Sentiment polarity (misclassification)
• Sentiment direction
– Object
– Self
– Other

22. Qualitative analysis of emotions:
misclassification of sentiment polarity
Established and Trusted users
are richer in technical details and jargon

23. Qualitative analysis of emotions:
direction of negative sentiment
New users express more frustration

24. Qualitative analysis of emotions:
direction of positive sentiment
New and Low reputation users
pay gratitude forward to potential helpers

25. No chit-chat

26. User survey
Logistic regression
model
87K questions
Online survey
open-ended questions
one multiple choice question for each predictor
43 developers

27. Some responses to
open-ended questions
People on Stack Overflow
can be extremely rude
Novice users get flamed
in this forum
Now I access Stack Overflow
through Google without a user
account
It’s my last resort, I don’t really
want to deal with their arrogant
comments
“
“

28. # Guideline
Success
factor
Empirical
support
User
perception
Source
1
Write questions using
a neutral emotional
style
Affect Yes Effective
Skeet, SO Help Center,
Kucuktunc et al., Bazelli et
al.
2
Provide sample code
and data
Presentation
quality
Yes Effective
Skeet, Asaduzzaman et al.,
Duijn et al., Treude et al.
3
Use capital letters
where appropriate
Presentation
quality
Yes Effective Skeet
4 Be concise
Presentation
quality
Yes Ineffective Skeet
5
Use short, descriptive
question titles
Presentation
quality
No Ineffective Skeet
6
Provide context
through tags
Presentation
quality
No Effective Skeet
7
Provide context
through URLs
Presentation
quality
No Effective Ponzanelli et al.
8
Be aware of low-
efficiency hours
Time Yes Ineffective Bosu et al.
=
=
=
=


= match
 mismatch


Netiquette for effective question writing on SO:
Evidence vs. User Perception

29. Work in progress:
SentiStrength → Senti4SD
• Supervised:
gold standard of over 4K
posts from Stack Overflow
– questions, answers, and
comments
manually annotated for
sentiment polarity
• Feature set
– Lexicon
– Keyword
– Semantics
• Reduced “negative” bias
– SentiStrength tends to
misclassify neutral as
negative
Tool and dataset
github.com/collab-uniba/Senti4SD

30. Work in progress:
Virtual mentor for Q&A (qavmentor)
Chrome plugin + API
github.com/collab-uniba/qavmentor-plugin github.com/collab-uniba/qavmentor

31. Conclusions: we do our best
to follow the scientific method
• Observe
• State the problem
• Theory
• Experimentation
• Communicate to
increase awareness
• Improve
instrumentation
• Tool building to
mitigate the problem
• Design new empirical
studies to test
changes to the theory

32. @lanubile
filippo.lanubile@uniba.it