Slides presented at WikiSym 2012. This paper presents a large-scale analysis of emotions in conversations among Wikipedia editors. Our focus is on the emotions expressed by editors in talk pages, measured by using the Affective Norms for English Words (ANEW). We find evidence that to a large extent women tend to participate in discussions with a more positive tone, and that administrators are more positive than non-administrators. Surprisingly, female non-administrators tend to behave like administrators in many aspects. We observe that replies are on average more positive than the comments they reply to, preventing many discussions from spiralling down into conflict. We also find evidence of emotional homophily: editors having similar emotional styles are more likely to interact with each other. Our findings offer novel insights into the emotional dimension of interactions in peer-production communities, and contribute to debates on issues such as the flattening of editor growth and the gender gap.

1. Emotions and dialogue in a peer-production
community: the case of Wikipedia
David Laniado Carlos Castillo
Barcelona Media Foundation Qatar Computing Research Institute
Andreas Kaltenbrunner Mayo Fuster Morell
Barcelona Media Foundation Berkman Center, Cambridge, MA, USA
August 27th , 2012
WikiSym ’12, Linz, Austria
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 1 / 37

2. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 2 / 37

3. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 3 / 37

4. Introduction
“Like other aspects of culture, emotions can be seen as an aspect of
all social action and social relations. They accompany rational acts as
fully as irrational ones, positive experiences as much as negative
ones” (Goodwin, 2001)
Goal: Shed light on the emotional dimension in a large peer
production community
focus: English Wikipedia
extensive analysis of emotions expressed in talk pages
Article talk pages → discussions about how to improve articles
User talk pages → a kind of public in-boxes
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 4 / 37

5. Research questions
1 How are the emotional styles of editors affected by their level of
experience?
2 How are the emotional styles of editors affected by their gender
and the topics they choose to work on?
3 How are the emotional expressions affected by interacting with
others in comment threads (emotional congruence)?
4 How are the emotional styles of editors related to those of the
editors they interact more frequently with (emotional
homophily)?
5 How are the emotional styles of editors expressed in their
personal spaces (user talk pages)?
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 5 / 37

6. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 6 / 37

7. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 7 / 37

8. Dataset: conversations
Extracting conversations among editors from the English Wikipedia
Articles 3 210 039
Articles with talk page (ATP) 871 485 (27.1%)
Total comments in ATP 11 041 246
Comments in ATP linking to policies 460 644 (4.1%)
Users who comment articles 350 958
Users with ≥ 100 comments on ATP 12 231 (3.5%)
Table: Data extracted from a complete dump of the English Wikipedia, dated
March 12th, 2010
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 8 / 37

9. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 9 / 37

10. Identifying user gender
≈ 12 000 users wrote ≥ 100 comments in articles talk pages
Gender identified through Wikipedia API for ≈ 2 000 of them
Out of the remaining ones, a sample of 1 385 users for manual
labelling through crowdsourcing (Crowdflower)
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 10 / 37

11. Manual user gender labelling
Gender could be identified only for ≈ 50% of users:
real name or username (50% of those identified)
implicitly stated gender (27% of females, 20% of males)
pronoun (15% of females, 10% of males)
other indicators: userboxes, pictures, links to personal blogs...
Non-admins Admins Total
Males 1 087 1 526 2 613
Females 68 97 165
Unknown 6 850 2 603 9 453
Total 8 005 4 226 12 231
Table: Users with ≥ 100 comments by gender and administrator status.
Category “unknown” includes:
8 708 users that were not included in the crowd-sourced task university-logo
745 users whose gender could not be identified by evaluators
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 11 / 37

12. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 12 / 37

13. Measuring the Emotional Content of Discussions
Use ANEW list
Rates a list of 1060 frequent words on a 9 point scale in three
dimensions:
Valence: positive
vs negative emotions
Arousal: strength of the
emotions
Dominance: feeling in control
vs feeling dominated
Bradley and Lang. (1999).
Affective norms for English words (ANEW) Technical report C-1.
The Center for Research in Psychophysiology, University of Florida, FL. university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 13 / 37

14. Example messages
Table: Example messages with their corresponding Valence, Arousal, and
Dominance scores (average over the words contained in each message).
V A D
Sounds like a good challenge - to be proven or disproven. I’m happy 7.4 5.3 6.2
if it can be shown to go further using closed cubic polynomial solu-
tions. The nice thing about these are that they are pretty easy to
test numerically . . .
–in “Exact trigonometric constants”
Seems you have not yet seen female lover after having sex who 5.5 7.0 5.2
do not wish to have sex with the same lover any more :) Once
you’ve seen it, you understand very well what war of Venus means
compared to war of Mars.
–in “House (astrology)”
What about the whirlie hazing, the alcohol abuse, the emotional 1.6 5.8 3.5
poverty, the suicide in 1995/6, the biotech plans which were
stopped by pitzer protests . . .
–in “Harvey Mudd College”
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 14 / 37

15. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 15 / 37

16. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 16 / 37

17. Emotions and edit count
corr= 0.12, p−value<10−8
1
Male
Female
mean dominance normalized
0.5
0
−0.5
100 1000 10000 100000
log. of edit count
Positive correlation between edit count and dominance
university-logo
→ more experienced users tend to feel “in control” more than newbies
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 17 / 37

18. Emotions and role
Distribution of the ANEW-valence scores of admins vs non-admins
valence
4.8 5 5.2 5.4 5.6 5.8 6 6.2 6.4 6.6 6.8 7
0.25 1
Normal
Admin 0.9
Normal: mean ± SEM
0.2 Admin: mean ± SEM 0.8
P(Admin|mean(valence)=x)
P(Admin|mean(valence)=x)
0.7
0.15 0.6
pdf
0.5
0.1 0.4
0.3
0.05 0.2
0.1
0 0
4.8 5 5.2 5.4 5.6 5.8 6 6.2 6.4 6.6 6.8 7
mean valence
Distribution of the ANEW-valence scores of admins vs non-admins
mean value and the interval of mean ± the standard error of the mean (SEM)
dashed lines indicate probability of being an administrator given the mean valence of
university-logo
a user (assuming equal sample sizes)
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 18 / 37

19. Emotions and role
Mean (±SEM) of valence and dominance for regular users and administrators
valence dominance
5.45
Admin
6.1
Non−admin
5.44
6.08
5.43
6.06 p-values
mean dominance
mean valence
6.04
5.42 *** p < 0.001
** p < 0.01
5.41
6.02 * p < 0.05
6 5.4
5.98 5.39
5.96 5.38
All le le All le le
ma Ma ma Ma
Fe Fe
Comments by administrators have higher valence and dominance
university-logo
No significant difference between female admins and female non-admins
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 19 / 37

20. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 20 / 37

21. Emotions and gender
Words more used by females and males
Size accounts for difference in frequency university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 21 / 37

22. Emotions and gender
Mean (±SEM) of valence and dominance form males and females
valence dominance
5.45
Male
6.1
Female
5.44
6.08
5.43
6.06 p-values
mean dominance
mean valence
6.04
5.42 *** p < 0.001
** p < 0.01
5.41
6.02 * p < 0.05
6 5.4
5.98 5.39
5.96 5.38
All ins ns All ins ns
dm mi m mi
a Ad ad Ad
n− n−
No No
Females use words associated to more positive emotions university-logo
Difference no more significant when normalizing by article
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 22 / 37

23. Topics, emotions and gender
N≥1 ANEW words; corr=−0.64 (p=0.002)
5.6
Computing
Arts
5.5 Philosophy
Language
5.4 Health
Mathematics
5.3 Belief
mean valence
Sports
5.2 Agriculture
Environment
5.1 Techn. & app. sci.
Law
5 Society
Business
Education
4.9
Culture
People
4.8
Science
Politics
4.7 Geography and places
0.82 0.84 0.86 0.88 0.9 0.92 0.94 0.96
prop. of male comments History and events
Figure: Mean valence for discussions of articles in different topic categories,
university-logo
vs the proportion of comments written by male editors
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 23 / 37

24. Links to Wikipedia policies
Table: Percentage of messages including links to Wikipedia policies for
different classes of users
Non-admins Admins All users
Males 2.5% 4.9% 3.9%
Females 6.2% 12.4% 9.8%
Differences between genders are statistically significant with p < 0.01
Comments containing links to Wikipedia policies have
higher valence, higher arousal and higher dominance
when editors invoke community norms, they tend to do it with a
remarkably positive and dominant tone, and with stronger
emotional load
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 24 / 37

25. Message length and gender
Females write longer messages
83 vs 71 words on average
Difference is accentuated for female and male administrators
85 vs 68 words on average
different leadership style?
p-values
Differences are statistically significant with p < 0.05
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 25 / 37

26. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 26 / 37

27. Emotional congruence
Replies are more positive
On average, editors tend to reply with:
higher valence: +0.05
higher dominance: +0.04
no statistically significant differences for arousal
Users tend to be more positive and dominant when replying, but
without recurring to words evoking stronger sentiments
Results computed on all comments in article talk pages
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 27 / 37

28. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 28 / 37

29. Emotional homophily
Mixing patterns: do users interact preferentially with similar users?
- By activity: users who write more comments tend to reply preferentially
to less active users, end viceversa
+ By gender: females interact more with other females
+ By style: users interact more with others expressing similar emotions
edges connecting users who have
exchanged at least 10 replies
red nodes → 15% users expressing
higher valence in article discussions
black nodes → 15% users
expressing lower valence in article
discussions
size → proportional to the number of
connections university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 29 / 37

30. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 30 / 37

31. Emotions expressed on personal pages
Comments written by users on their own talk page
written by females: higher valence, dominance and arousal
by male administrators: higher valence and dominance, lower
arousal
male administrators are positive and dominant, but write less
emotional content
on the contrary women, including female administrators, express
stronger emotions on their personal pages
Comments received on personal talk pages
more positive emotions than in one’s own comments
females receive more positive messages, except for female
administrators
a supposed gender difference seems to be out-weighed by the type
of interactions the administrator tasks bring with them university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 31 / 37

32. Outline
1 Introduction
2 Framework of analysis
Data acquisition and pre-processing
User gender labelling
Sentiment analysis
3 Results
Emotions and experience
Topics and gender
Emotional congruence
Emotional homophily
Emotions in personal (“User talk”) pages
4 Conclusions
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 32 / 37

33. Conclusions (1)
Wikipedia editors express themselves in general with a positive
emotional tone
responses tend to be more positive than the comments to which
they reply
users receive more positive messages than they write on their
personal pages
Administrators and experienced users play a pivotal role
they tend to interact especially with less experienced users
they express themselves with a more positive tone
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 33 / 37

34. Conclusions (2)
Female editors are different in their emotional expression and
relationships
they work on topics in which the discussions have a more positive
tone
they receive more positive messages on their personal pages
they tend to behave similarly to administrators
however, there are some differences in the leadership style
females write longer messages, link more often Wikipedia policies,
and express stronger emotions in personal spaces
Wikipedians of a feather fly together
editors communicate more with others having a similar style
females interact more with other females
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 34 / 37

35. Recommendations
Design for positive emotional expressions
Besides the neutral point of view, provide spaces for emotional
expression, favoring positive exchanges and channeling negative
feelings in a non-disctructive manner
in particular, build upon the positive tone of user talk pages
Encourage experienced editors to contribute to positive emotions
leverage contagiousness of emotional styles to “cool down”
conflictive discussions
positive tone of messages linking to WP policies
Gender-aware recruiting
it may be advisable that females are invited by other females
consider that certain topics may be more attractive to females
university-logo
seek participation of females where the gender gap is more acute
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 35 / 37

36. Future work
Comparison with other metrics
preliminary results on the same data using other lexicons are
largely consistent
Longitudinal analysis
how do emotional styles of editors change over time and with
increasing experience?
how do emotions in the discussion affect participation?
Need for qualitative analysis and human annotation
include non-textual emotional aspects such as emoticons, barn
stars and virtual gifts
cover also less experienced users
deal with sarcasm, measure the extent of condescending or
paternalistic language in comments addressed at female editors
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 36 / 37

37. Main references I
Bradley and Lang.
Affective norms for English words (ANEW) Technical report C-1.
The Center for Research in Psychophysiology, University of Florida, FL, 2012.
B. Collier and J. Bear.
Conflict, criticism, or confidence: an empirical examination of the gender gap in wikipedia
contributions.
In Proc. of CSCW, 2012.
O. Kucuktunc, B. B. Cambazoglu, I. Weber, and H. Ferhatosmanoglu.
A large-scale sentiment analysis for Yahoo! answers.
In Proc. of WSDM, 2012.
S. K. Lam, A. Uduwage, Z. Dong, S. Sen, D. R. Musicant, L. Terveen, and J. Riedl.
WP:clubhouse? an exploration of Wikipedia’s gender imbalance.
In Proc. of WikiSym, 2011.
D. Laniado, R. Tasso, Y. Volkovich, and A. Kaltenbrunner.
When the Wikipedians talk: Network and tree structure of Wikipedia discussion pages.
In Proc. of ICWSM, 2011.
university-logo
Laniado, Castillo, Kaltenbrunner, Fuster Morell Emotions and dialogue in a peer-production community 37 / 37