1. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Making Sense of User Comments
Identifying Journalists’ Requirements for a Software Framework
Wiebke Loosen, Marlo Häring, Zijad Kurtanović, Lisa Merten,
Julius Reimer, Lies van Roessel & Walid Maalej

2. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
User comments: an ambivalent
phenomenon
2
Shift in the understanding of user comments: from deliberative
hopes to being a “necessary evil”, or even a threat to deliberation
(e.g., Blom et al., 2014; Kropczynski et al., 2015; Pond, 2016)

3. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
The paradox of audience participation
▶ Acknowledging the need to include audiences while
trying to maintaining professional distance from
audiences (e.g., Heise et al., 2014; Schmidt et al., 2013; Singer et al., 2011;
WAN-IFRA & Huang, 2016)
▶ Many journalists engage with user comments
(e.g., Chen & Pain, 2016; Graham & Wright, 2015; Santana, 2011)
▶ Finding good user comments is like finding a needle in
a haystack! (cf. Braun & Gillespie, 2011)
▶ Handling user comments takes time, effort, and a
cross-platform approach
3

4. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
▶ Constructive approach: Help journalists (and others)
make sense of user comments with digital tools
▶ Formulate and validate requirements for a software
framework for analyzing user comments by means
of an iterative approach
1. Identify features and develop a first mock-up
2. Discuss mock-up with journalists and comment
moderators to refine requirements
Considerations & research goals
4

5. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Method

6. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017 6
Research design
Literature Review,
own preliminary
research
Explorative
Interviews
Mock-up
Development
Group Discussion
Editors
Group Discussion
Audience
Engagement Team
Mock-up &
Requirements

7. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Topics for group discussions
7
▶ Practices: handling user comments within the newsroom
▶ Challenges: pressing issues when dealing with user comments
▶ Quality: what makes comments especially valuable or helpful?
▶ Mock-up as ‘stimulus’ to discuss and develop possible software
features

8. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Results

9. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Group Discussion: General insights
▶ Technical assistance in ‘making sense of user comments’ appreciated
▶ ‘Multichannel function’ considered extremely useful
▶ Help is needed to identify ‘response-worthy’ comments
▶ Classifying pro- and contra-arguments particularly useful
▶ Visualizing dynamics of news discussions valuable for newsrooms and users
▶ Off-topic and redundant comments are annoying (particularly for moderators)
▶ Enabling ‘commenter typologies’ was welcomed, but also met with data protection
concerns
▶ Editors also regard user comments as constructive feedback more than moderators

10. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Analytical dimensions derived from
literature review & interviews
10
▶ Topics and actors
▶ Debate
▶ Quality indicators
▶ Comparison
▶ Sociodemographics
▶ Further features

11. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Mock-up: actors & topics
11
‘multichannel function’
regarded extremely
useful

12. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Mock-up: debate
12
Classifying pro- and contra-
arguments considered
particularly useful
Visualizing the dynamics of
news discussions regarded
valuable for newsrooms and
users

13. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Mock-up: quality indicators
13
Off-topic and redundant comments are
annoying (particularly for moderators)
Editors regard user comments as
constructive feedback and would like
response-worthy comments to be
filtered
‘commenter typologies’ welcomed, but
also caused data protection concerns

14. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Overview
of features
14

15. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Discussion

16. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Challenges for technical implications
▶ Large volumes of typically natural, informal, and low-quality text across broad
range of topics
▶ Lexical and Syntactical machine learning needed to capture topics, actors &
arguments
▶ Availability of high-quality corpora of pre-labeled text
A lack of resources rather than a lack of ideas impede making sense of user
comments

17. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017 17
Thanks for listening!
https://scan.informatik.uni-hamburg.de/
Contact: w.loosen@hans-bredow-institut.de

18. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
References
▶ Blom, R., Carpenter, S., Bowe, B. J., & Lange, R. (2014). Frequent contributors within U.S. newspaper comment forums. An examination of their civility
and information value. American Behavioral Scientist, 58(10), 1314–1328.
▶ Braun, J., & Gillespie, T. (2011). Hosting the public discourse, hosting the public. When online news and social media converge. Journalism Practice,
5(4), 383–398.
▶ Chen, G. M., & Pain, P. (2016). Normalizing online comments. In: Journalism Practice. doi:10.1080/17512786.2016.1205954.
▶ Cision, & Canterbury Christ Church University. (2015). 2015 global social journalism study. Retrieved from http://cision-wp-
files.s3.amazonaws.com/de/wp-content/uploads/2015/09/Cision-SJS-2014-15-International.pdf
▶ Graham, T., & Wright, S. (2015). A tale of two stories from „below the line“. Comment fields at the Guardian. The International Journal of
Press/Politics, 20(3), 317–338.
▶ Heise, N., Loosen, W., Reimer, J., & Schmidt, J.-H. (2014). Including the audience. Comparing the attitudes and expectations of journalists and users
towards participation in German TV news journalism. Journalism Studies, 15(4), 411–430.
▶ Kropczynski, J., Cai, G., Carroll, J. M., Zhang, J., Puron-Cid, G., & Gil-Garcia, J. R. (2015). Characterizing democratic deliberation in an online forum.
Information Polity: The International Journal of Government & Democracy in the Information Age, 20(2/3), 151–165.
▶ Pond, P. (2016). Twitter time. A temporal analysis of tweet streams during televised political debate. Television & New Media, 17(2), 142–158.
▶ Rowe, I. (2015). Deliberation 2.0: comparing the deliberative quality of online news user comments across platforms. Journal of Broadcasting &
Electronic Media, 59(4), 539–555.
▶ Santana, A. (2011). Online readers’ comments represent new opinion pipeline. Newspaper Research Journal, 32(3), 66–81.
▶ Schmidt, J.-H., Loosen, W., Heise, N., & Reimer, J. (2013). Journalism and participatory practices – blurring or reinforcement of boundaries between
journalism and audiences? Recherches En Communication, (39), 91–109.
▶ World Association of Newspapers and News Publishers (WAN-IFRA), & Huang, C. L. (2016). Do comments matter? Global online commenting study
2016 (WPT/WEF Trends Report No. 1). Frankfurt/Main: World Association of Newspapers and News Publishers (WAN-IFRA).
18