A recent talk given at Syracuse University

1. Supporting Rationale Awareness in Large-Scale
Online Open Participative Activities
Lu Xiao
Associate Professor
Faculty of Information & Media Studies, Department of Computer Science
The University of Western Ontario
London, ON, Canada
lxiao24@uwo.ca

2. Outline
• Brief Introduction of my research background
• Rationale and Rationale Awareness – Definition and Related Studies
• Rationale Detection – Our Computational Approach
• Design for Rationale Awareness Support
• Future Work
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

3. About Me…
How to enrich computer-
mediated group activities?
Information
Science
Human-
Computer
Interaction
(HCI)
Data Science
Social
Sciences
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

4. Between-Team Communication in the Intercultural Context
Collaborative
decision-making
1. Xiao, L., & Huang, D. Y. (2015). Between-Team Communication in Cross-Cultural Context, Information, Communication & Society,
http://www.tandfonline.com/doi/abs/10.1080/1369118X.2015.1067709#.VcNEDPmniDg
2. Xiao, L., Buchel, O., & Huang, D. Y. (2014). The Role of Team’s Communication Practices in Between-Team Decision Making Activities,
Proceedings of iConference 2014
3. Xiao, L., Buchel, O., Martin, J., & Huang, D. Y. (2014), Towards Understanding of Contextual Factors in Information Sharing Behaviors of Inter-
Team Activities, 42nd Annual Conference of Canadian Association for Information Science & Inaugural Librarians' Research Institute Symposium
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

5. Design-based Parent-Child Mathematics Workshops
1. Xiao, L., Namukasa, I., & Zhang, Y. B. (2016). Design-based Mathematics Workshop, New Library World, to appear
2. Xiao, L., Cai, T. T., & Eastmure, V. (2015). Parent-Child Dialogues and Artifact Control Behavior in Computer vs. Non-Computer Mediated
Parental Interactions, Proceedings of iConference
3. Xiao, L., Martin, J. (2012) Supporting Parent-Young Child Activities with Interactive Tabletops: A Conceptual Analysis, In Proceedings of
the ACM 2012 conference on Computer Supported Cooperative Work Companion (CSCW '12)
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

6. Curating Testimony: Living
Archives of the Rwandan
Diaspora in Canada
1. Xiao, L., Elueze, I., & Kavanaugh, J. R. (2014) Human Rights Researchers' Data Analysis and Management Practices, Proceedings of the
77th Annual Meeting of Association for Information Science and Technology (ASIS&T)
2. Xiao, L., Yan, X. H., Miller, B., & Shrestha, A. (2014). Towards Visual Analytics for Digging into Human Rights Violations Data, Proceedings
of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)
3. Xiao, L., Elueze, I., & Kavanaugh, J. R. (2014) Human Rights Researchers' Data Analysis and Management Practices, Proceedings of the
77th Annual Meeting of Association for Information Science and Technology (ASIS&T)
4. Xiao, L., Luo, Y., & High, S. (2013). CKM: A Shared Visual Analytical Tool for Large-Scale Analysis of Audio-Video Interviews, at workshop
“Big Data and the Humanities” IEEE Big Data 2013
5. Vashchilko, T., & Xiao, L. (2013). Literature review: how content analysis is used to study human rights violations in political science?
Proceedings of 2013 iConference
Information Retrieval Pattern Identification
Information Sharing
Collaboration
Support
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

7. About Me…
How to enrich computer-
mediated group activities?
Information
Science
Human-
Computer
Interaction
(HCI)
Data Science
Social
Sciences
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

8. What is Data Science?
From the large size (Unstructured text) datasets,
using scientific (Natural Language Processing + Machine Learning) methods to:
1). identify and extract (useful) data to support the online social activities
2). identify language patterns to model online social behavior
Data
Pattern
Heterogeneity Size, Storage and Access
Serendipity
Batch, real-time
Science
Systematic Methods
Knowledge DiscoveryPrediction
ReliabilityValidity
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

9. Large-Scale Online Open Participative Activities
Opportunities & Challenges:
social footprints: users' profile attributes, their social context
and ties, and their published content
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

10. Detect Representative Rationale from
Wikipedia’s Article for Deletion (AfD)
Discussions
1. Mao, W. T., Xiao, L., & Mercer, R. (2015). Toward a Knowledge Repository for Wikipedia Article for Deletion (AfD) Discussions, poster at
the Third International Symposium of Chinese CHI. ACM, New York, NY, USA., to archive
2. Mao, W. T., Mercer, R., & Xiao, L. (2014a). Extracting Imperatives from Wikipedia Article for Deletion Discussions, First Workshop on
Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics (ACL),
http://acl2014.org/acl2014/W14-21/pdf/W14-2117.pdf
3. Mao, W. T., Xiao, L., & Mercer, R. (2014b). Using Text Similarity and Sentiment Analysis to Identify Representative Rationales in Large-
Scale Online Deliberations, 5th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis at the 52nd
Annual Meeting of the Association for Computational Linguistics (ACL), http://acl2014.org/acl2014/W14-26/pdf/W14-2624.pdf
4. Xiao, L., Mao, W. T., Mercer, R., & Nickerson, J. (under review). A computational approach to recognizing imperatives in online
discussions
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

11. 1. Khazaei, T., Xiao, L., & Mercer, R. (2015). “Change My View”: Detecting Persuasive Dialogs in Online
Discussions, the 2015 Social Media & Society Conference (Jul. 27 – 29, Toronto, Canada)
2. Xiao, L., Khazaei, T., & Mercer, R. (under review), Writing to persuade: Analysis and Discovery of Persuasive
Comments in Online Interactions
Detect Persuasive Texts in “Change
My View” Sub-Reddit Discussions
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

12. Detect Users’ Concern for Privacy in
Twitter Data
1. Khazaei, T., Xiao, L., Mercer, R., & Khan, A. (2016), Privacy Behaviour and Profile Configuration in Twitter, In WWW Workshop on
Modelling Social Media (MSM '16) (April 11, Montreal, Canada)
2. Khazaei, T., Xiao, L., Mercer, R. (2016). Detecting Privacy Preferences from Online Social Footprints: A Literature Review, 2016
iConference (Philadelphia, PA, USA)
3. Khazaei, T., Xiao, L., Mercer, R., Khan, A. (2015). Social Computing and Intelligence: Exploring Opportunities for the Public and
the Enterprise, 2015 IBM CASCON Emerging Technology Track
4. Khazaei, T., Xiao, L., Mercer, R., & Khan, A. (under review), Privacy Preference Inference via Collaborative Filtering
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

13. Large-Scale Online Open Participative Activities
Challenge for the Participants:
To identify, interpret, and evaluate
the others’ ideas and opinions
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

14. Rationale: one’s justification
of her ideas and opinions
Rationale awareness: one’s
awareness of the others’
rationales in the activities
Supporting the Participants’ Rationale Awareness
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

15. Rationales and Argumentation
“justifying a claim is making a
communicative move that involves
pragmatic as well as semantic constraints”
(Bermejo-Luque, 2011)
Luque, L. B. (2011). Giving reasons: A linguistic-pragmatic approach to argumentation theory (Vol. 20). Springer
Science & Business Media.
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

16. Rationale Studies – The Individual Level Focus
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

17. Rationale Awareness in the Group Activities
Brainstorming activities in small groups and large online crowd contexts
1. Xiao, L. (2014). Effects of rationale awareness in online ideation crowdsourcing tasks, Journal of the American Society for Information
Science and Technology (JASIST), 65, 1707-1720, doi: 10.1002/asi.23079
2. Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study. International Journal of e-
Collaboration, 11(4), Article 2, 11-39
3. Xiao, L., & Carroll, J. M. (2013) The Effects of Rationale Awareness on Individual Reflection: Processes in Virtual Group Activities,
International Journal of e-Collaboration, 9(2), 78 - 95
4. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software ,
86(7), 1727 – 1737
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

18. Rationale Awareness in the Group Activities
Decision-Making Activities: Wikipedia’s Article for Deletion (AfD) and small
group decision-making
1. Xiao, L., & Askin, N. (2014). What Influences Online Deliberation? A Wikipedia Study, Journal of the American Society for Information
Science and Technology (JASIST), 65: 898–910
2. Xiao, L., & Askin, N. (2015). Rationale Sharing in Large-Scale Online Deliberations, Proceedings of iConference
3. Xiao, L., & Witherspoon, R. (2015). The Effects of Pre-Discussion’s Note-Taking in a Hidden Profile Tasks, Proceedings of the 78th Annual
Meeting of Association for Information Science and Technology (ASIS&T)
4. Xiao, L., & Witherspoon, R. (2015). Group Decision Making As Story Construction, Proceedings of the 78th Annual Meeting of
Association for Information Science and Technology (ASIS&T)
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

19. Knowing the others’ rationales in the group activities is
beneficial
Individual
• Understanding and evaluating the others’ ideas and perspectives
• Knowledge awareness
• Contribution awareness
• Reflective thinking
Group
• Quality monitoring and control
19
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

20. • Authority claim detection (Bender et al., 2011; Marin et al., 2011)
• Opinionated claim detection (Rosenthal & McKeown, 2012)
• Justification identification (Biran and Rambow, 2011)
- detect the “common” discourse relations in the rationale texts
- Rhetorical Structure Theory (RST)
- 12 selected RST relations based on the RST Tree Bank corpus
Rationale Detection: Related Work
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work
Biran, O., & Rambow, O. (2011). Identifying justifications in written dialogs by classifying text as argumentative.
International Journal of Semantic Computing, 5(04), 363-381.

21. Rhetorical Structure Theory (RST): a theory of text organization created in the 1980s
[J.1] My first
challenge is fairly
general for the
project, but still very
much important.
2-4
Interpretation
Meeting remotely in
groups can be very
difficult
Joint
3-4
Joint
and having each
member meet at the
same time is
important in
having group
synergy. (37)
Purpose
Rhetorical Structure Theory
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

22. Rationale Detection: A Corpus-based Lexical Cue Graph Model
(Khazaei & Xiao, 2015; Khazaei, Xiao, & Mercer, 2015)
Step 1: Identify the RST relations that are commonly present in the
rationale texts
Three RST relations are reported to be commonly present in the rationales:
CIRCUMSTANCE, EVALUATION, and ELABORATION
1. Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study.
International Journal of e-Collaboration, 11(4), Article 2, 11-39
2. Xiao, L. (2013) Do Members Converge to Similar Reasoning Styles in Teamwork? A Study of Shared Rationales
in Small Team Activities, Proceedings of 2013 iConference
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

23. Rationale Detection: A Corpus-based Lexical Cue Graph Model
(Khazaei & Xiao, 2015; Khazaei, Xiao, & Mercer, 2015)
Step 1: Identify the RST relations that are commonly present in the
rationale texts
Step 2: Identify the lexical cues for these RST relations based on two corpora
but it has not been
sufficiently effective,
CONCESSION
EXPLANATIONEnergetic and concrete action
has been taken in Colombia
during the past 60 days against
the mafiosi of the drug trade, because, unfortunately, it came
too late.
Corpora
• RST Discourse Treebank
• SFU Review Corpus
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

24. Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work
Step 1: Identify the RST relations that are commonly present in the rationale texts
Step 2: Identify the lexical cues for these RST relations based on two corpora

25. Step 3: Use the lexical cues to detect these RST relations so as to detect the
rationale texts
1. Khazaei, T., & Xiao, L. (2015a), Corpus-based Analysis of Rhetorical Relations: A Study of Lexical Cues, In Proceedings of the
IEEE International Conference on Semantic Computing (ICSC), pp. 417-423, 2015
2. Khazaei, T., & Xiao, L. (2015). Computational Analysis of Collective Intelligence –Towards Automatic Detection of
Rationales in Online Deliberations, In Proceedings of 2015 Collective Intelligence
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work
Step 1: Identify the RST relations that are commonly present in the rationale texts
Step 2: Identify the lexical cues for these RST relations based on two corpora

26. Step 3: Use the lexical cues to detect these RST relations so as to detect the
rationale texts
Step 4: Use the probabilistic graph model G (V, E) to disambiguate the cues
V: the set of all possible tokens that may appear in the representations (e.g., POS tags)
E: the set of all possible ordered transitions between any two tokens
Weight of an edge E (i,j): the probability of a transition from token i to token j
Khazaei, T., Xiao, L., Mercer, R. (2015), Identification and Disambiguation of Lexical Cues of Rhetorical Relations across Different
Text Genres, Workshop "Computational Semantics: Linking Lexical, Sentential and Discourse-level Models"(LSDSem) at 2015
Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work
when
NNP
root
advcl advmod
Step 1: Identify the RST relations that are commonly present in the rationale texts
Step 2: Identify the lexical cues for these RST relations based on two corpora

27. Our approach vs. direct usage of syntax
RST
SFU
Best performance in RST relation labeling in the literature (F-measure):
61.5% on single genre (Ji and Eisenstein, 2014)
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

28. Rationale Detection: A Corpus-based Lexical Cue Graph Model
(Khazaei & Xiao, 2015; Khazaei, Xiao, & Mercer, 2015)
Step 1: Identify the RST relations that are commonly present in the
rationale texts
But…
Are these three RST relations commonly present in
the rationales independent of the context?
Three RST relations are reported to be commonly present in the rationales:
CIRCUMSTANCE, EVALUATION, and ELABORATION (Xiao, 2013; Xiao & Carroll, 2015)
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

29. Rutgers Argument Mining Corpora
(Wacholder, Muresan, Ghosh, Aakhus, 2014)
Corpora Total No. of
Callouts
Total No. of Callouts that
contain rationales
android_100 177 76
ban_100 138 103
iPad_100 234 100
Layoffs_100 200 151
Twitter_100 191 97
Overview of the Rutgers Argument Mining Corpora and Our Annotations
Re: #18 You must be a shill for the RIAA and others. Truth is that file sharing in fact
has been a boon to truly independent filmmakers and other non affiliated content
producers, because they've become better known; it only really harms the big sharks,
who have been robbing everyone blind and suppressing the little guy for decades.
Secondly, piracey is illegal. You aren't sharing anything. It's just a eufimisom for
STEALING, and that's exactly what it is. It cheats the creator out of their money.
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

30. RST Annotation in the Rationale Texts
(Xiao, under review)
Step 1: Rationale Region Annotation
1) Iterative open-coding process to identify the coding guidelines
2) Inter-coder reliability check:
Cohen’s Kappa (the annotation of 347 comments): 70.3%
3) Two researchers coded the rest data and examined each other’s results
Step 2: RST Relation Annotation
• An experienced RST annotator
• Intra-coder reliability check (relation labeling): 70% exact match and 7%
partial match
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

31. "Common" RST
relations
Percentage in
Coded RST
relations
Justify 19.11%
Elaboration 13.31%
Conjunction 9.74%
Contrast 8.81%
Joint 7.73%
Evaluation 5.93%
Circumstance 4.75%
Condition 4.61%
Non Volitional
Cause
3.81%
Concession 3.22%
81.03%
Corpora Percentage in
coded RST
relations
android 77.81%
ban 76.79%
ipad 81.82%
layoff 84.08%
twitter 82.05%
Common RST relations in the callouts that
have rationales
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

32. RST Relations
from (Biran & Rambow, 2011)
Contrast
Non Volitional Cause
Concession
Justify
Conjunction
Joint
Evaluation
Circumstance
Condition
Elaboration
Common RST relations in the callouts
that have rationales
Contrast
Cause
Concession
Antithesis
Analogy
Consequence
Evidence
Example
Explanation –
argumentation
Purpose
Reason
Result
1. Xiao, L. (under review), Discourse Relations in Rationale-Contained Text Segments
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

33. Automatic
detection of the
individuals’
rationales
Data-Centered
Study
When and How to
present the rationales
Design
•Performance
measure
•Behavior change
Data-Centered
Evaluation
The effects of the
rationale
awareness
Rationale
Studies
Research Methodology: Data-Centered Approach
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

34. Shared Rationale Space
Shared
Rationale
Space
34
WYSIWIS
1. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software,
86(7), 1727 – 1737
2. Xiao, L. (2011) Design for articulating and sharing rationales in virtual group learning activities, The IASTED International Conference on
Technology for Education and Learning (TEL)

35. Shared Rationale Space
Shared
Rationale
Space
35
1. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software,
86(7), 1727 – 1737
2. Xiao, L. (2011) Design for articulating and sharing rationales in virtual group learning activities, The IASTED International Conference on
Technology for Education and Learning (TEL)
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

36. Future Work
• Improvement of the discourse relation detection approach for
identifying RST relations
• Exploration of other rationale indicators
• Design guidelines for rationale awareness tools
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work

37. Large-Scale Online Open Participative Activities
Data Science Opportunities:
Social footprints
Outline Introduction Rationale
Automated Detection
of Rationales
Design of Awareness
Tools
Future Work
Challenges
• Common Ground
• Skill Development (e.g., social
deliberative skill, reflection skill)
• Knowledge Management

38. Acknowledgement
Colleagues at Penn State and the University of Western Ontario