I gave a talk at Xerox Europe Research Center (Grenoble, France) on Mar. 3. 2014. This included a couple of projects in my research lab that examine shared rationales in group activities.

1. Shared Rationales in Group Activities
1
Lu Xiao
Faculty of Information & Media Studies
Department of Computer Science
T h e U n i v e r s i t y o f We s t e r n O n t a r i o
Http://hii.fims.uwo.ca
14-03-04

2. Shared Rationales in Group Activities
2
Rationale - the information that justifies one’s ideas, approach, and solution in
group activities.
Related Studies:
Explanations in Knowledge-based Systems (KBS)
Types of explanations, Content of explanations, Effects of KBS Explanations (explanation
use behavior, learning, perceptions, and judgmental decision making)
Shared Information in Group Activities
— How and why group members share information
— The factors of information sharing, information pooling phenomenon
— The information practices and cultures that members develop
— The effects of shared information and aspects of the shared information (e.g.,
representation strategy, the use of language)
¡ Influence the change of people’s attitudes
¡ Shared reflections
Tools to support information sharing in group activities
— Group decision support systems for hidden profiles
— Tools for capturing, archiving, and reusing design rationales
— Tools to foster reflective thinking in group learning activities
14-03-04

3. Research Gaps
3
1). the effects of shared rationales in group activities
2). Design requirements to promote the processes of
articulating, sharing, and managing rationales in
group activities
14-03-04

4. The Role of Shared Rationales in Group Ideation
and Deliberation Activities
4
The effects of rationale awareness in
— small group ideation activities
— Large online crowdsourcing ideation activities
The effects of shared rationales in
— Large online deliberation activities
Rationale Awareness, as part of Activity Awareness (Carroll et al., 2003, 2005, 2011; Carroll, Rosson, Farooq, & Xiao,
2009), refers to one’s awareness of the other group members’ rationales in a group activity
14-03-04

5. The Effects of Rationale Awareness in Small
Group Ideation Activities
5
The effects of rationale sharing (Xiao, 2011a, 2012; Xiao &
Carroll, 2013)
• Rationale awareness can contribute to one’s
• awareness of others’ knowledge and intellectual
contribution; can affect the development of his/her
reflection skills
• Explicit rationale sharing has potential downsides such as
groupthink
14-03-04

6. The Effects of Shared Rationales in Online
Crowdsourcing Ideation Activities
6
Related Work:
— Quality measure of different means
¡ Providing
real-time assessment
¡ Collecting multiple assessment
¡ Analyzing workers’ behavior
¡ Parallel vs. iterative approach
14-03-04

7. The Effects of Shared Rationales in Large Online
Ideation Activities
7
In an ideation task performed through online
crowdsourcing processes, whether and how sharing
previous workers’ good rationales of their generated
ideas affects the ideas’ quality in the task?
Pros: awareness of the others’ knowledge and intellectual
contributions
Cons: problems with explicit rationale sharing
14-03-04

8. Research Design
In an idea evaluation task performed through online
crowdsourcing processes, whether and how showing the
idea’s rationale affect its evaluation?
Hypothesis:
Making the ideas’ rationales available to all of its
evaluators reduce the variation between evaluations by
multiple raters

9. Research Design
— Two iterative conditions in the idea generation task:
presence vs. absence of previous workers’ rationales
— Two idea evaluation conditions in the idea evaluation task:
presence vs. absence of the idea’s rationale
— Manipulation of the rationale’s quality: Experiment 1,
Experiment 2, and Experiment 3
¡
¡
The quality of the ideas and rationales was checked after all the
iterations were completed (Experiment 1)
The quality of the ideas and rationales was checked at the end of each
iteration (Experiment 2 and 3) – stricter and better quality control
of ideas and rationales

10. Research Design (Little et al.,2010)
Brainstorming/Idea
Generation Task
• Six company descriptions
• Five names for a company description
in each iteration
• Six iterations for each company
Rating/Idea
Evaluation Task
Each name had 10 ratings

11. Turkit
• Open source software: Java/JavaScript API for
running iterative tasks on Mechanical Turk.

12. Manipulation Check
12
14-03-04

13. Results – Average Quality of the Ideas
13
14-03-04

14. Results – Best Quality of the Ideas
14
14-03-04

15. Results: Rationale Awareness in Idea Evaluation

16. Findings: the Shared Rationales in Online
Crowdsourcing Ideation Task
— In an idea generation task, the awareness of previous
workers’ rationales may slightly improve the average
quality but NOT the best quality of the generated ideas
in iterative approach
— In an idea evaluation task, the awareness of an idea’s
rationale can affect the evaluation outcome and the
quality of the rationale may play a significant role on
the evaluation
(Xiao, CSCW, 2012; Xiao, CI, 2012; Xiao, JASIST, to appear)

17. The Role of Shared Rationales in Group Ideation
and Deliberation Activities
17
The effects of rationale awareness in
— small group ideation activities
— Large online crowdsourcing ideation activities
The effects of shared rationales in
— Large online deliberation activities
14-03-04

18. Shared Rationales in Large Online Deliberation
18
— Deliberation concept
¡ Habermas (1989) – public sphere
¡ Halpen and Gibbs (2013)
a communication process that involves at least two individuals;
that focuses on a social or political issue where the solutions
are identifiable by participants; and that values equality among
participation and emphasizes rational thinking and logic
instead of a power struggle.
14-03-04

19. Shared Rationales in Large Online Deliberation
19
Wikipedia’s Article for Deletion (AfD) discussions
Step 1 : Types of rationales; factors of deliberation outcome
What are the types of rationales used in the deliberation?
Are there any relationships among the kinds of votes, the
article’s topic, the discussion situation (unanimous or
non-unanimous), and the final decision?
14-03-04

20. Possible outcomes of deletion discussions
Outcome
Keep
Withdrawn
No consensus
Procedural close
Delete
Speedy delete
Userfy
Incubate
Merge
Rename or Move
Convert
Transwiki
Redirect
Split
Coded
as…
Explanation
Article is kept. Changes may or may not be suggested as part of debate.
Keep
20
Nomination is withdrawn.
Debate is inconclusive. This might result from disagreement, lack of
participation, or other factors. Article defaults to keep.
Debate is closed because of problems with the nomination.
Article is deleted.
Delete
Article is deleted under the “speedy” criteria outlined at WP:CSD.
Article is deleted but a copy is given to a user to work on as a draft, and may Other
be recreated as an article later.
As with userfy, but in a communal space rather than related to a single user.
Article is deleted but some or all of its content is added into one or more
existing articles.
Article’s title is changed. Its scope may or may not be amended.
Article is converted into another type of page, usually one with a structural
function. For example, a list might be changed into a category to be added to
the list entries.
Article is deleted from English Wikipedia but moved to another Wikimedia
project as appropriate – for example, a French article to French Wikipedia
or an image gallery to Wikimedia Commons.
Article’s content is replaced with a pointer to another page.
Article is divided into one or more new pages, or part of the article is moved
to another page.
14-03-04

21. Research Methodology
— Qualitative Analysis
¡ Open coding process to classify rationales used in deletion
debates on three selected dates
— Quantitative Analysis
¡ Chi Square Tests
¡ Relationships among articles’ topics and deliberation
outcomes, discussion situations
¡ Relationship between the SOPA act event and the deliberation
outcome
SOPA (Stop Online Piracy Act) act event: On January 18, 2012, the English
Wikipedia, Google, and an estimate of 7,000 other smaller websites coordinated a
service blackout, to raise awareness.

22. Sample
— Date selection for qualitative analysis
Day
# of
articles
1 Jun. 2010
89
1 Jun. 2011
73
15 Jan. 2012
67
Total
votes
for
“keep”
127
Total
votes for
“delete”
280
Total
“other” votes
(merge,
userfy, etc)
37
119
212
23
109
200
63
— Date selection for quantitative analysis
¡ Previous sample
¡ 20 dates for chi-square tests that require larger sample size (a priori power
analysis)
÷
÷
Before the SOPA act event: Jan. 1 - 10, 2012 and Nov. 1 - 10, 2011 (N = 1453)
After the SOPA act event: Jan. 20 - 29, 2012 and March 20- 29, 2012 (N = 1202)

23. Findings - Types of rationales; factors of
deliberation outcome
23
— Rationales are mainly about the articles’ notability (50%)
and credibility (12%); Wikipedia policies are often
referred to as well (10%)
— Relationship between the deliberation outcome
÷ and
the type of votes: in the case that the decision is delete, there
tend to have more delete votes than keep votes, whereas in the case
that the decision is keep, the delete votes are not more than keep
votes; the votes other than keep and delete significantly affect those
decisions that would change the articles’ status.
÷ and the topic of article : articles about people, for-profit
organizations, and definitions are slightly more likely to be deleted
than expected; articles about locations or events are more likely to be
kept than expected; and articles about non-profit organizations and
media are more likely to be suggested for other options (e.g., merge,
redirect, etc) than expected
(Xiao & Askin, JASIST, to appear)
14-03-04

24. Findings - Types of rationales; factors of
deliberation outcome
24
— Relationship between the discussion situation
¡ and the type of rationale: more agrees in non-unanimous
situations
¡ and the deliberation outcome: in non-unanimous
situations, it is more likely to have final decisions as keep or
other solutions (e.g., merge, redirect, etc)
¡ and the community participation:
÷ more
unique Wikipedia IDs in non-unanimous discussions
÷ More participants in a non-unanimous discussion; the most
involved participant was more likely to be recognized in the
discussion
(Xiao & Askin, JASIST, to appear)
14-03-04

25. Findings – the Impact of SOPA blackout event on
the Deliberation
25
Before the blackout of the site in response to the proposed
Stop Online Piracy Act (SOPA) law, there were slightly less
keep cases than expected and after the event there were
slightly more keep cases than expected.
The effect was more significant on the decisions which
would change the articles’ status. These articles were more
likely to be deleted before the Act, whereas after that Act
(or during the discussions about it) they were more likely
to be offered suggestions for other options
14-03-04

26. Shared Rationales in Large Online Deliberation
26
Wikipedia’s Article for Deletion (AfD) discussions
Step 1 : Types of rationales; factors of deliberation outcome
Step 2: Computational linguistic approaches to extract the rationales
14-03-04

27. Rationale Extractions for Knowledge Management
27
— Motivation – to benefit new editors
— Approach - extraction of rationales that reflect the
needed knowledge on Wikipedia policies in AfD
discussions
— Technique –Illocutionary Act (Searle, 1976)
¡ Representatives
¡ Directives
¡ Commissives
¡ Expressives
¡ Declarations
14-03-04

28. Rationale Extractions for Knowledge Management
28
— Directives (Ervin-Tripp, 1976)
¡
¡
¡
¡
¡
¡
Need statements
Imperatives
Imbedded imperatives
Permission directives
Question directives
Hints
14-03-04

29. Detect Imperatives
29
1. A verb (in its base form) as the root in the phrase structure and this
particular verb has no subject child in the dependency structure.
(ROOT (S (INTJ (VB please)) (VP (VB refrain) (PP (IN from) (S (VP (VBG making)
(NP (JJ personal) (NNS attacks)))))) (. .)))
2. A personal pronoun or noun (e.g., you,
they, username) followed by a modal verb
(e.g., should, must)
"You must discuss the matter there, and you need
to be specific”
14-03-04

30. Rationale Extractions for Decision-Making Support
30
— Motivation – to facilitate efficient final decisionmaking
— Approach – elimination of redundancy by identifying
representative rationales in the discussion
— Technique – text similarity and sentiment analysis
14-03-04

31. Rationale Extractions for Decision-Making Support
Discussion
group similar rationales
classify by sentiment
polarity
Group A
positive
neutral
Group B
negative
positive
neutral
Select representative rationales –
similarity score, number of users,
policies
negative

32. Text Similarity
• SEMILAR, a semantic similarity toolkit, was used to
compute text similarity
— Compared the performance of similarity measure
among algorithms and human evaluation:
Weighted Latent Semantic Analysis (LSA)
¡ Latent Dirichlet Allocation (LDA)
¡

33. Sentiment Analysis
— Determine the sentiment polarity of a rationale in
our language context (“notable”)
¡ MPQA Subjectivity Lexicon + additional words
Data Input
Stanford
Parser
Part-of-speech
tagged text
Dependency
relations
Check modified
MPQA subjectivity
lexicon to obtain
the prior polarity
(if not in MPQA,
marked as ‘nonsentiment’)
MPQA format:
type=strongsubj len=1 word1=aberration pos1=adj stemmed1=n priorpolarity=negative

34. Sentiment Analysis
• Local negation: A not usually modifies the sentiment
word.
–
“The place is not notable.”
• Predicate negation: using verbs with negative
polarity.
–
“I disagree that the place is notable.”
• Subject negation: a subject leads to the negation of
its predicate.
–
“Neither one of us agrees that the place is notable.”

35. Sentiment Analysis
— Preposition negation: the polarity of the object
following the preposition “of” can be changed by the
word modified by the preposition.
¡
“It
is
a
viola&on
of
notability.”
— Modifier negation: some sentiment word’s polarity
can be negated by its modifier.
¡
“The
place
is
of
indeterminable
notability.”

36. Sentiment Analysis
• Modifier negation
– Phrase in the following combination:
Noun modified by adjective
Noun modified by noun
Adjective modified by adverb
Adverb modified by adverb
Verb modified by verb

37. Sentiment Analysis
— Using machine learning methods to determine the
polarity of a phrase that has a modifier and a word
–
Features:
•
•
•
•
•
•
l
First word token
Second word token
First word polarity
Second word polarity
First word part-of-speech
Second word part-of-speech
Performance of Naïve Bayes, k-nearest neighbor (KNN) and
decision tree:
÷ Data:
961 instances (phrases)
÷ Evaluation:10 folds cross validation
Naïve Bayes
Accuracy (%)
K-nearest neighbor
Decision Tree
77.94
83.77
80.65

38. Sentiment Analysis
— Bottom-up (recursive) algorithm
¡
Based on dependency structure

39. Sentiment Analysis
— Evaluation
¡ Data:
236 sentences from discussions in AfD
¡ 3 classes: positive, negative, neutral
¡ Accuracy: 58.47%

40. The Role of Shared Rationales in Group Ideation
and Deliberation Activities
40
The effects of rationale awareness in
— small group ideation activities
— Large online crowdsourcing ideation activities
The effects of shared rationales in
— Large online deliberation activities
Current research plan
To automatic detect rationales from online ideation activities and deliberation activities
14-03-04

41. Current Research Plan
Rhetorical Structure Theory has been recently used to identify justifications in the
social Web (Biran, and Rambow, 2011), where the existence of certain discourse
structures has been considered argument indicators.
Justification is defined as:
1.
2.
3.
4.
5.
Recommendation for action, and motivation for proposed action.
Statement of like or dislike or of desires and longing, and subjective reason for this like
or dislike or desire or longing
Statement of like or dislike or of desires and longing, and claimed objective reason for
this like or dislike or desire or longing
Statement of subjectively perceived fact, with a proposed objective explanation
A claimed general objective statement and a more specific objective statement that
justifies the more general one
Presentational relations from RST Treebank were primarily considered

42. The Effects of Rationale Awareness in Small
Group Ideation Activities
42
Discourse relations in shared rationales in the small group
ideation activities (Xiao, 2013):
Most used strategies in justifying one’s ideas in the
activities were: providing contextual information
(circumstance), additional information
(elaboration), and evaluation of the information
(evaluation)
We are extending Biran and Rambow’s (2011) approach by conducting further
analysis on these discourse relations and their potential connections to different
types of reasoning.
14-03-04

43. Acknowledgement
43
John M. Carroll, Mary Beth Rosson, Craig Ganoe,
Shawn Clark, Steven High, Anabel Quan-Haase, Yan
Luo, Tatiana Vashchilko, Trina Joyce Sajo, William
Klie, Becky Ellis, Mengshuo Chen, Nicole Askin, Jill
Kavanaugh, Lindsay Baker, Achchana Nadarajah,
Yumo Yin, Vadim Mazalov, Wanting Mao, Taraneh
Khazaei
Funding support: NSERC Discovery,
NSERC Engage, MITACS, SSHRC,
FIMS Internal Funding, SSHRC 4A
assistance
14-03-04