A brief review of some recent research projects on supporting group ideation and deliberation activities

1. Lu Xiao
Assistant Professor
The University of
Western Ontario
2013
Fostering reflection in
About me
small group ideation
activities
online crowdsourcing
activities
small group deliberation
large-scale deliberation
18/09/2013 1

2. Chemical Engineering
Information Sciences and
Technology
Computer Engineering
Analytical
Problem solving
Statistics
OOP
Algorithm
Theory of Computation
User-Centered Design
Participatory Design
User Experience Research
Project Management
 Name: Lu 露 (solar term: 白露, a sign of Autumn)
18/09/2013 2

4. 18/09/2013 4

5.  Reflection (Schön, 1985; Boud et al, 1985;
Van Manen, 1977)
 Design to support reflection (Baker &
Land, 1997; Kriplean et al., 2012)
 Reflection in group setting (Yukawa, 2006;
Mamykina et al., 2008;
Gagnièrea, Betrancourta, Détienneb, 2012)
18/09/2013 5

6. 18/09/2013 6
Articulating and sharing rationales
small group ideation activities
Articulating and sharing rationales in ideation
activities in online crowdsourcing
Promoting awareness of group information
processing in small group deliberation
Extracting and visualizing important information
large-scale deliberation

7. Group Project: How to manage a distributed team
project for each project phase?
Execution ClosureControllingPlanningInitiation
Five Phases of a Project
Each project
phase
Identify
Challenges
Evaluate
Collaborative
Tools
Suggest
Best
Practices
Mini-report
7
Articulating and sharing rationales
small group ideation activities

8.  Virtual Teamwork (two days)
 Each group of students needs to answer two
questions:
◦ Brainstorming Challenges: What are the
challenges for the success of distributed
teamwork at this phase?
◦ Evaluating Challenges: What are the three most
and three least important challenges for the
success of distributed teamwork at this phase?
Document Rationale:
Why is it a challenge?
Document Rationale:
Why those challenges?
18/09/2013 8

9. Execution phase:
Challenge
Update project management plan
Rationale Statement
The execution phase relies critically on good
management. Therefore it is wise to update the
project management plan during the execution
phase in order to compensate for any unforeseen
discrepancies with the scope, or interpersonal
relationships with project team members. This can
be challenging because it is necessary to assess all
angles of the project's functionality.
18/09/2013 9

10. Execution ClosureControllingPlanningInitiation
MR MRMRMRMR
Virtual
Teamwork
(2 days)
One and half week
Final Presentation and Report
(One and half week)
Project Duration: nine weeks
: Challenges assessment activity MR : The rest course activities for the phase
18/09/2013 10

11. 11
Shared Data Spreadsheet
for Posting Ranking
Decisions
User
List
Group
Folders
Group
Chat Shared
Rationale
Space
18/09/2013

12. 12
 Survey
◦ Background survey
◦ Rationale Awareness survey
◦ Collaboration experience survey (includes
feedback on design of the shared rationale space)
 Interview
◦ Semi-structured interviews
 Three selected groups
◦ Iterative coding process (open coding)
 54 codes, 712 quotations
 Artifacts
◦ Shared rationale space
◦ Rationale statements (729 statements)
 Two independent evaluators (Inter-coder
agreement: Scott’s Pi = 0.68)
18/09/2013

13.  The evaluation of the workspace design
(Xiao, 2008)
 The students’ learning experiences (Xiao et
al., 2008)
 The impact of rationale awareness in the
group activities
◦ Knowledge awareness and contribution
awareness (Xiao, 2011a, 2011b)
◦ Practices related to articulating and sharing
rationales (Xiao, 2011c)
◦ Development of individual reflection skill (Xiao &
Carroll, 2013)
18/09/2013 13

14. 18/09/2013 14

15.  Rhetorical Structure Theory (RST) –
pragmatics in the language
RST: a theory of text organization created in the
1980s (Mann & Thompson, 1988)
18/09/2013 15
Probably the most extreme case of Visitors Fever I have ever
witnessed was a few summers ago when I visited relatives in the
Midwest.
Probably the most extreme
case of Visitors Fever I have
ever witnessed was a few
summers ago
when I visited relatives in
the Midwest.
relation

16.  Rhetorical Structure Theory (RST) http://www.sfu.ca/rst/
◦ Text units as nucleus and satellite
◦ Three categories of relations (based on writer’s
intention):
 subject matter relations: to convey the relation
 presentational relations: to convince the reader
 multinuclear relations: do not carry a definite
selection of one nucleus
18/09/2013 16
Probably the most extreme
case of Visitors Fever I have
ever witnessed was a few
summers ago --- (nucleus)
when I visited relatives in
the Midwest. – (satellite)
relation
circumstance

17. 18/09/2013 17http://www.sfu.ca/rst/
Tempting as it may be, we shouldn't embrace every popular
issue that comes along.
relation
concession
Tempting as it
may be, --satellite
we shouldn't embrace
every popular
issue that comes
along. --nucleus
Animals heal, but trees compartmentalize.
relation
contrast
An example of presentational relation
Animals heal, but trees compartmentalize.
An example of multinuclear relation

18. 18/09/2013
18
analyzed twice
Explained
RST to
Research
Assistant A
Three sets
of analysis
results

19. 18/09/201319
Three sets of
analysis results
examined by
The principle
investigator
Identified segments
of all the statements

20. 18/09/2013
20
Research
Assistant A
Identified
segments
of all the
statements
Reviewed by Analyzed by
Research
Assistant A
Final result

21. 18/09/2013 21
Team
Presentational RST
Relations
Subject Matter
RST Relations
Multi-nuclear RST
Relations
1 4.6% 87.7% 7.7%
2 3.7% 88.9% 7.4%
3 3% 85.6% 11.4%
Table 1. Percentage of Different Types of RST Relations
Team No. of RST relations not
appeared in the analysis
RST relations that were not used
in all three team
1 10 Evidence, Justify, Motivation,
Restatement, Unconditional,
Disjunction, Multinuclear
Restatement (7 in total)
2 8
3 11
Table 2. RST relations that did NOT occur in the Rationales

22. 18/09/2013 22
Team
RST Relations and their percentages in the coded RST
Relations
1 Circumstance(20%) Evaluation (17%)
Elaboration
(14.4%)
2 Elaboration (21.8%) Evaluation (18%)
Circumstance
(14%)
3 Evaluation (16.4%) Elaboration (13.7%)
Circumstance
(13.5%)
Table 3. Three Most Occurred RST Relations and Their Percentages

23. 18/09/2013 23

24. 18/09/2013 24

25.  The team members shared similar reasoning
styles across teams, and
 Team seems to be an influencing factor on the
individuals’ strategy of using additional
information to support one’s main point
Limitation of the study
 Exploratory study
18/09/2013 25

26. Research Question:
In a brainstorming task performed through
human computation processes, whether and how
knowing the others’ rationales of their ideas
affects the idea quality of the task?
18/09/2013 26
Articulating and sharing rationales in
ideation activities in online
crowdsourcing
Project II

27.  Classification of human computation
processes (Quinn and Bederson,2009; Malone et
al., 2009; Little et al., 2010)
 Quality measure of different approaches
(Little et al., 2010; Dow et al., 2012)
Project II

28.  Hypotheses (Kolodner and Schank, 1987;
Clark, 1986; Endsley, 1995; Xiao, 2011)
1. Making previous workers’ rationales of their
ideas available to the current worker can
improve the quality of the iterative approach
2. Making this information available to all of its
evaluators reduce the variation between
evaluations by multiple raters
Project II

29. Project II

30. • Open source software: Java/JavaScript API
for running iterative tasks on Mechanical
Turk.
Project II

31. Brainstorming/Idea
Generation Task
•Six company descriptions
•Five names for a company
description in each iteration
•Six iterations for each company
Rating/Idea
Evaluation Task
Each name had 10 ratings
Project II

32.  Three generation conditions: Parallel, Iterative
with and without showing previous workers’
rationales
 The research validity of submitted HITs for a
company was checked at each iteration
 Brainstorming tasks were posted to Mechanical
Turk through several sessions: in one
session, each generation condition had two
companies; and the condition for a company
followed Latin square order
Project II

33. Rating condition: Idea’s rationale not shown

34. Company
No.
Brainstorming Condition
Parallel
approach
Showing
previously
generated
names only
Showing
previously
generated
names AND
rationales
1 6.6 6.2 6
2 6.6 6.9 6.5
3 6.7 6.3 6.3
4 6.6 6.2 6
5 6.8 6.3 6.5
6 6.3 6.9 6.1
Project II

35.  There is a statistically significant difference
between the ratings: rationale awareness in
the rating task improved the name’s rating.
 There is a statistically significant difference
between the standard deviations of ratings
(2.58 vs. 2.41): Reduced the variations of the
ratings
Project II

36.  In an idea generation task, the awareness of
previous workers’ rationales may improve the
average quality but NOT the best quality of the
generated ideas in iterative approach
 In an idea evaluation task, the awareness of an
idea’s rationale can affect the evaluation
outcome and the quality of the rationale may
play a significant role on the evaluation
(Xiao, CSCW, 2012; Xiao, CI, 2012)

37.  Deliberation concept
◦ Habermas (1989) – public sphere
◦ Halpen and Gibbs (2013)
18/09/2013 37
a communication process that involves at least two individuals; that
focuses on a social or political issue where the solutions are
identifiable by participants; and that values equality among
participation and emphasizes rational thinking and logic instead of a
power struggle.

38.  Information Pooling Phenomenon (Stasser &
Titus, 1985)
18/09/2013 38
The unique information which needs to be shared and discussed
most remained unique (i.e., not shared) or unfamiliar to the group
(i.e., shared but did not receive enough attention during the
discussion).
Lu, L., Yuan, Y. C. and McLeod, P. L. Twenty-Five Years of Hidden Profiles in Group
Decision Making. Personality and Social Psychology Review, 16, 1 (February 1, 2012
2012), 54-75.

39.  Assumption: with improved awareness of group
information processing, it will help
◦ promote reflection on deliberation process and
outcome
◦ help identify hidden profiles in the decision-making
process
18/09/2013 39
Project III
Promoting awareness of group information
processing in small group deliberation

40. ◦ Version I: Message Visualization in IM (Xiao, 2012;
Xiao & Mazalov, 2012)
18/09/2013 40
Project III
Promoting awareness of group information
processing in small group deliberation

41.  Usability study (Xiao & Haase, 2012)
◦ One factor: tool condition
 Gtalk (no highlighting in the message)
 Jitsi
 Jitsi with visualization
 Two rooms for the experiment
 Tangram task
◦ Director
◦ Matcher
 Usability survey, interview, task performance,
chat log
18/09/2013 41
Promoting awareness of group information
processing in small group deliberation
Project III

42. 18/09/2013 42
 “strongly disagree” (-2), “disagree” (-1), “agree”
(1), “strongly agree” (2).
Promoting awareness of group information
processing in small group deliberation
Project III

43.  Cognitive Task vs. Experiment Task
◦ Describing -- director
◦ Clarifying -- director
◦ Interpreting -- matcher
◦ Matching – matcher
 Correctness – whose credit/fault?
18/09/2013 43
Project III

44. ◦ Version II: Web-based Deliberation Tool
18/09/2013 44
Project III
Promoting awareness of group information
processing in small group deliberation

45. 18/09/2013 45
Project IV
Wikipedia’s Article for Deletion (AfD) discussions
Research Questions:
What are the types of rationales used in the deliberation?
Are there any relationships among the article’s topic, the
kinds of votes, the discussion situation (unanimous or
non-unanimous), and the final decision?
Extracting and visualizing important
information large-scale deliberation

46.  Qualitative Analysis
◦ Open coding process to classify rationales used in
deletion debates on three selected dates
 Quantitative Analysis
◦ Chi Square Tests
◦ Relationships among articles’ topics and
deliberation outcomes, discussion situations
◦ Relationship between the SOPA act event and the
deliberation outcome
Article’s Topics: biography, definition, event, for-profit, non-profit,
location, media, and other

47.  Date selection for qualitative analysis
Keep, 19
Delete, 6
0
Other, 1
0
01-Jun-10 (89 articles)
Keep, 2
0
Delete,
50
Other,
3
01-Jun-11 (73 articles)
Keep, 1
4
Delete,
31
Other,
22
15-Jan-12 (67 articles)

48.  Date selection for quantitative analysis
◦ Previous sample
◦ 20 dates for chi-square tests that require larger
sample size (a priori power analysis)
 Before the SOPA act event: Jan. 1st to Jan. 10th, 2012
and Nov. 1st to Nov. 10th, 2011 (N = 1453)
 After the SOPA act event: Jan. 20th, 2012 to Jan.
29th, 2012 and March 20th to March 29th, 2012 (N =
1202).
18/09/2013 48

49. Qualitative Analysis:
1 June 2010: 443 unique votes for 89 debates, 813 coded
rationales
Agree
6%
Disagree
1% Credibility
15%
Notability
54%
Policy
10%
Precedent
2%
Procedural
4%
Richness
4%
Utility
3%
None
1%
1 June
2010
1 June
2011
15 January
2012
Agree 50 34 29
Credibility 119 75 66
Disagree 5 5 4
None 7 6 12
Notability 436 314 305
Policy 79 71 56
Precedent 19 23 40
Procedural 36 50 46
Richness 33 43 93
Utility 29 15 11

50.  Quantitative Analysis
◦ Relationship between the type of votes and the
deliberation outcome
 In the case that the decision is delete, there tend to
have more delete votes than keep votes, whereas in
the case that the decision is keep, the delete votes are
not more than keep votes.
 The votes other than keep and delete significantly
affect those decisions that would change the articles’
status.
18/09/2013 50

51.  Quantitative Analysis
◦ Relationship between the articles’ topic and the
deliberation outcome
 In AfD, articles about people, for-profit
organizations, and definitions are slightly more likely
to be deleted than expected, while articles about
locations or events are more likely to be kept than
expected, and articles about non-profit organizations
and media are more likely to be suggested for other
options (e.g., merge, redirect, etc) than expected.
18/09/2013 51

52.  Quantitative Analysis
◦ Relationship between the articles’ topic and the
deliberation situation and outcome
18/09/2013 52
Article’s Topic More likely
outcome than
expected
People, for-profit
org., definition
delete
Location, events Keep
Not-for-profit,
media, definition
Other (merge,
redirect, etc)
Article’s Topic More likely discussion
situation than
expected
People, for-profit Unanimous
Not-for-profit,
location, event
Non-unanimous
Media, other
topics
Non-unanimous
(slightly more likely)
Definition No effect

53. Funding support: NSERC Discovery, NSERC
Engage, MITACS, SSHRC, FIMS Internal Funding, SSHRC
4A assistance
John M. Carroll, Mary Beth Rosson, Craig Ganoe,
Shawn Clark, Anabel Quan-Haase, Tatiana Vashchilko,
Trina Joyce Sajo, William Klie, Becky Ellis, Mengshuo Chen,
Nicole Askin, Jill Kavanaugh, Lindsay Baker,
Achchana Nadarajah, Yumo Yin, Vadim Mazalov
18/09/2013 53

54. Lu Xiao
Assistant Professor in:
 Faculty of Information & Media Studies
 Computer Science (cross appointment)
The University of Western Ontario
Director, Human-Computer Interaction
Lab: http://hci.fims.uwo.ca
Research and Teaching Interests
Usability Engineering
Participatory Design
Collaborative and Social Computing
Thank You
Lu Xiao
lxiao24@uwo.ca
the University of
Western Ontario
18/09/2013 54