Colaboração

1. Gamification of Collaborative Idea
Generation and Convergence
Abstract
Collaborative brainstorming does not always result in
more ideas or higher quality ideas than working
individually. We designed a system with game elements
to incent participation in a collaborative creative idea
generation processes of brainstorming followed by a
convergence activity. We compared teams using the
system with and without game elements to investigate
the effect of the elements on collaborative work
activities. Preliminary results suggest that game
elements can help teams produce more ideas during
brainstorming and engage in more discussion during a
subsequent convergence activity, without negatively
affecting idea quality.
Author Keywords
Gamification; Brainstorming; Convergence;
Divergence; Collaboration
ACM Classification Keywords
H.5.3 [Information Interfaces and Presentation
(e.g.,HCI)]: Group and Organization Interfaces---
Computer-supported cooperative work;
Introduction
A common strategy for collaborative problem solving is
to brainstorm a number of ideas and then converge on
a subset through discussion, refinement and selection.
Permission to make digital or hard copies of part or all of
this work for personal or classroom use is granted without
fee provided that copies are not made or distributed for
profit or commercial advantage and that copies bear this
notice and the full citation on the first page. Copyrights for
third-party components of this work must be honored. For
all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
CHI 2014, Apr 26 - May 01 2014, Toronto, ON, Canada
ACM 978-1-4503-2474-8/14/04.
http://dx.doi.org/10.1145/2559206.2581253
Ali Moradian
IBM
36 York Mills Road Suite 200,
Toronto, ON, Canada. M2P 2E9
moradian@ca.ibm.com
Maaz Nasir
University of Toronto
40 St. George Street, Toronto,
Ontario, Canada. M5S 2E4
maaz.nasir@mail.utoronto.ca
Kelly Lyons
University of Toronto
140 St. George Street, Toronto,
Ontario, Canada. M5S 3G6
kelly.lyons@utoronto.ca
Rock Leung
SAP
910 Mainland Street Vancouver,
British Columbia, Canada. V6B 1A9
rock.leung@sap.com
Susan Elliott Sim
Many Roads Studios,
Toronto, Ontario, Canada.
M4N 1S6
ses@drsusansim.org
Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada
1459

2. Collaborative brainstorming involves generating new
ideas, typically without consideration of quality and
using divergent group thinking [10] . In contrast,
judgment, critique, and selection of ideas from a list
generated by brainstorming requires convergent group
thinking and is often accomplished by refining,
combining, and/or selecting ideas [10]. Past research
has found that brainstorming in groups does not always
produce more ideas or higher quality ideas than
working individually [10]. Phenomena such as free
riding have been identified as barriers to group idea
generation. Computer-based group systems have been
designed to help by enabling anonymity in idea
generation and displaying ideas as they are entered in
real time [10]. However, the impact of computer-based
systems is not uniformly positive [2].
We are interested in how game elements can help
support collaborative brainstorming and convergence.
Gamification is an approach to enhancing existing
systems by adding game-related elements such as
goals, feedback, achievements, and rankings, for the
purpose of steering users towards sought-after
behaviors [8]. The use of these elements leverage
research in goal setting and social psychology [7],[9].
Gamified systems need not look like traditional
computer games [4]. In fact, the progress bar in
LinkedIn’s profile, goal setting and progress tracking in
a gym, and creating competition among customers in
Foursquare are examples given of gamification [4].
We created a system to support collaborative
brainstorming and convergence in order to evaluate the
effects of added game elements. Preliminary results
suggest that game elements can have a positive impact
by increasing the quantity of ideas produced in
collaborative brainstorming and may increase the
amount of discussion during the convergence activity.
A Brainstorming and Convergence System
We designed a system to support the collaborative
brainstorming and convergence processes depicted in
Figure 1. The system enables small teams (5 or fewer)
working on a pre-defined problem to quickly produce a
set of ideas [1]. The system was designed to work
either with collocated individuals or individuals working
over distance. We developed our system on the public
development version of SAP’s StreamWork [5], using
OpenSocial API.
During the brainstorming activity, participants generate
ideas anonymously and synchronously in parallel.
Submitted ideas are displayed immediately to peers.
Our implementation of the convergence activity was
modeled after the FastFocus ThinkLet [1]. In this
activity, each team member receives a Segmented List
of Ideas and is asked to select one idea. After all team
members have done this, they must perform
Clarification and Reduction by discussing each of the
selected ideas in turn by posting comments, during
which the team must agree on a final phrasing of the
idea and decide whether to add it to the Final List of
Ideas. After each selected idea has been discussed, the
process iterates over receiving another Segmented List
of Ideas, followed by Clarification and Reduction.
Figures 2 and 3 show the user interface (UI) for the
baseline version of the brainstorming and convergence
tools respectively. In the gamified version, the game
elements shown in Figures 4-8 were added to a panel
on the right side of the UI. These game elements have
been used in past systems to increase motivation and
engagement in performing a task [4], [6], [8]. Other
Figure 1: Brainstorming and
Convergence system overview.
Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada
1460

3. than that everything else was the same between two
conditions (e.g. task description and questions).
Goal Setting
Giving specific high-challenge goals, compared to
simply letting people do their best, has been found to
lead to better performance [6], [9]. We selected goals
that encourage collaboration, are relatively difficult,
specific to the task, and can be achieved in a short
amount of time. For the brainstorming activity, each
team using the gamified version was given the ultimate
goal to generate 20 ideas but was presented with a
number of intermediate goals (10 ideas, 5 more, 3
more, 2 more, and bonus ideas; see Fig. 4). For the
convergence activity, gamification condition teams were
given four intermediate goals: generate five ideas in
the final list; all members in a team participate in the
discussion; complete the first round of discussion;
complete three rounds of discussion (see Fig. 6).
Progress Bar
Feedback that indicates progress towards a goal can
increase the usefulness of goals [9]. For the
brainstorming activity, we added a progress bar to
indicate the team’s progress toward each goal. We used
a power function (rather than a linear one) to indicate
faster progress as participants approach the final goal
[3]. For example, when teams generated 6 ideas
towards a goal of 10, the progress shown is 37.45%
(instead of 60%) but for 9/10, the progress 81% (see
Fig. 5). This characteristic was expected to provide
encouragement during brainstorming because idea
generation increases in difficulty over time [2].
Achievement Points and Leader Board
Awarding points is a common approach for providing
feedback and rewarding people for achieving task-
related goals [6], [9]. For the brainstorming activity,
we designed a leader board that displays the number of
ideas that each participant in the team has contributed
so far, thereby enabling participants to evaluate their
performance and compare it with others [7].
For the convergence activity, we created a scoring
system that awards points for both individual
achievements (e.g., selecting an idea for discussion,
participating in the discussion) and team achievements
(e.g., all participants in the team finish a round of
discussion). Participants can evaluate their relative
performance by reviewing the point history (see Fig. 8).
Method
In our study, teams of three collocated individuals
worked on the same collaborative problem-solving task
(brainstorming followed by convergence) using either
the baseline or gamified version of the system. The
teams were assigned randomly to condition-specific
rooms, thereby blinding them to the treatment.
We investigated two research questions: “How does
gamification affect productivity?” and “How does
gamification affect the quality of ideas produced?” To
answer the first question, we looked at the number of
ideas generated by each team during brainstorming,
and the number of ideas discussed, as well as the
number of discussion comments for each idea made by
each team during convergence. To answer the second
one, we examined the quality of ideas generated during
brainstorming and the quality of ideas selected during
convergence. We also conducted a post-activity survey
on participants’ experience.
Participants
The participants in the study were students in a
graduate-level project management course in an
Figure 2: The user interface for
the brainstorming activity
Figure 3: The user interface for
the convergence activity
Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada
1461

4. iSchool. The students had not worked together before
but were assigned to a team for the duration of the
term. Seven teams of three students each (total 21)
participated in the study: four teams used the baseline
system and three used the gamified version.
Procedure
Each student sat at a single computer. The participants
in each team sat near each other but were instructed to
only communicate using the system and not verbally.
All teams were asked to brainstorm on the topic:
“Given the iSchool mission and goals, identify project
ideas that you think will help iSchool achieve their
mission and goals.” Participants were provided a paper
copy of the mission and goals. For brainstorming,
participants were told: “You should think about project
ideas to support the iSchool mission and goals, as well
as focus on the quantity and variety of ideas. You will
have time to refine and reject ideas later. At this stage
all ideas are valuable.” For the convergence activity,
they were told: “You will go through several rounds of
reviewing and refining your brainstormed project ideas
through group discussions. The ultimate goal is to
select and agree on a final list of project ideas.”
All teams were given 75 minutes to complete in the
task. They were told to spend roughly 20 minutes in
the brainstorming activity and the rest of the time in
the convergence activity, but were able to decide as a
group when to switch to the convergence activity.
Data Analysis
We analyzed the data at the team level, which we
present below. In this study, individuals from different
teams did not interact with each other and since the
conditions were fixed, observations between teams are
independent. We used a one-way ANOVA to test the
differences in means and below we report mean (M),
standard deviation (SD) and F ratio.
We analyzed the quality of the brainstormed ideas and
final selected ideas by having one of the co-authors
rate all ideas on a scale from 0 (lowest) to 4 (highest).
The reliability of the rating scheme was validated by
having another co-author rate a random subset of the
data (~25% of ideas), both raters were blind to the
condition; substantial agreement was found (a linearly
weighted Kappa co-efficient = 0.63).
Results
Brainstorming
Although teams in the gamification and baseline
conditions spent similar amounts of time on the
brainstorming activity (M=16 min, SD=6.3 vs. M=15
min, SD=3.5, respectively), teams in the gamification
condition generated significantly more ideas than those
in the baseline condition (F1,5=7.1, p<.05). The
gamification condition teams produced on average 21.3
ideas per team (SD=4.0), while baseline condition
teams produced 13.8 ideas per team (SD=3.5). In fact,
two teams in the gamification condition exceeded the
goal of 20 ideas and none in the baseline condition did.
Teams from both conditions produced a similar number
of good ideas (rated 3 or 4 out of 4) (Gamification:
M=11.7, SD=5.5, 55% of all ideas; Baseline: M=9.0,
SD=3.4; 65% of all ideas).
Convergence
Teams in the gamification condition spent more time on
the convergence activity than those in the baseline
condition (M=50 min, SD=3.6 vs. M=38 min, SD=11.5,
respectively). Gamification condition teams on average
also engaged in more discussion than those in the
baseline condition in terms of average number of
Figure 4: Brainstorming tool
achievement list.
Figure 5: Brainstorming tool
progress bar.
Figure 6: Convergence tool
achievement list.
Figure 7: Convergence tool
leader board.
Figure 8: Convergence tool
history of the points.
Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada
1462

5. comments per idea (M=23.11, SD=7.9 vs. M=13.4,
SD=6.9, respectively), but this difference was not
statistically significant (F1,5 = 2.9, ns). Despite these
differences, teams in the gamification and baseline
conditions discussed a similar number of ideas (M=6.7,
SD=1.1vs. M=6, SD=2.9, respectively; F1,5=0.13, ns).
The gamification condition teams, compared to those in
the baseline condition, selected on average more ideas
(M=5.3, SD=1.2 vs. M=4.0, SD=1.8, respectively) and
produced more good ideas in the final list (M=4.0,
SD=2.0 vs. M=2.8, SD=1.7, respectively), but this
difference was not significant (F1,5=0.11, ns).
Participants’ experience
Comments from the post-activity survey suggest that
the added game elements may have positively affected
participants’ perceptions of the brainstorming and
convergence activities. Four people in the gamified
condition chose to answer the open-ended question and
all commented positively: “Loved this program … Really
useful for brainstorming, which I'm usually really bad
at.”; “This is an excellent tool!”; “I really like this online
system. Makes it much easier for me, as a person
traditionally more quiet and observant than outgoing in
group work, to toss ideas out with the buffer of internet
anonymity!”; and “This is a really great system. I feel
that I am usually a wallflower in group discussions. I
tend to be better at adding to ideas and I need more
time, however, the buffer with this system made it
really inviting to be in the group discussion, To be
honest, I was dreading doing this assignment, but it
was great!”.
Three people in the baseline condition answered the
open-ended question, only one making a similar
positive comment, “I thought the tool was extremely
useful for organizations who have project members
distributed over a large geographic area. I did miss the
creativity that can result from verbal/face-to-face
interaction but in situations where this is not possible I
think this tool is an extremely valuable resource…” The
other two comments were less positive, for example:
“The only problem I had with the activity had to do with
the initial brainstorming process. I feel more
comfortable coming up with ideas if there is a dialogue
exchange -- as there was for the other stages of the
activity. I find it more difficult to come up with ideas if I
don't have the ability to discuss my ideas and receive
feedback, suggestions and other from other group
members. I find that good ideas come out of dialogue.”
Discussion and Future Work
In summary, we found that our added game elements
increased idea generation in the brainstorming activity
and may also have increased the amount of discussion
and number of ideas selected during the convergence
activity, though the increases were not statistically
significant.
Comments from the gamification condition teams offer
some evidence that the added game elements helped
them participate in the activity. While participants did
not comment directly on their experience with the
game elements, they also did not state that any
elements made it more difficult to participate (e.g.,
competitive pressure), which was a possibility. We also
noted a common theme in all comments was that
participants did not think they would do well in the
activity. However, participants in the gamification
condition teams found their version of the system to be
“really useful”, “much easier for me”, and “really
inviting.” Perhaps the extra guidance from the game
Figure 9: Quality of ideas produced
during brainstorming by each team.
Good ideas (scores 3 or better) are
shaded dark.
Figure 10: Quality of ideas produced
after the convergence activity by
each team. Good ideas (scores 3 or
better) are shaded dark.
Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada
1463

6. elements made it easier to participate. Better
understanding of why game elements helped support
the collaborative activity warrants further investigation.
In general, our work suggests there may be
opportunities for using game elements to support
brainstorming and convergence activities, and further
investigation is needed. First, we plan to run our study
with more teams to increase the statistical power of our
study and confirm if trends observed in our results are
significant. Further studies should also look at the effect
of our added game elements to smaller or larger teams
as the effort involved in collaborative brainstorming and
convergence may change with different team sizes. In
addition, the participants in our study were graduate
students who had not previously worked together. It
would be interesting to repeat the experiment in a
professional work setting with individuals who work
together on a regular basis. Finally, we only looked at
the combined effect of several gamification elements,
and more work is needed to understand the effect of
individual game elements on creative idea generation
processes. Specific elements may have a greater
influence than others over the outcome of the activity.
It may be also possible to fine-tune the design of each
game element for effectiveness in particular situations.
Acknowledgements
We thank anonymous reviewers for feedback, Dr. F.
Camacho (Damos) for help with statistical analysis, and
Dr. S. Szigeti for help with data collection. This work
was supported by an NSERC CRD Grant with SAP
Canada.
References
[1] Briggs, R.O., De Vreede, G.J., Nunamaker, J.F., &
Tobey, D. ThinkLets: Achieving predictable, repeatable
patterns of group interaction with group support
systems (GSS). In Proc. of HICSS 2001, 9 pages.
[2] Ferreira, A., Antunes, P., and Herskovic, V.
Improving group attention: An experiment with
synchronous brainstorming. Group Decision and
Negotiation 20, 5 (2011), 643–666.
[3] Harrison, C., Amento, B., Kuznetsov, S., and Bell,
R. Rethinking the progress bar. In Proc. of UIST 2007,
115–118.
[4] Huotari, K. and Hamari, J. Gamification from the
perspective of service marketing. In Proc. of CHI 2011
Workshop Gamification, (2011).
[5] Kanaracus, C. SAP’s ‘Virtual War Room’ Tool Gets a
Name: StreamWork | PCWorld. PCWorld, 2010.
http://www.pcworld.com/article/192903/article.html.
[6] Li, W., Grossman, T., and Fitzmaurice, G.
GamiCAD: a gamified tutorial system for first time
autocad users. In Proc. of UIST 2012, 103–112.
[7] Ling, K., Beenen, G., Ludford, P., Wang, X., Chang,
K., Li, X., Cosley, D., Frankowski, D., Terveen, L.,
Rashid, A.M., Resnick, P., and Kraut, R. Using social
psychology to motivate contributions to online
communities. Journal of Computer-Mediated
Communication. 10, 4 (2005).
[8] Liu, Y., Alexandrova, T., and Nakajima, T.
Gamifying intelligent environments. In Proceedings of
the 2011 international ACM workshop on Ubiquitous
meta user interfaces, (2011), 7–12.
[9] Locke, E. A. and Latham, G. P. Building a
Practically Useful Theory of Goal Setting and Task
Motivation: A 35-Year Odyssey. American Psychologist
57, 9, (2002), 705-717.
[10] Rietzschel, E.F.,Nijstad, B.A., and Stroebe, W.
Productivity is not enough: A comparison of interactive
and nominal brainstorming groups on idea generation
and selection. Journal of Experimental Social
Psychology 42, 2 (2006), 244–251.
Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada
1464