This document reviews the intersection of gamification and crowdsourcing, highlighting how gamified elements like points and leaderboards are used to enhance user engagement and quality of output. Empirical studies suggest that gamification positively affects crowdsourcing participation and task performance, with various implementations suited to different types of crowdsourcing. It emphasizes the need for further research to explore the psychological and contextual impacts of gamification in crowdsourcing scenarios.

1. KIT – University of the State of Baden-Württemberg and
National Research Center of the Helmholtz Association
KARLSRUHE SERVICE RESEARCH INSTITUTE (KSRI)
www.kit.edu
www.ksri.kit.edu
www.kit.edu
Gamification in Crowdsourcing
A review
HICSS 49 Kauai 2016
Benedikt Morschheuser, Juho Hamari, Jonna Koivisto
Photo: CC Roberto Taddeo

2. The authors
2
Benedikt Morschheuser
Juho Hamari (D.Sc.Econ)
Jonna Koivisto

3. Crowdsourcing & Gamification
Two booming research trends
3
Crowdsourcing: “the act of a company or institution taking a function once performed by
employees and outsourcing it to an undefined (and generally large) network of people in
the form of an open call.” Howe, J. (2006)
Gamification: “a process of enhancing services with (motivational) affordances in
order to invoke gameful experiences and further behavioral outcomes.” Hamari et al. (2014)

4. How crowdsources are normally motivated?
4
$

5. Gamification and crowdsourcing
two interwoven phenomena
5
Gamified
crowdsourcing
system with avatars,
points, levels and
leaderboards
30 mil users 2012
Google bought
Waze for $1.15
billion in 2013

6. However, we miss a coherent overview
6
How is gamification used in crowdsourcing?
Does gamification work?
What kind of gamification fits what kind of crowdsourcing?

7. The review
 Scopus Database (incl. AIS,
ACM, IEEE, Sciencedirect ...)
 Search for „GAMIF* AND
CROWD*“ in abstract, title and
keywords
 Selection criteria
 full paper available
 paper is in English and published in
an int. venue
 paper is about gamification and
crowdsourcing
 not a duplicate
 the paper contains empirically
derived results
 50 hits
 28 match the criteria
7KSRI - August 2013

8. Frameworks used
8
Geiger & Schader 2014
Behavioral
outcomes
Psychological
outcomes
Motivational
affordances
Huotari & Hamari 2012
Hamari et al. 2014
Points
Level
Badges
Leaderboards
…
Motivation
Enjoyment
Flow
Attitude
...
Use
Interaction
Learning
Contribution
...

9. Gamified crowdsourcing

10. How is gamification used in crowdsourcing?
10
points / score
28%
leaderboards /
rankings
26%
badges / achievements
13%
levels
9%
progress
6%
missions
2%
rewards
4%
feedback
5%
storytelling
4%
virtual territories
3%
N = 28

11. 11
gamification
75% gamification +
financial rewards
25%
Incentive orchestration
N = 28

12. 0
1
2
3
4
5
6
7
8
9
Compared a non-gamified with a
gamified approach
No comparison (Interviews, user
feedback, etc.)
Compared between different
gamification designs
Does gamification work?
12
N = 17
Empirical studies on the effectiveness of gamification

13. Does gamification work?
13
 All empirical studies on the effectiveness of gamification in CS report
that gamification has a positive impact on crowdsourcing work
(N=17).
 Empirical studies which compare a gamified with a non-gamified
approach report the increase of engagement, quality of the output
or less cheating compared to traditional paid crowdsourcing (N=8).
 The studies highlight the importance of how gamification is
implemented. Different gamification affordances have different
effect (N=4).

14. How does gamification work?
14
Behavioral outcomes
Psychological
outcomes
Motivational
affordances
Points
Leaderboards / Rankings
Badges / Achievements
Levels
Progress
Feedback
Rewards
Storytelling
Missions
Virtual territories
Motivation
Attitude
Fun / Enjoyment
Participation engagement
(quantitative /
qualitative)
Quality of output
Cheating behavior

15. Differences between various types of crowdsourcing
15
0
3
6
9
Crowdrating
Crowdprocessing
Crowdsolving
Crowdcreating

16. Differences between various types of crowdsourcing
16
Less studied so far, manifold forms of
gamification seems to be suitable.
Promoting
cooperation rather
than competition
could potentially
be beneficial.
Manifold forms of
gamification
including missions,
storytelling, virtual
territories.
Scoring mechanisms are contextual and
depend on the possibilities to
measure task fulfillment / quality.

17.  Gamification seems to work
 Gamification can take a variety of forms in crowdsourcing. However, points (in
22 cases) and leaderboards (in 20 cases) were clearly the most implemented
gamification affordances
 Differences exist how gamification is implemented across different
crowdsourcing types
 Crowdsourcing initiatives with homogenous and monotonous tasks commonly
use points and other simple gamification implementations
 Crowdsourcing initiatives that seek diverse and creative contributions employ
gamification in more manifold ways with a richer set of affordances
17
Summary

18. Future research
 The topic is still very much in its infancy
 Future research should try to avoid currently present
methodological gaps
 Use psychological measurements and analyze what people feel
using gamification
 Conduct full experiments with control groups
 Try to control the effects of single affordances
 Identified research gaps
 Can gamification promote cooperative behavior?
 Examine crowdcreation approaches
 Context factors and personal/crowd differences
18

19. Thanks
For more research on that topic:
Benedikt Morschheuser issd.iism.kit.edu
Juho Hamari juhohamari.com
Jonna Koivisto jonnakoivisto.com
Or follow us on
Researchgate
19