HOW TO FAIL AT
#GAMIFICATION RESEARCH
Lennart Nacke, PhD
Associate Professor
The Games Institute and
Stratford School of Interaction Design and Business
@acagamic | slideshare.net/acagamic | linkedin.com/in/nacke/
*Thanks, Henrik!
OVERVIEW OF HOW TO FAIL
1. WHY FAILING IS NECESSARY
2. FAILING AT GAMIFICATION EXPERIMENTS
3. FAILING TO UNDERSTAND AND DO GAMIFICATION
4. FAILING AT WRITING GAMIFICATION PAPERS
5. SUMMARY
WHY FAILING IS
NECESSARY
WHAT DID GOOGLE LEARN?
• Keep expectations in check
• Be clear about your products purpose
• Do not launch before features are ready
• Offer value to customers
TWITTER DEVELOPED FROM A FAILED
PODCASTING APP
FROM ODEO TO TWITTER
• Only a few people interested in creating audio
• They did a design brainstorm to fix their idea of making
everyone a creator
• Writing is easier than creating audio
• Pivot to Twitter in 2006, a listening and writing tool in
140 characters
SLACK EMERGED FROM A FAILED
VIDEO GAME
GLITCH WAS BASED ON SOFTWARE
TOOLS THAT BECAME SLACK
• Slack used to be a game developer called Tiny Speck
• Glitch was a multiplayer game they developed
• Whimsical illustrated characters and cooperative,
nonviolent gameplay
• Of course that would never be successful ;)
• Slack used the tech Glitch used for synching
• Slackbot came from a pet rock in Glitch that explained
the game world
ABRAHAM WALD AND
SURVIVORSHIP BIAS
IN WHICH AREAS OF THE B-29
SHOULD WE ADD ARMOR?
FAILING AT
GAMIFICATION
EXPERIMENTS
DO WE FALL PRAY TO SURVIVORSHIP
BIAS IN GAMIFICATION RESEARCH?
POST HOC ERGO PROPTER HOC?
WE NEED AN OPEN SCIENCE
COLLABORATION FOR
GAMIFICATION
SHARE DATA AND REPLICATE STUDIES
273 EMPIRICAL GAMIFICATION
STUDIES
• What if data from all of these studies or even the review
database were shared among us?
• How to find emerging focus areas beyond
• Health
• Crowdsourcing
• Social networking
• The non-significant, non-empirical, non-successful
gamification approaches are not catalogued – we
cannot review them (Jonna, Apu?)
Koivisto, J., & Hamari, J. (2019). The rise of motivational information systems: A review of gamification
research. International Journal of Information Management, 45, 191-210.
MY OWN STORY OF FAILURE
• Published BrainHex in 2014, but the analysis was flawed
• Basically a failure, even though it became highly popular and
well-cited
• Revisited the data in 2015 with my research team
• First fixing attempt in 2018:
• Gustavo Fortes Tondello, Deltcho Valtchanov, Adrian Reetz, Rina R. Wehbe, Rita Orji & Lennart E. Nacke (2018):
Towards a Trait Model of Video Game Preferences, International Journal of Human–Computer Interaction, DOI:
10.1080/10447318.2018.1461765
• Next attempt in 2019:
• Just accepted into INTERACT 2019: Tondello et al.: “I don’t fit into
a single type”: A Trait Model and Scale of Game Playing Preference
FROM 7 NON-VALID TYPES TO 5 TRAITS
•7 types collapsed into:
•action orientation (Conqueror, Daredevil)
•aesthetic orientation (Socializer, Seeker)
•goal orientation (Mastermind, Achiever,
Survivor)
•Suggested additions:
•social orientation
•immersion orientation
FAILING TO
UNDERSTAND AND
DO GAMIFICATION
START WITH WHY
What?
How?
WHY?
Sinek’s Golden Circle
Apple changing the music industry
Apple catching up with Steam…
WHAT DOES GAMIFICATION
RESEARCH NEED?
• We keep doing studies on points, badges, and
leaderboards – there is so much more to build with
• Let’s not turn into Apple...
• Where are our conceptual models?
• What is the purpose of our research? How do we move
past user engagement?
• Gamification for wellbeing? Improving lives?
OUR GAMEFUL EXPERIENCE MODEL
• Gameful experience as fundamental state of gameplay
• Gameful systems defined by qualities of interventions
and environments that create gameful experiences
• Gameful design is the process of creating these systems
• Gamefulness is then: designer actions, system
characteristics, or the user’s psychological experience
Landers, R. N., Tondello, G. F., Kappen, D. L., Collmus, A. B., Mekler, E. D., & Nacke, L. E. (2018). Defining gameful experience as a
psychological state caused by gameplay: Replacing the term ‘Gamefulness’ with three distinct constructs. International Journal of
Human-Computer Studies.
IS THERE REALLY A DARK SIDE OF GAMIFICATION?
https://darkpatterns.org
WHY ARE WE NOT WORKING MORE
WITH INDUSTRY?
• There are great models out there done by gameful
designers that we can use and run studies on
• Example: Hexad (we did several studies)
• But also Gamified.UK’s 52 gamification mechanics and
elements
• Many more models out there that help us actually do
gameful designs beyond our simple implementations
• Let’s look at an example
Design for Skill-Building
Your users needs and goals change over time. Know
their journey and design with the journey in mind.
26Players Journey By Amy Jo Kim, see http://amyjokim.com/blog/2014/04/08/the-players-journey/
Your users learn your systems as a visitor
first, then a newcomer, a regular, and finally
they become experts or enthusiasts.
Build a core learning loop first as a simple
version of the regular user experience you
are creating.
Stage 1: Discovery
This stage is for visitors or
people who have not yet used
your service or product, here you
show your value proposition.
What do most important early
customers need to learn during
Discovery?
Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
Stage 2: Onboarding
This stage is for newcomers, who are
using the product or service but need
to learn and get value out of it quickly.
What are the most important skills to
develop / things to learn during
Onboarding?
Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
Stage 3: Habit-building
This is the hook or core loop in
your application that the user
keeps coming back to: a
pleasurable, repeatable activity.
What repeatable, pleasurable
activity will pull people back for
Habit-building?
Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
FOCUS POINT: The Learning Loop
How to create habit-building experiences
• Cues and triggers
• Internal, Situational, External, Engaged
• What’s the internal trigger/urge/need that
drives someone to seek out my product?
• Repeatable pleasurable activity
• Triggered by emotion
• Internal urge or need
• In games, we call this a core mechanic (a rule
in action)
• Skill-building feedback
• Provide feedback and help players improve
• Feedback loops to help people get better at
core activity
• Progress and investment
• Show a progress and investment path
• Allow users to customize, build, collect
Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
Stage 4: Mastery
In games, this is the late/elder game; only your
best customers experience this, building full
customization and community.
What powers, access, roles, or privileges can
they earn/unlock for achieving Mastery?
Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
How might we address…
Put
hurdles/enablers
here
Put fitting design
lens/category
here
Put desired change here
(, using )
to achieve ?
Thanks to Sebastian Deterding for the examples.
How might we address…
AVOID WORKER
ERRORS
Put fitting design
lens/category
here
WAREHOUSE SELECTION TASKS
(, using )
to achieve ?
http://www.funkydesignspaces.com/plex/
How might we address…
AVOID WORKER
ERRORS
Put fitting design
lens/category
here
WAREHOUSE SELECTION TASKS
(, using )
to achieve ?
http://getmentalnotes.com/
FAILING AT WRITING
GAMIFICATION
PAPERS
START WITH THE END IN MIND
• Write focused on the WHY
• Consider your contribution to the field
• Read Jonna’s reviews
• Situated your work in the gamification space and be
clear about how to expand the field
• Consider reporting long-term studies
• If you can, then consider doing a randomized controlled
trial or a replication study
SUMMARY
• Let’s embrace failure as scientists
• More replication of results
• Build a stronger field and stay accountable
• Rapid pivoting and better improvement similar to
industry
• Let’s get inspired by industry
Questions?
Get in touch
HCI Games Group, University of
Waterloo, Ontario, Canada
Web: www.hcigames.com
Twitter: @hcigamesgroup
@acagamic
Facebook: facebook.com/hcigames
E-Mail: lennart.nacke@acm.org
Phone: (+1) 519-888-4567 x38251
BUYGamesUserResearch(OxfordUniversityPress)

GAMIFIN 2019 Conference Keynote: How to fail at #gamification research

  • 1.
    HOW TO FAILAT #GAMIFICATION RESEARCH Lennart Nacke, PhD Associate Professor The Games Institute and Stratford School of Interaction Design and Business @acagamic | slideshare.net/acagamic | linkedin.com/in/nacke/ *Thanks, Henrik!
  • 2.
    OVERVIEW OF HOWTO FAIL 1. WHY FAILING IS NECESSARY 2. FAILING AT GAMIFICATION EXPERIMENTS 3. FAILING TO UNDERSTAND AND DO GAMIFICATION 4. FAILING AT WRITING GAMIFICATION PAPERS 5. SUMMARY
  • 3.
  • 5.
    WHAT DID GOOGLELEARN? • Keep expectations in check • Be clear about your products purpose • Do not launch before features are ready • Offer value to customers
  • 6.
    TWITTER DEVELOPED FROMA FAILED PODCASTING APP
  • 7.
    FROM ODEO TOTWITTER • Only a few people interested in creating audio • They did a design brainstorm to fix their idea of making everyone a creator • Writing is easier than creating audio • Pivot to Twitter in 2006, a listening and writing tool in 140 characters
  • 8.
    SLACK EMERGED FROMA FAILED VIDEO GAME
  • 9.
    GLITCH WAS BASEDON SOFTWARE TOOLS THAT BECAME SLACK • Slack used to be a game developer called Tiny Speck • Glitch was a multiplayer game they developed • Whimsical illustrated characters and cooperative, nonviolent gameplay • Of course that would never be successful ;) • Slack used the tech Glitch used for synching • Slackbot came from a pet rock in Glitch that explained the game world
  • 10.
  • 11.
    IN WHICH AREASOF THE B-29 SHOULD WE ADD ARMOR?
  • 12.
  • 13.
    DO WE FALLPRAY TO SURVIVORSHIP BIAS IN GAMIFICATION RESEARCH? POST HOC ERGO PROPTER HOC?
  • 14.
    WE NEED ANOPEN SCIENCE COLLABORATION FOR GAMIFICATION SHARE DATA AND REPLICATE STUDIES
  • 15.
    273 EMPIRICAL GAMIFICATION STUDIES •What if data from all of these studies or even the review database were shared among us? • How to find emerging focus areas beyond • Health • Crowdsourcing • Social networking • The non-significant, non-empirical, non-successful gamification approaches are not catalogued – we cannot review them (Jonna, Apu?) Koivisto, J., & Hamari, J. (2019). The rise of motivational information systems: A review of gamification research. International Journal of Information Management, 45, 191-210.
  • 16.
    MY OWN STORYOF FAILURE • Published BrainHex in 2014, but the analysis was flawed • Basically a failure, even though it became highly popular and well-cited • Revisited the data in 2015 with my research team • First fixing attempt in 2018: • Gustavo Fortes Tondello, Deltcho Valtchanov, Adrian Reetz, Rina R. Wehbe, Rita Orji & Lennart E. Nacke (2018): Towards a Trait Model of Video Game Preferences, International Journal of Human–Computer Interaction, DOI: 10.1080/10447318.2018.1461765 • Next attempt in 2019: • Just accepted into INTERACT 2019: Tondello et al.: “I don’t fit into a single type”: A Trait Model and Scale of Game Playing Preference
  • 17.
    FROM 7 NON-VALIDTYPES TO 5 TRAITS •7 types collapsed into: •action orientation (Conqueror, Daredevil) •aesthetic orientation (Socializer, Seeker) •goal orientation (Mastermind, Achiever, Survivor) •Suggested additions: •social orientation •immersion orientation
  • 18.
  • 19.
  • 20.
    Apple changing themusic industry
  • 21.
    Apple catching upwith Steam…
  • 22.
    WHAT DOES GAMIFICATION RESEARCHNEED? • We keep doing studies on points, badges, and leaderboards – there is so much more to build with • Let’s not turn into Apple... • Where are our conceptual models? • What is the purpose of our research? How do we move past user engagement? • Gamification for wellbeing? Improving lives?
  • 23.
    OUR GAMEFUL EXPERIENCEMODEL • Gameful experience as fundamental state of gameplay • Gameful systems defined by qualities of interventions and environments that create gameful experiences • Gameful design is the process of creating these systems • Gamefulness is then: designer actions, system characteristics, or the user’s psychological experience Landers, R. N., Tondello, G. F., Kappen, D. L., Collmus, A. B., Mekler, E. D., & Nacke, L. E. (2018). Defining gameful experience as a psychological state caused by gameplay: Replacing the term ‘Gamefulness’ with three distinct constructs. International Journal of Human-Computer Studies.
  • 24.
    IS THERE REALLYA DARK SIDE OF GAMIFICATION? https://darkpatterns.org
  • 25.
    WHY ARE WENOT WORKING MORE WITH INDUSTRY? • There are great models out there done by gameful designers that we can use and run studies on • Example: Hexad (we did several studies) • But also Gamified.UK’s 52 gamification mechanics and elements • Many more models out there that help us actually do gameful designs beyond our simple implementations • Let’s look at an example
  • 26.
    Design for Skill-Building Yourusers needs and goals change over time. Know their journey and design with the journey in mind. 26Players Journey By Amy Jo Kim, see http://amyjokim.com/blog/2014/04/08/the-players-journey/ Your users learn your systems as a visitor first, then a newcomer, a regular, and finally they become experts or enthusiasts. Build a core learning loop first as a simple version of the regular user experience you are creating.
  • 27.
    Stage 1: Discovery Thisstage is for visitors or people who have not yet used your service or product, here you show your value proposition. What do most important early customers need to learn during Discovery? Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
  • 28.
    Stage 2: Onboarding Thisstage is for newcomers, who are using the product or service but need to learn and get value out of it quickly. What are the most important skills to develop / things to learn during Onboarding? Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
  • 29.
    Stage 3: Habit-building Thisis the hook or core loop in your application that the user keeps coming back to: a pleasurable, repeatable activity. What repeatable, pleasurable activity will pull people back for Habit-building? Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
  • 30.
    FOCUS POINT: TheLearning Loop How to create habit-building experiences • Cues and triggers • Internal, Situational, External, Engaged • What’s the internal trigger/urge/need that drives someone to seek out my product? • Repeatable pleasurable activity • Triggered by emotion • Internal urge or need • In games, we call this a core mechanic (a rule in action) • Skill-building feedback • Provide feedback and help players improve • Feedback loops to help people get better at core activity • Progress and investment • Show a progress and investment path • Allow users to customize, build, collect Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
  • 31.
    Stage 4: Mastery Ingames, this is the late/elder game; only your best customers experience this, building full customization and community. What powers, access, roles, or privileges can they earn/unlock for achieving Mastery? Amy Jo Kim. Game Thinking Explained. https://medium.com/@amyjokim/game-thinking-explained-fa6da3e8debb
  • 32.
    How might weaddress… Put hurdles/enablers here Put fitting design lens/category here Put desired change here (, using ) to achieve ? Thanks to Sebastian Deterding for the examples.
  • 33.
    How might weaddress… AVOID WORKER ERRORS Put fitting design lens/category here WAREHOUSE SELECTION TASKS (, using ) to achieve ? http://www.funkydesignspaces.com/plex/
  • 34.
    How might weaddress… AVOID WORKER ERRORS Put fitting design lens/category here WAREHOUSE SELECTION TASKS (, using ) to achieve ? http://getmentalnotes.com/
  • 35.
  • 36.
    START WITH THEEND IN MIND • Write focused on the WHY • Consider your contribution to the field • Read Jonna’s reviews • Situated your work in the gamification space and be clear about how to expand the field • Consider reporting long-term studies • If you can, then consider doing a randomized controlled trial or a replication study
  • 37.
    SUMMARY • Let’s embracefailure as scientists • More replication of results • Build a stronger field and stay accountable • Rapid pivoting and better improvement similar to industry • Let’s get inspired by industry
  • 38.
    Questions? Get in touch HCIGames Group, University of Waterloo, Ontario, Canada Web: www.hcigames.com Twitter: @hcigamesgroup @acagamic Facebook: facebook.com/hcigames E-Mail: lennart.nacke@acm.org Phone: (+1) 519-888-4567 x38251 BUYGamesUserResearch(OxfordUniversityPress)