TRACK 5(2) | MENTAL HEALTH PART1 | DAY 2 - 1 NOV 2016 Jim Lumsden, PhD Student | University of Bristol (UK) Games for Health Europe 2016

1. The State of Play
A systematic review of gamification in cognitive
assessment and cognitive training
Prepared for TRACK 5: Mental Health
Games for Health Europe, Utrecht, 2016
Jim Lumsden
jim.lumsden@bristol.ac.uk
School of Experimental Psychology,
University of Bristol,
BS8 1TU,
Bristol, UK

2. Abstract
Cognitive tasks are typically viewed as effortful, frustrating, and repetitive, which often leads to participant disengagement. This, in turn, may
negatively impact data quality and/or reduce intervention effects. However, gamification may provide a possible solution. If game design features can
be incorporated into cognitive tasks without undermining their scientific value, then data quality, intervention effects, and participant engagement
may be improved.
In this talk I will discuss the process and findings of our recent systematic review of gamified cognitive testing and training. We aimed to explore and
evaluate the ways in which gamification has been used for cognitive training and assessment purposes We hope to answer 2 main questions: Why
have researchers opted to use gamification and how successful has gamification been in cognitive research thus far?
Lumsden J, Edwards E, Lawrence NS, Coyle D, Munafò M. (2016)
Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy: JMIR
Serious Games

3. Cognitive Assessment and Cognitive Training
• Cognitive tasks are a staple tool in psychological research
• They are computer-based challenges that can be used to
either measure or train an aspect of cognition
• They are often designed to measure a single aspect of
brain function and are therefore simplistic in appearance
• Involve lots of repetition to filter out noise

4. The Problem with Cognitive Tasks
• Cognitive training often needs several sessions to show an effect
• Cognitive tests cannot produce valid measures if the participant is bored
• Cognitive tasks need to be engaging…
• And Gamification may be the answer

5. A Heterogeneous Literature
• Gamelike tasks have been part of psychological research since
1980s, but their appearances have always been sporadic
• Terminology is varied:
• We set out to bring the recent literature together
‘game’
(Boendermaker et al., 2015; Dunbar et al,. 2014; Hawkins et al., 2013; Dubbels et al,. 2013)

6. Search Strategy
• (gamif* OR game OR games)
• AND (cognit* OR engag* OR behavi* OR health* OR attention OR motiv*)
• Published between 2007 and 2015
Inclusion Criteria Exclusion Criteria
Primary research article Non-peer reviewed article
Novel gamelike task Game Engine/3D Environment only
Measure or train cognition
Validated or Piloted

7. Cohen’s K was calculated on 20% of the
articles from the title stage onwards
(7,590 articles checked).
There was moderate agreement
between the two reviewers
(k = .526, 95% CI, .416 to .633, p < .001).

8. Search Results
• 15 studies using gamified training tasks
• 17 studies using gamelike assessments
• 1 game that both assessed and trained
McPherson and Burns, 2007
McPherson and Burns, 2008
Trapp et al., 2008
Gamberini et al., 2009
Gamberini et al., 2010
Dovis et al., 2011
Delisle and Braun, 2011
Prins et al., , 2011
Lim et al., 2012
Heller et al., 2013
Hawkins et al., 2013
Verhaegh et al., 2013
Aalbers et al., 2013
Fagundo et al., 2013
Anguera et al., 2013
van der Oord et al., 2014
Brown et al., 2014
Tong and Chignell, 2014
Katz et al., 2014
Dunbar et al., 2013
Lee et al., 2013
Miranda and Palmer, 2013
Atkins et al., 2014
Dörrenbächer et al., 2014
McNab and Dolan, 2014
O’Toole and Dennis, 2014
Tenorio Delgado et al., 2014
De Vries et al., 2015
Dovis et al., 2015
Kim et al., 2015
Manera et al., 2015
Ninaus et al,. 2015
Tarnanas et al., 2015

10. Reasons given for using a gamelike task # of Studies
To increase participant motivation 16
To increase usability/intuitiveness for the target age group 11
To increase long-term engagement 8
To investigate the effects of game-like tasks 7
To stimulate the brain 6
To increase ecological-validity 6
No clear reason given 5
To increase suitability for the target disorder 4
Reasons for using gamelike tasks

11. Can gamified cognitive tasks be valid?
• 21/33 studies compared a gamelike task against a non-game counterpart
• Majority of gamelike cognitive assessments validated fairly successfully, but
mixed-domain measures were an issue
• The effect of gamification in intervention studies was less robust, often due to
a lack of non-gamelike control tasks.
(Atkins et al., 2014) (McPherson J, Burns NR, 2008) (Aalbers et al., 2013)

12. Can gamified cognitive tasks be engaging?
• Study authors were generally enthusiastic about gamification
• Where reported, subjective (e.g. questionnaires) and objective (e.g. training
time) measures of participant engagement were also positive
(Dörrenbächer et al., 2014) (Katz et al,. 2014) (Dovis et al,. 2012)

13. Conclusions
• No evidence that gamification can improve data quality (e.g. reducing
between-subject noise, improving performance)
• Gamification is, at best, performance neutral and may have a negative impact
• Some evidence that gamification may enhance cognitive training
• (But this is tentative due to numerous methodological problems)
• Gamified tasks are more engaging than traditional tasks
• Further research and rigorous validation is needed to understand the interplay
between game mechanics and cognitive processes

14. Acknowledgements
Tobacco and Alcohol Research Group:
Angela Attwood Postdoc
Emily Crowe PhD Student
Kayleigh Easey PhD Student
Meg Fluharty PhD Student
Therese Freuler Research Assistant
Suzi Gage Postdoc
Meryem Grabski PhD Student
Gemma Hammerton Postdoc
Eleanor Kennedy PhD Student
Jasmine Khouja PhD Student
Glenda Lassi Postdoc
Rebecca Lawn PhD Student
Jim Lumsden PhD Student
Olivia Maynard Postdoc
Andy Skinner Postdoc
Alex Board Administrator
Amy Taylor Postdoc
Gemma Taylor Postdoc
Chris Stone Research Assistant
David Troy PhD Student
Miriam Cohen PhD Student
Andy Eastwood PhD Student
My supervisors:
Dr Jenny Barnett
Dr David Coyle
Dr Charlotte Housden
Prof Natalia Lawrence
Prof Marcus Munafò
jim.lumsden@bristol.ac.uk
@jl9937

15. References
• Brown H, Zeidman P, Smittenaar P, Adams R, McNab F, Rutledge R, et al. Crowdsourcing for Cognitive Science - The Utility of Smartphones. PloS One 2014 Jul 15;9.
• Dörrenbächer S, Müller P, Tröger J, Kray J. Dissociable effects of game elements on motivation and cognition in a task-switching training in middle childhood. Front Psychol 2014
• Manera V, Petit P, Derreumaux A, Orvieto I, Romagnoli M, Lyttle G, et al. 'Kitchen and cooking,' a serious game for mild cognitive impairment and Alzheimer's disease: a pilot
study. Front Aging Neurosci 2015;7:24
• Atkins S, Sprenger A, Colflesh G, Briner T, Buchanan J, Chavis S, et al. Measuring working memory is all fun and games: a four-dimensional spatial game predicts cognitive task
performance. Exp Psychol 2014;61(6):417-438.
• Hawkins GE, Rae B, Nesbitt KV, Brown SD. Gamelike features might not improve data. Behav Res Methods 2013 Jun;45(2):301-318.
• Prins PJ, Brink E, Dovis S, Ponsioen A, Geurts H, de Vries M, et al. “Braingame Brian”: Toward an Executive Function Training Program with Game Elements for Children with ADHD
and Cognitive Control Problems. Games Health J 2013 Feb;2(1):44-49.
• Aalbers T, Baars Maria A E, Olde Rikkert Marcel G M, Kessels Roy P C. Puzzling with online games (BAM-COG): reliability, validity, and feasibility of an online self-monitor for
cognitive performance in aging adults. J Med Internet Res 2013;15(12)
• Dennis TA, O'Toole L. Mental Health on the Go: Effects of a Gamified Attention Bias Modification Mobile Application in Trait Anxious Adults. Clin Psychol Sci 2014
• Tong T, Chignell M. Developing a Serious Game for Cognitive Assessment: Choosing Settings and Measuring Performance. In: Proceedings of the Second International Symposium
of Chinese CHI. 2014 Apr 26 Presented at: Chinese CHI 2014; April 26-27, 2014; Toronto, Canada p. 70-79.
• McPherson J, Burns NR. Assessing the validity of computer-game-like tests of processing speed and working memory. Behav Res Methods 2008 Nov;40(4):969-981.
• Dovis S, Van der Oord Saskia, Wiers R, Prins Pier J M. Can motivation normalize working memory and task persistence in children with attention-deficit/hyperactivity disorder? The
effects of money and computer-gaming. J Abnorm Child Psychol 2012 Jul;40(5):669-681
• Boendermaker WJ, Boffo M, Wiers RW. Exploring Elements of Fun to Motivate Youth to Do Cognitive Bias Modification. Games Health J 2015 Dec;4(6):434-443.
• Dunbar N, Miller C, Adame B, Elizondo J, Wilson S, Lane B, et al. Implicit and explicit training in the mitigation of cognitive bias through the use of a serious game. Computers in
Human Behavior 2014;37:307-318.
• Dubbels B. Gamification, Serious Games, Ludic Simulation, and Other Contentious Categories. International Journal of Gaming and Computer-Mediated Simulations 2013
• Katz B, Jaeggi S, Buschkuehl M, Stegman, Shah. Differential effect of motivational features on training improvements in school-based cognitive training. Front Hum Neurosci 2014

16. 1623 November 2016
Any questions?