TRACK 2(3) | DAY 1 - 2 OCT 2017 Jim Lumsden, PhD Student of University of Bristol (GB) Games for Health Europe 2017

1. Can gamification reduce attrition from cognitive
testing studies?
Jim Lumsden
jim.lumsden@bristol.ac.uk
School of Experimental Psychology,
University of Bristol,
BS8 1TU,
Bristol, UK
@jl9937

2. Cognitive Tests
• Cognitive tests are a staple tool in psychological research
• Aim to collect pure measures of cognitive function and are therefore simplistic
and involve lots of repetition
• Unfortunately this makes them rather boring

3. Difficulties with Cognitive Tasks
1. Cognitive tests cannot produce valid measures if the participant is bored
2. Psychology is moving into a new era of internet based research but the
possibility for huge sample sizes is limited by participant reimbursement
costs
More engaging tasks could solve both these problems
Gamification may be the answer

5. Research Questions
1. Do different game mechanics influence willingness to take part?
2. Do different game mechanics effect subjective ratings of engagement?
3. Do different game mechanics effect the cognitive data collected?

6. A Reductionist Approach
• Three variants of a Stop Signal Task, using a single game
mechanic each (Non-Game, Points and Theme)
• Longitudinal design (10 days long)

7. Participants and Procedure
(the attrition study)
• Aimed for 291 participants total (97 in each variant)
• Reimbursed between £4 and £7
Signup through
Prolific
Academic
Randomly
assigned to
variant
4 consecutive days of test sessions
Reimbursed £4 after all completed
Day 1 Day 2 Day 3 Day 4
6 optional days of test sessions
Reimbursed £0.50 for each completed
Day 5 Day 6 Day 7 Day 8 Day 9
Day
10
Study ends

8. Subjective Engagement Questionnaire
1. How enjoyable did you find that?
2. How frustrating did you find that?
3. How difficult was it to concentrate for the duration of that?
4. How well do you think you performed on that?
5. How mentally stimulating did you find that to be?
6. How boring did you find that?
7. How much effort did you put in throughout that?
8. How repetitive did you find that?
9. How willing would you be to do that again tomorrow?
10. How willing would you be to recommend the study to a friend?
Very enjoyable Not very enjoyable

9. Does Gamification affect attrition?
All participants
(95% CI)
Conforming participants
(95% CI)
Non-Game 4.9 (4.4 to 5.5) 7.4 (6.8 to 8.0)
Points 5.1 (4.5 to 5.6) 7.5 (7.0 to 8.0)
Theme 5.3 (4.7 to 5.9) 8.0 (7.5 to 8.6)
N = 260
Log-Rank: (X2 (2, N = 265) = 3.022, p = 0.22)
ANOVA: (F [2,262] = 1.534, p = 0.21, partial η2 = 0.012)
Points = Non-Game (BF = 0.163)

10. Does Gamification affect attrition (part 2)?
Log-Rank: (X2 (2, N = 482) = .816, p = .67)
N = 297N = 482
Log-Rank: (X2 (2, N = 297) = 1.418, p = .49)

11. Does Gamification affect subjective ratings of a task?
Does score predict return next day? Logistic regression: (β = 0.008, SE = 0.005, Wald(1) = 2.166, p = 0.141)

12. Cognitive data
• Small negative effects of game mechanics on cognitive measures
SSRT: F [2,255] = 2.954, p = 0.054, partial η2 = 0.022
Points = Non-Game (BF = 0.223)
Non-Game != Theme (BF = 0.590)

13. Summary
• No effect of gamification on attrition
• Differences in subjective ratings didn’t correspond to
attrition
• Small negative effect of gamification on cognitive data

14. Why did subjective ratings not correspond to dropout?
• Two different concepts of Engagement (Perski et al,. 2016)
• Engagement as Usage
• Engagement as Subjective Experience
• Games are played for a variety of reasons
• Boredom, Loneliness, Escapism (Boyle et al., 2007)
• Financial incentives may disrupt autonomy
• Therefore disrupting intrinsic motivation (Deci et al., 1999)
• Expectation Violation
• “Looks like a game but doesn’t play like one” (Boendermaker et al., 2016)

15. Limitations & Conclusions
• Points collected comparably valid SSRTs and increased participant enjoyment
• But, our (rather limited) gamification had no effect on drop out rates
• Studies of broader literature show that gamification is effective (Looyestyn
et al., 2017)
• Gamification is applied more liberally
• But studies within psychology show it isn’t (Brown et al., 2016)
• Researchers are typically conservative
• If we want to create truly engaging cognitive tests we need to be bolder,
moving away from typically cognitive tests and towards fully gamelike tasks
• As for me, there’s one study left in my PhD. I’m going to look at short term
engagement (within a single session), rather than engagement over days

16. Acknowledgements
Tobacco and Alcohol Research Group:
Angela Attwood Postdoc
Emily Crowe PhD Student
Kayleigh Easey PhD Student
Meg Fluharty PhD Student
Therese Freuler Research Assistant
Suzi Gage Postdoc
Meryem Grabski PhD Student
Gemma Hammerton Postdoc
Eleanor Kennedy PhD Student
Jasmine Khouja PhD Student
Glenda Lassi Postdoc
Rebecca Lawn PhD Student
Jim Lumsden PhD Student
Olivia Maynard Postdoc
Andy Skinner Postdoc
Alex Board Administrator
Amy Taylor Postdoc
Gemma Taylor Postdoc
Chris Stone Research Assistant
David Troy PhD Student
Miriam Cohen PhD Student
Andy Eastwood PhD Student
My supervisors:
Dr Jenny Barnett
Dr David Coyle
Dr Charlotte Housden
Prof Natalia Lawrence
Prof Marcus Munafò
jim.lumsden@bristol.ac.uk
@jl9937

17. 1730 October 2017
Any questions?

20. Cognitive Task Results
(the attrition study)
SSRT:
theme vs non-game (BF = .278)