This document describes Dr. Jacob Habgood's research into evaluating game-based learning using a case study of his game Zombie Division. It discusses four designs of his classroom research studying the game, comparing intrinsic and extrinsic approaches. Design 4 found that an intrinsic group using teacher-led reflection had larger learning gains and gains that persisted longer according to delayed post-testing. The document also discusses the practical challenges of conducting educational game research in real classrooms.

1. Zombie Division:
AMethodological Case Studyfor theEvaluation
of Game-Based Learning
Dr. Jacob Habgood
Sheffield Hallam University, UK

2. Who am I?
• Ex-game developer
• Author
• Teacher of PlayStation
development
• Researcher of zombies
(or am I?)
Jake

3. Zombie Division
The backstory

4. Zombie Division
• PhD Research 2005/6
• Zombie count = 0
• Intrinsic Integration
• Kafai (2001)
• Results published in 2011
• Empirical evidence supporting
an intrinsic approach as more
educationally effective
• Download the game from
ResearchGate (not Win8+)
Habgood & Ainsworth. The Journal of the
Learning Sciences 20.2 (2011): 169-206
Habgood (2007)
“The Effective
Integration of Digital
Games and Learning
Content”
“Chocolate-covered broccoli”
Bruckman (1999)
BAD
Download and play it
for yourself!

5. Research Design
Asking the right research question…

6. Epistemology
• GBL = “Broad Church”
• Began in late 60’s early 70’s
• Are we able to clearly
demonstrate that GBL is
effective yet?
• Lack of high-quality
empirical evaluations
• “Effect on learning qualified
as moderately positive”
(Wouters et al., 2015)

7. HAA-PEE-RR
ZOM-BEE
HAA-PEE
ZOM-BEE
Educationally
Effective?
• Learning gains
• Pre-post test
• Motivation
• Self-reported measures
• Time-on-task
• Meaningful comparisons
• Improvement and/or
engagement isn’t
necessarily anything to
shout about!
+
Before
After

8. Making the Right
Comparison
• Classroom comparisons of
learning
• More effective than a cup of
coffee?
• Circular problem
• Home comparisons of
motivation
• More time than what?
• Circular problem
Is GBL more effective than traditional classroom teaching?
Is GBL as motivating as mainstream digital entertainment?
Is classroom teaching more effective combined with GBL?
Is GBL motivating enough to compete for time in the home?

9. Making the Right
Comparison
• “Status Quo”: Progress
• Chocolate-Covered Broccoli
• Loosely coupled learning and
gameplay
• “Intrinsically Integrated”
• Tightly coupled learning and
gameplay

10. Classroom Research
The practicalities of “real world research”

11. Ethics
• Duration of Intervention
• Competing demands
• Control group
• 20 minutes per day for week
• Age ratings
• PEGI: Europe
• ESRB: America
• Recommend Google Play’s
automated system (requires a
developer account)
?
Treatment group
Treatment group
Control group

12. Treatment Groups
• Group size n >= 20
• 3 groups = 60+ participants
• 2 classes per year group
• <= 30 students per class
• Combine year groups
• Eldest from year below
• Youngest from year above

13. Group Assignment
• Randomised assignment
• Small group sizes can lead to
confounding factors
• Matched Designs
• Sorted by matched variable
• Balanced dependant variable
• Repeat by gender
• Absenteeism
• Delay between pre-test and
intervention
• Inevitable drop-out rate?
80% 79%
76% 71%
63% 58%
55% 52%
66%
67.5%

14. Gaming “Episode”
• 120 minutes of play!
• Familiar genre/controls
• Zelda / Spyro
• Focussed learning content
• NOT a commercial product!
• The inverse relationship
between multiplication and
division (National Curriculum)

15. Zombie Division
The research design “journey”

16. Design 1:
Learning Gains
• Sixty-four 7-9 year olds
• School day in ICT suite
• Two conditions and control
• Control = just chocolate!
• Pre to Post-test design
• Paper-based tests
• 20 minutes playing per day
x64
Extrinsic group
Intrinsic group
Control group

17. x64
Design 1:
Learning Gains
• No main effect of group!
• Absenteeism didn’t help
• Ceiling effect of pre/post tests
• Making link between hundred
square and division problems
was too hard
• Swapped for multiplication grid
EEE-SEE!

18. Design 2:
Time-on-task vs.
alternative
• Forty-four 7-8 year olds
• School holidays in learning centre
• Two conditions (no control)
• Free choice
• Zombie Division
• BBC Website games
• No absentees / ceiling effects
• A single day (135 minutes)
• Computerised test (fixed time,
unlimited questions)
• Automated group assignment
x44
Extrinsic groupIntrinsic group

19. Design 2:
Learning Gains
• No main effect of group!
• No absenteeism
• No ceiling effects
• Contamination
• Competition between groups
• Frustration with extrinsic
!
x44
Extrinsic groupIntrinsic group
MEEE WIN NO MEEE

20. Design 3:
Time-on-task with
free switching
• Sixteen, 7-8 year olds
• After school club (3x45 mins)
• Free switching
• Played both versions
• Randomised order
• No loss of progress
• Huge preference for intrinsic
• 61% of time vs. 8%
• Deep design understanding
• “subliminal learning with maths”
x16

21. Design 4:
Learning gains
with teacher-led
reflection
• Sixty-four 7-8 year olds
• School day in ICT suite
• Two conditions and control
• Control = just chocolate!
• Pre, post, delayed-test design
• Computer-based tests
• 20 minutes playing per day
• Additional reflection activities
• Delayed test
• Transfer tests
x58
Extrinsic group
Intrinsic group
Control group
Day 1 Computer-based pre-tests.
Matched assignment: intrinsic,
extrinsic & control.
2-3 20 minutes playing the game.
4 30 mins reflection + 20 mins play.
5-6 20 minutes playing the game.
7 Post-test
2 weeks
8 1 x 35 minutes playing the game.
Delayed-test and transfer-test.

22. Design 4:
Learning gains
with teacher-led
reflection
• Intrinsic group differences!
Larger overall percentage gains
(15%pts).Percentage gains
between post and delayed were
significantly higher
• Results published in 2011
• Habgood M. P. J. & Ainsworth, S.
Motivating Children to Learn
Effectively: Exploring the Value of
Intrinsic Integration in
Educational Games
The Journal of the Learning
Sciences 20.2 (2011): 169-206
Habgood & Ainsworth. The Journal of the
Learning Sciences 20.2 (2011): 169-206
10
20
30
40
50
60
70
Pre Post Delayed
PercentageScore
Intrinsic
Extrinsic
Control

23. Process Data
The practicalities of “real world research”

24. Process Data
• Time-stamped log files
• Post-hoc analysis
• Baker, Ryan SJD, et al.
"Modelling the acquisition of
fluent skill in educational
action games." User Modelling
2007. Springer Berlin
Heidelberg, 2007. 17-26.
• Behaviour analytics
• E.g. in and out of rooms
• E.g. pausing strategy

25. Input Streaming
• Deterministic
• Controller input streams
• Replay against game exe
• Pseudo-random numbers
• 130 hours = 8MB storage!
• Out of sync errors
• Butterfly effect
• Self-correcting data
• Periodic position/orientation
• Can’t help overall game state

26. The End
Any questions?
j.habgood@shu.ac.uk