Commercial Video Games as Preparation for Future Learning
1. Commercial
Video
Games
as
Prepara1on
for
Future
Learning
Dylan
Arena
Stanford
University
June
2012
Tuesday, June 26, 12
Hi!
OK,
I’m
gonna
get
started.
I’ll
be
talking
today
about
the
poten1al
of
using
commercial
video
games
as
prepara1on
for
future
learning.
3. Tuesday, June 26, 12
We
in
the
GLS
community
claim
that
gameplay
can
be
great
for
learning,
and
by
that
we
tend
to
mean
two
things:
4. Tuesday, June 26, 12
First,
that
we
can
design
learning-‐based
games
(like
Quest
Atlan1s)
to
support
“schoolish”
learning…
5. Tuesday, June 26, 12
…and
second,
that
we
can
observe
other
interes1ng
kinds
of
learning
even
in
commercial,
off-‐
the-‐shelf
games
(like
World
of
WarcraY).
6. Tuesday, June 26, 12
A
third
claim—that
simply
playing
commercial
games
recrea1onally…
9. Tuesday, June 26, 12
—seems
a
bit
silly
at
first,
if
only
because
most
schoolish
tests
focus
on
retrieval
of
facts
that
most
commercial
games
aren’t
designed
to
teach.
10. Tuesday, June 26, 12
But
if
we
broaden
our
view,
it’s
a
bit
less
silly.
12. Tuesday, June 26, 12
Imagine
spli]ng
a
class
in
half:
half
get
to
explore
a
forest
(turning
over
rocks,
looking
inside
ro_en
logs,
poking
ant
hills)
and
half
don’t
(they
just
stand
quietly
facing
the
wall).
13. Forest
ecosystem
test
Tuesday, June 26, 12
Then
bring
both
groups
back
into
class
and
give
‘em
a
tradi1onal
mul1ple-‐choice
test
about
forest
ecosystems.
You’d
expect
no
difference,
right?
It’s
unlikely
that
just
exploring
the
forest
would
give
kids
the
kind
of
knowledge
they’d
need
to
pick
correct
answers
on
a
mul1ple-‐choice
test.
14. Forest
ecosystem
test
Forest Ecosystem
Lecture
Tuesday, June 26, 12
But
imagine
then
giving
a
LECTURE
about
forest
ecosystems—ground
cover,
canopy,
decay
and
new
growth,
etc.
It’s
possible
that
the
kids
who
had
just
been
out
exploring
a
forest
might
engage
more
with
that
lecture
and
hence
learn
more.
15. Forest
ecosystem
test
Forest Ecosystem
Lecture
Forest
ecosystem
test
Tuesday, June 26, 12
If
you
then
test
everyone
again
aYer
the
lecture,
you
might
observe
some
previously
hidden
benefits
of
the
forest
field
trip.
The
point
is
that
what
we
bring
into
a
learning
situa1on
(like
a
lecture)
is
obviously
very
important.
But
it
can
be
hard
to
measure
what
we
bring
in,
especially
when
it
isn’t
stable,
well-‐
structured
factual
knowledge.
16. Prepara1on
for
Future
Learning
(PFL)
[Bransford
&
Schwartz,
1999]
Bad (or no)
Good Experience
Experience
Sequestered-‐Problem-‐Solving
test
Future Learning
Prepara1on-‐for-‐Future-‐Learning
test
Tuesday, June 26, 12
That’s
where
the
“Prepara1on
for
Future
Learning”
part
of
my
1tle
comes
in.
PFL
is
an
assessment
framework
designed
to
measure
inchoate
forms
of
prior
knowledge
that
tradi1onal
(or
“sequestered-‐problem-‐solving”)
tests
miss.
This
slide
shows
a
generalized
diagram
of
the
forest-‐test-‐lecture-‐test
scenario
I
just
described:
some
learners
have
a
good
(which
is
to
say,
learning-‐relevant)
experience,
and
others
don’t.
On
a
Sequestered-‐Problem-‐Solving
test,
they
look
about
the
same.
But
if
you
then
provide
a
learning
opportunity
that
is
designed
to
mold
that
prior
experience
into
a
formal
knowledge
structure
and
test
‘em
again,
you
can
detect
the
benefits
(or
lack
thereof)
of
the
experience.
17. Study Design
Tuesday, June 26, 12
In a moment I’ll talk about how I used the PFL framework, but first, a point about my
study-design goals.
Reports on the state of the field, like the 2011 National Research Council report,
describe the evidence for games supporting schoolish learning as “emerging”,
“inconclusive”, and “very limited”, with “gaps and weaknesses” that “make it difficult
to...demonstrate their effectiveness...”
These statements reflect the fact that many stakeholders want from the GLS community
something like an FDA study: a randomized field trial with an intention-to-treat analysis
and very traditional, schoolish operationalizations of learning. So I decided to try to run
one.
18. Study Design
Tuesday, June 26, 12
Here’s a first pass at the study design, stripped down so you can see the parallels with
the forest-field trip example (more details will follow).
I randomly assigned community-college students to three conditions: play Civilization 4,
play Call of Duty 2, or play no game. (I just gave gameplay participants the games
they’d been assigned and told ‘em to play at home however they normally play, for at
least 15 hours over the course of about 5 weeks.)
19. Study Design
Tuesday, June 26, 12
Then I had all participants come in and take a 16-item multiple-choice test about World
War II history.
20. Study Design
Tuesday, June 26, 12
Then I had them watch a 20-minute narrated-slideshow lecture about World War II
history.
21. Study Design
Tuesday, June 26, 12
Then I gave ‘em another multiple-choice test about World War II history (this time 36
items).
22. Study Design
Tuesday, June 26, 12
So that’s the basic study design. Here I’ll fill in a few more details.
23. Study Design
Tuesday, June 26, 12
First, the participants were 102 local community-college students (Control: n = 33;
CoD2: n = 34; Civ4: n = 35; 16-42 yrs, median 20 yrs; 64% female) whom I
compensated with course credit and (if they played for the full 15 hours) pay; all had
completed a huge demographic questionnaire (roughly 280 questions) as part of their
research-participation program; and the way I explained the study was that everyone
would get a free game and (possibly) a gift card, with the only differences being
*which* game and *when* the gameplay would occur (before or after the in-person
session); this way Control participants wouldn’t feel shortchanged.
24. Study Design
Tuesday, June 26, 12
The way I verified gameplay was by collecting and analyzing participants’ save-game
files (auto-generated by games so players can pick up where they left off).
25. Study Design
Tuesday, June 26, 12
All players whose save-game files showed evidence of at least 15 hours of gameplay
were compensated with $75 gift cards (45 people earned ‘em: 11 Control, 15 CoD2,
19 Civ).
(And you can see here that the “control” participants did receive a game to play for 15+
hours and got compensated if they did so.)
27. The Games
Tuesday, June 26, 12
I chose these games because they were (a) both really popular with players and critics
when they were released in 2005; (b) from successful franchises of games; (c) and
old enough to be playable on any modern computer but still new enough to seem
“cool”.
29. The Games: Civilization IV
Tuesday, June 26, 12
You play as the immortal, autocratic ruler of a civilization, and your task is to guide your
people through roughly 6000 years of history by making lots of choices.
30. The Games: Civilization IV
Tuesday, June 26, 12
You’re plopped down on some arbitrary Earth-like world, and you settle cities, build
infrastructure, engage with other civilizations through diplomacy and/or warfare, and
create wonders based on those in Earth’s history (like the Parthenon, or Rock ‘n’ Roll,
or the Manhattan Project).
31. The Games: Civilization IV
Tuesday, June 26, 12
By the end of the game, you’ll have built a bunch of cities, fought some wars, and made
a ton of choices.
32. The Games: Call of Duty 2
Tuesday, June 26, 12
CoD2 is a first-person-shooter game.
33. The Games: Call of Duty 2
Tuesday, June 26, 12
You play as a lowly soldier: a Soviet peasant repelling the German invasion; then later a
Brit in the North African campaign; and finally an American in the invasion of France,
ending the game by crossing the Rhine into Germany.
34. The Games: Call of Duty 2
Tuesday, June 26, 12
In contrast to Civ4, CoD2 is a real-time game in which you are required to navigate a
3D environment and shoot things (like teddy bears).
35. The Games: Call of Duty 2
Tuesday, June 26, 12
You play as part of a small squad of soldiers overcoming various obstacles to reach the
next objective, which is marked as a gold star on your map (lower left). CoD2
gameplay is fast, twitchy, and visceral, not much time for thoughtful reflection—you
just shoot whatever threatens you and move toward the next gold star. But it’s all
happening in the historical context of WWII theaters of war.
36. The Lecture
Tuesday, June 26, 12
The lecture covered WWII from the initial troubles in Asia in the 1930s to the dropping of
the atomic bombs. My primary resource for the lecture was a SparkNotes guide (like
Cliff’s Notes: high-schoolers might use it to study for their history tests). I wrote the
lecture to cover all of WWII but also to focus on two sets of themes, corresponding to
the gameplay experiences that I predicted the two games would produce.
37. The Lecture
Tuesday, June 26, 12
I had hoped that (because of their gameplay experiences) Civ4 players would engage
more with the Nations themes and CoD2 players would engage more with the Battles
themes.
38. The Tests
[SparkNotes] Pre2.1 From the perspective of Western leaders, Stalin’s actions as leader of
the Union of Soviet Socialist Republics reflected an emphasis on which of the following
concepts?
individualism
freedom
human dignity
aggression
[NAEP] Post2.31 When the United States entered the Second World War, one of its allies was
Germany
Japan
the Soviet Union
Italy
[CST] Post2.36 The purpose of the Manhattan Project was to
provide economic aid to Latin American countries
develop atomic weapons for the U.S. military
bring about an end to poverty in U.S. urban areas
offer assistance to relocated European refugees
Tuesday, June 26, 12
I built the pre- and post-lecture tests using traditional multiple-choice items that I pulled
from three sets of standardized tests: a quiz from the back of the SparkNotes guide I
mentioned; the National Assessment of Educational Progress; and the California
Standards Tests. Here are three items.
39. Open-Ended Questions
After post-lecture test, two scenarios not mentioned in the
lecture
Tuesday, June 26, 12
So these traditional tests were my nod to the conservative folks out there who think that
learning is factual retrieval. But we at GLS know better! So in addition to these
traditional tests, I also included two sets of open-ended questions that described
scenarios not mentioned in the lecture.
40. Open-Ended Questions
After post-lecture test, two scenarios not mentioned in the
lecture
One for Nations, one for Battles
Tuesday, June 26, 12
One of these two scenarios was designed to pick up on a focus on the Nations themes I
had tried to build into the lecture (and hence to favor Civ4 players), while the other
was designed to pick up on a Battles focus (and hence favor the CoD2 players).
41. Open-Ended Questions
After post-lecture test, two scenarios not mentioned in the
lecture
One for Nations, one for Battles
Two questions per scenario:
What’s going on?
What would you want to ask to learn more?
Tuesday, June 26, 12
Participants were asked what they thought was going on in each scenario and, more
importantly, what questions they’d want to ask to learn more.
42. Open-Ended Questions
(Nations focus)
In 1940, in Mers-el-Kebir, Algeria, commanders of some British ships spoke with
commanders of some French ships, and then the British ships fired on the French
ships, sinking the ships and killing over 1,200 French sailors.
Why do you think this might have happened? (Feel free to guess.)
What questions would you ask to figure out why this happened? (Don't just say, "I would ask
why this happened." That's too easy. Think about what kinds of facts about the situation you
would want to know.)
Tuesday, June 26, 12
Here’s the first scenario (the Nations-focus one) and its two questions…
43. Open-Ended Questions
(Battles focus)
On June 6, 1944, an American Ranger battalion landed on the beach at the foot of the
cliffs of Pointe du Hoc, in France. They then climbed those cliffs under fire from the
Germans to destroy a set of large artillery guns.
Why do you think the Americans wanted to destroy the guns? (Don't just say, "To stop the
Germans from using them." Be specific. Think about where the Germans might have wanted
to use the guns.)
What questions would you ask to figure out why this happened? (Don't just say, "I would ask
why this happened." That's too easy. Think about what kinds of facts about the situation you
would want to know.)
Tuesday, June 26, 12
…and here’s the Battles-focus scenario and questions.
44. Results
Tuesday, June 26, 12
My analysis protocol for the multiple-choice tests was to fit ANCOVA models using test
scores as the outcome variables. To choose predictors, I created a candidate list of
nine that I had reason to believe might be useful...
45. Candidate predictors for ANCOVA models
• Gameplay condition
Tuesday, June 26, 12
Gameplay condition (operationalized differently by analysis)
46. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
Tuesday, June 26, 12
quarter of data collection (to account for cohort effects)
47. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
Tuesday, June 26, 12
gender
48. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
• Age
Tuesday, June 26, 12
age
49. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
• Age
• Prior gameplay history
Tuesday, June 26, 12
prior-gameplay history (4-level ordinal from “never” to “> 6 times”)
50. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
• Age
• Prior gameplay history
• English proficiency level
Tuesday, June 26, 12
English proficiency (lots of non-native speakers in my sample)
51. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
• Age
• Prior gameplay history
• English proficiency level
• Prior social-studies interest
Tuesday, June 26, 12
prior social-studies interest (5-level Likert)
52. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
• Age
• Prior gameplay history
• English proficiency level
• Prior social-studies interest
• Enjoyment of the assigned game
Tuesday, June 26, 12
enjoyment of the assigned game (obviously only relevant for gameplay participants)
53. Candidate predictors for ANCOVA models
• Gameplay condition
• Quarter of data collection
• Gender
• Age
• Prior gameplay history
• English proficiency level
• Prior social-studies interest
• Enjoyment of the assigned game
• Pre-lecture-test scores (for post-lecture-test ANCOVA)
Tuesday, June 26, 12
and pre-lecture-test scores (obviously only for the post-lecture-test ANCOVA).
I then fed this candidate set of predictors into an all-possible-subsets selection
procedure, which examines all combinations of the predictor set to find the model with
the highest adj-R^2. To avoid capitalizing on spurious patterns in the data set, I also
constrained it to include only models all of whose predictors were at least marginally
significant. I call the resulting model the “parsimonious” model.
54. Parsimonious model for pre-lecture-test scores
Source df SSTypeIII F η2 p
English proficiency 1 70.86 13.32 .12 .00042**
Residuals 100 532.16
R2adj = .11, F(1, 100) = 13.32, p = .00042
Tuesday, June 26, 12
The only predictor of pre-lecture-test scores was English proficiency. This is the SPS
test from our PFL model. NOTE: If this were all we did to test the learning benefits of
recreational commercial gameplay, we’d find no benefit. But on the post-lecture-test...
55. Parsimonious model for post-lecture-test scores
Source df SSTypeIII F η2 p
Received a game 1 88.97 4.05 .026 .047*
Quarter 2 125.40 2.85 .036 .063.
Age 1 123.55 5.63 .036 .020*
English proficiency 1 104.52 4.76 .030 .032*
Prior SS interest 4 334.12 3.80 .096 .0068**
Game enjoyment 4 239.07 2.72 .069 .035*
Pre-lecture test 1 145.08 6.60 .042 .012*
Residuals 87 1910.95
R2adj = .36, F(14, 87) = 5.06, p < .0001
Tuesday, June 26, 12
...all of a sudden a lot is going on. The key points for this talk are (a) many things are
involved in how players will learn from gaming experiences, and (b) gameplay
participants significantly outscored control participants—i.e., they learned more from
the lecture.
In fact, gameplay participants scored about 6% higher on the post-lecture test than did
control participants (without considering covariates, just a straight means
comparison). In the language of school, that translates to over four percentage points
on an exam (74.3% for control participants and 78.6% for gameplay participants), or
nearly half of a letter grade. Cohen’s d = .27, which is substantial for a randomized
field trial.
56. Responses to open-ended questions
Nations focus Control CoD2 Civ4 Battles focus Control CoD2 Civ4
No 22 21 15 No 22 16 26
Yes 11 13 20 Yes 11 18 9
Fisher’s exact test: p = .058 Fisher’s exact test: p = .030
Tuesday, June 26, 12
The other outcome measure, remember, was participants’ responses to open-ended
questions about novel WWII scenarios. It turned out that participants’ gameplay
experiences affected the focus of their responses, with Civ4 participants adopting a
more global “Nations” focus and CoD2 participants adopting a more local “Battles”
focus.
(The specific operationalization of my coding scheme for this scenario was to code
participants as having a “Nations” focus if and only if (a) the participant’s questions
mentioned Resources (including territory), Empires (including colonies), Defenses
(including enemies), or Alliances (including treaties), or (b) the participant wrote of the
actors as being the nations themselves (e.g., Britain, France) rather than agents of
those nationalities (e.g., British commanders, French ships).
The operationalization of my coding scheme for this scenario was to code participants
as having a “Battles” focus if and only if the participant’s questions mention (a)
Weaponry (including capabilities of particular weapons), Terrain (including avenues of
ingress for the engagement), Communication (but not including prior intelligence
about the engagement), or Objectives (but not including consequences of the
engagement) or (b) such tactical elements as the time course of the engagement,
casualties, or troop size. (CoD2 participants’ responses tended to reflect the in-the-
moment viewpoint of a soldier anticipating climbing those cliffs to engage an enemy.))
57. Discussion
Tuesday, June 26, 12
I’ve got three basic takeaways from these results, and then three suggestions for
various stakeholders.
58. Summary of Findings
• Playing enjoyable video games at home can help students learn in school(ish
settings)
Tuesday, June 26, 12
First, the results of this study support the claim that playing enjoyable video games at
home can help both male and female students learn in school, if the formal instruction
leverages the students’ gameplay experiences. (The strong predictive effect of prior
social-studies interest shows the importance of also leveraging students’ interests.)
59. Summary of Findings
• Playing enjoyable video games at home can help students learn in school(ish
settings)
• Different game experiences lend themselves to different types of instruction
Tuesday, June 26, 12
Second, the results from the open-ended questions underscore the notion that different
games will offer different types of experiences that prepare players preferentially for
different topics of formal instruction.
60. Summary of Findings
• Playing enjoyable video games at home can help students learn in school(ish
settings)
• Different game experiences lend themselves to different types of instruction
• Gameplay can influence both retention of facts and choices about what to learn
Tuesday, June 26, 12
And third, the open-ended-question results further suggest that these gameplay
experiences can improve not only retention of facts presented by direct instruction but
also students’ choices about what to learn.
61. Considerations for Practice
• Curriculum Designers: From task analysis of games to curriculum
Tuesday, June 26, 12
Now, my three recommendations for folks who want to cash these results out in some
way.
First, for folks who want to build curricula to leverage gameplay experiences, I’d
suggest a careful task analysis of gameplay to determine the relevant properties of its
experiences (e.g., for Civ4 it was thinking as a nation; for CoD2 it was probably as
simple as just being exposed to the historical context of WWII).
62. Considerations for Practice
• Curriculum Designers: From task analysis of games to curriculum
• Educators: Gameplay doesn’t have to be wasted time
Tuesday, June 26, 12
For educators who have to deal with their students playing games for hours each week,
I’d suggest that they recognize that gameplay is pervasive and powerful and that they
embrace it (by tying the compelling experiences found in games with the powerful
explanatory structures found in the standard curriculum).
63. Considerations for Practice
• Curriculum Designers: From task analysis of games to curriculum
• Educators: Gameplay doesn’t have to be wasted time
• Game Designers: You can be chickens, not pigs
Tuesday, June 26, 12
And for commercial game designers—who know how hard it is to make a good game,
let alone a good learning game, and have therefore steered clear of the educational
game space—I’d say that this study suggests that they needn’t try to cram all of the
curricular content into the game itself. Instead, they can continue to let the game do
what it does best (provide great experiences) with perhaps some small tweaks here
and there to better serve as foundations upon which educators might build.
(from a joke about eggs/bacon for breakfast: the chicken is interested, but the pig is
committed)
64. Acknowledgement
Financial support for this dissertation was provided by a
SUSE Dissertation Support Grant and a Gerald J.
Lieberman Fellowship
Tuesday, June 26, 12
65. Tuesday, June 26, 12
And that’s it—thanks very much for your time!
Questions?