G4LI Games for Learning Day at G4C 2011

1. Games, Learning, and
Stealth Assessment
Valerie Shute
Florida State University
Games for Learning @ G4C (June 22, 2011)

2. Games, Learning, Assessment
Claim 1 Claim 2 Claim 3
+ + +
Good games can act Learning is best Stealth assessment
as transformative when it is can collect dynamic
environments to active, interesting, g evidence of learning
support skill oal-oriented, and in real-time, at
development and contextualized various grain sizes
deep, meaningful (i.e., features of good (and use info to
learning. games). support learning).

3. Game Plan
 Define 21st century
competencies & good games
 Overview evidence-centered
design
 Describe stealth assessment
 Illustrate approach in a worked
example

4. It’s Time for a Change
The world has changed a lot in the past 100 years. Education has not.
Classroom photo, 1910 Classroom photo, 2010

5. 21st Century Skills
 Today’s kids need new
skills to be successful
and productive
(e.g., creativity, collabor
ation, communication).
 The world in which we
live is complex and
interconnected
(especially compared to
the really old days).

6. Collaboration
Critical Thinking Systems Thinking
Curiosity Creativity
Grit Empathy

7. Challenges
 One problem with
embracing new skills is
the lack of valid &
reliable assessments for
them.
 Old ways of testing won’t
work. Learning and
succeeding in our
complex world can’t be
optimally be measured
by MC tests.
 Re-think assessment!

8. Games

9. Good Game Elements
1. Interactive problem
solving
2. Specific goals/rules
3. Adaptive challenges
4. Control
5. Ongoing feedback
6. Uncertainty
7. Sensory Stimuli

10. Gestalt of Games
Interactivity
Stimuli Goals
Uncertainty Challenges
Feedback Control

11. Evidence-Centered Design
(e.g., Mislevy, Steinberg, & Almond, 2003)

12. Assessment Design
Competency Model
What do you want to say about the
person?
Evidence Model
What observations would provide best
evidence for what you want to say?
Task/Action Model
Model
What kinds of tasks let you make the
necessary observations?

13. Design & Diagnosis

14. Stealth
Assessment

15. Stealth Assessment Features
Seamless & Ubiquitous Accurate & Rich
Assessment Learner Models
Aligned along 3 dimensions: vertically, horizontally, and temporally

16. Stealth Assessment
 Advances in measurement let us administer
evidence-based assessments to:
 Extract ongoing information from a learner
 Make accurate inferences of competencies
 React in immediate and helpful ways.
 Accomplished via automated scoring and
machine-based reasoning techniques.
 When assessment is so seamlessly woven into
the fabric of the learning environment that it’s
invisible, this is stealth assessment.

17. Elder Scrolls IV
Oblivion

18. Elder Scrolls IV: Oblivion
 First person 3D RPG set in a medieval world
 Can be one of many characters
(e.g., knight, mage, elf), each who has (or can obtain)
various weapons, spells, and tools
 Primary goal—gain rank & complete quests
 Quests may include locating a person to obtain
info, figuring out a clue for future quests, etc.
 Multiple mini quests along the way, and a major quest
that results in winning the game (100s of hr of game play)
 Players have the freedom to complete quests in any order

19. Competency Model
Success in Oblivion
Cognitive Dispositions
Creative
Problem
Solving
Problem Persistence
Creativity
Solving
Reading
Attention Efficiency Novelty
Comp
Working Background Reasoning Exploratory
Reflection
Memory Knowledge Skills Behavior

20. Example ECD Models
Competency Model
Creative Problem Solving
Unobservables
Problem Solving Creativity
Efficiency Novelty
Evidence Model
Scene 1
The Glue Scene 2
Action Model
Scene 1
Observables Scene 2
Action Indicators

21. Action Model w/Indicators
Problem: Cross river filled with dangerous fish to get to the
cave on the other side.
Relevant* Action Novelty Efficiency
Swim across the river n = 0.12 e = 0.22
Levitate over the river n = 0.33 e = 0.70
Freeze water with a spell and slide across n = 0.76 e = 0.80
Find a bridge over the river n = 0.66 e = 0.24
Dig a tunnel under the river n = 0.78 e = 0.20
* Relevant refers to any action included in a successful solution.

22. Indicators Per Action
Novelty: 1 – frequency
Efficiency: Inverse fn (resources, time)
Action: Find a bridge over the river
Indicators: Novelty = 1 - 0.34 = 0.66
Efficiency = 1 / [(3 0.4) + (5 0.6)] = 0.24
• Resources Used = Weapon (1, fight monster w/sword) + Health (1, damage
from monster) + Object (1, magic potion) = 3 resources (weight = 0.4)
• Time expended = 5 minutes (weight = 0.6)

23. Bayes Model—Case 1
CreativeProblemSolving
Low 0.60
High 0.40
ProblemSolving Creativity
Low 0.64 Low 0.11
High 0.36 High 0.89
Efficiency Novelty
Low 0.86 Low 0.02
High 0.14 High 0.98
ObservedEfficiency ObservedNovelty
0 to 0.25 1 0 to 0.25 0
0.25 to 0.5 0 0.25 to 0.5 0
0.5 to 0.75 0 0.5 to 0.75 0
0.75 to 1 0 0.75 to 1 1
0.20 0.07 0.78 0.07
Dig a tunnel under the river: e = 0.20; n = 0.78

24. Wrapping it Up
 Bayes nets can be used in various ways to
improve learning and performance.
 Continuously gather evidence for
accurate, real-time estimates of comp’s.
 Info on competencies used by (a) teachers
(to adjust instruction & give good
feedback), (b) system (to select new gaming
experiences), and/or (c) students (to reflect
on how they’re doing).
 Current estimates of competency levels can be
integrated into the game and displayed as
progress indicators.
 This elevates valued competencies to
the same level as health and weapons!

25. Summary
 To address educational challenges and harness potential of
immersive games, I presented an ECD-inspired idea re:
• Specifying competencies to be acquired from the game
• Defining EMs that link game behaviors to competencies
• Updating the learner model regularly
 Using ECD, stealth assessment, and automated data collection
and analysis tools is meant to collect valid evidence of students’
emerging competencies, and reduce teachers’ workload allowing
them to focus on fostering student learning.
 Next steps: adapt content (feedback, difficulty
levels, scaffolding, etc.) in game to fit current needs of player, test
plug ’n play of models, evaluate learning effects, etc.

26. Thank you!
Questions?
Email: vshute@fsu.edu
Website: http://www.myweb.fsu.edu/vshute

27. Really Old Days
 In the Stone Age, kids
learned how to start a fire
with sticks. Hunting skills
were important too.
 The world in which they
lived was fairly simple and
flat. Return