Games, Learning, andStealth Assessment Valerie Shute Florida State University Games for Learning @ G4C (June 22, 2011)
Games, Learning, Assessment Claim 1 Claim 2 Claim 3 + + +Good games can act Learning is best Stealth assessmentas transformative when it is can collect dynamicenvironments to active, interesting, g evidence of learningsupport skill oal-oriented, and in real-time, atdevelopment and contextualized various grain sizesdeep, meaningful (i.e., features of good (and use info tolearning. games). support learning).
Game Plan Define 21st century competencies & good games Overview evidence-centered design Describe stealth assessment Illustrate approach in a worked example
It’s Time for a ChangeThe world has changed a lot in the past 100 years. Education has not. Classroom photo, 1910 Classroom photo, 2010
21st Century Skills Today’s kids need new skills to be successful and productive (e.g., creativity, collabor ation, communication). The world in which we live is complex and interconnected (especially compared to the really old days).
CollaborationCritical Thinking Systems ThinkingCuriosity Creativity Grit Empathy
Challenges One problem with embracing new skills is the lack of valid & reliable assessments for them. Old ways of testing won’t work. Learning and succeeding in our complex world can’t be optimally be measured by MC tests. Re-think assessment!
Assessment Design Competency Model What do you want to say about the person? Evidence Model What observations would provide best evidence for what you want to say? Task/Action Model Model What kinds of tasks let you make the necessary observations?
Stealth Assessment Features Seamless & Ubiquitous Accurate & Rich Assessment Learner ModelsAligned along 3 dimensions: vertically, horizontally, and temporally
Stealth Assessment Advances in measurement let us administer evidence-based assessments to: Extract ongoing information from a learner Make accurate inferences of competencies React in immediate and helpful ways. Accomplished via automated scoring and machine-based reasoning techniques. When assessment is so seamlessly woven into the fabric of the learning environment that it’s invisible, this is stealth assessment.
Elder Scrolls IV: Oblivion First person 3D RPG set in a medieval world Can be one of many characters (e.g., knight, mage, elf), each who has (or can obtain) various weapons, spells, and tools Primary goal—gain rank & complete quests Quests may include locating a person to obtain info, figuring out a clue for future quests, etc. Multiple mini quests along the way, and a major quest that results in winning the game (100s of hr of game play) Players have the freedom to complete quests in any order
Competency Model Success in Oblivion Cognitive Dispositions Creative Problem Solving Problem Persistence Creativity Solving ReadingAttention Efficiency Novelty Comp Working Background Reasoning Exploratory Reflection Memory Knowledge Skills Behavior
Example ECD Models Competency Model Creative Problem SolvingUnobservables Problem Solving Creativity Efficiency Novelty Evidence Model Scene 1 The Glue Scene 2 Action Model Scene 1 Observables Scene 2 Action Indicators
Action Model w/Indicators Problem: Cross river filled with dangerous fish to get to the cave on the other side. Relevant* Action Novelty Efficiency Swim across the river n = 0.12 e = 0.22 Levitate over the river n = 0.33 e = 0.70 Freeze water with a spell and slide across n = 0.76 e = 0.80 Find a bridge over the river n = 0.66 e = 0.24 Dig a tunnel under the river n = 0.78 e = 0.20 * Relevant refers to any action included in a successful solution.
Indicators Per ActionNovelty: 1 – frequencyEfficiency: Inverse fn (resources, time)Action: Find a bridge over the riverIndicators: Novelty = 1 - 0.34 = 0.66 Efficiency = 1 / [(3 0.4) + (5 0.6)] = 0.24• Resources Used = Weapon (1, fight monster w/sword) + Health (1, damage from monster) + Object (1, magic potion) = 3 resources (weight = 0.4)• Time expended = 5 minutes (weight = 0.6)
Bayes Model—Case 1 CreativeProblemSolving Low 0.60 High 0.40 ProblemSolving CreativityLow 0.64 Low 0.11High 0.36 High 0.89 Efficiency Novelty Low 0.86 Low 0.02 High 0.14 High 0.98 ObservedEfficiency ObservedNovelty 0 to 0.25 1 0 to 0.25 0 0.25 to 0.5 0 0.25 to 0.5 0 0.5 to 0.75 0 0.5 to 0.75 0 0.75 to 1 0 0.75 to 1 1 0.20 0.07 0.78 0.07 Dig a tunnel under the river: e = 0.20; n = 0.78
Wrapping it Up Bayes nets can be used in various ways to improve learning and performance. Continuously gather evidence for accurate, real-time estimates of comp’s. Info on competencies used by (a) teachers (to adjust instruction & give good feedback), (b) system (to select new gaming experiences), and/or (c) students (to reflect on how they’re doing). Current estimates of competency levels can be integrated into the game and displayed as progress indicators. This elevates valued competencies to the same level as health and weapons!
Summary To address educational challenges and harness potential of immersive games, I presented an ECD-inspired idea re: • Specifying competencies to be acquired from the game • Defining EMs that link game behaviors to competencies • Updating the learner model regularly Using ECD, stealth assessment, and automated data collection and analysis tools is meant to collect valid evidence of students’ emerging competencies, and reduce teachers’ workload allowing them to focus on fostering student learning. Next steps: adapt content (feedback, difficulty levels, scaffolding, etc.) in game to fit current needs of player, test plug ’n play of models, evaluate learning effects, etc.