What is Stealth Assessment?


Published on

G4LI Games for Learning Day at G4C 2011

Published in: Education, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

What is Stealth Assessment?

  1. 1. Games, Learning, andStealth Assessment Valerie Shute Florida State University Games for Learning @ G4C (June 22, 2011)
  2. 2. Games, Learning, Assessment Claim 1 Claim 2 Claim 3 + + +Good games can act Learning is best Stealth assessmentas transformative when it is can collect dynamicenvironments to active, interesting, g evidence of learningsupport skill oal-oriented, and in real-time, atdevelopment and contextualized various grain sizesdeep, meaningful (i.e., features of good (and use info tolearning. games). support learning).
  3. 3. Game Plan Define 21st century competencies & good games Overview evidence-centered design Describe stealth assessment Illustrate approach in a worked example
  4. 4. It’s Time for a ChangeThe world has changed a lot in the past 100 years. Education has not. Classroom photo, 1910 Classroom photo, 2010
  5. 5. 21st Century Skills Today’s kids need new skills to be successful and productive (e.g., creativity, collabor ation, communication). The world in which we live is complex and interconnected (especially compared to the really old days).
  6. 6. CollaborationCritical Thinking Systems ThinkingCuriosity Creativity Grit Empathy
  7. 7. Challenges One problem with embracing new skills is the lack of valid & reliable assessments for them. Old ways of testing won’t work. Learning and succeeding in our complex world can’t be optimally be measured by MC tests. Re-think assessment!
  8. 8. Games
  9. 9. Good Game Elements1. Interactive problem solving2. Specific goals/rules3. Adaptive challenges4. Control5. Ongoing feedback6. Uncertainty7. Sensory Stimuli
  10. 10. Gestalt of Games Interactivity Stimuli GoalsUncertainty Challenges Feedback Control
  11. 11. Evidence-Centered Design (e.g., Mislevy, Steinberg, & Almond, 2003)
  12. 12. Assessment Design Competency Model What do you want to say about the person? Evidence Model What observations would provide best evidence for what you want to say? Task/Action Model Model What kinds of tasks let you make the necessary observations?
  13. 13. Design & Diagnosis
  14. 14. StealthAssessment
  15. 15. Stealth Assessment Features Seamless & Ubiquitous Accurate & Rich Assessment Learner ModelsAligned along 3 dimensions: vertically, horizontally, and temporally
  16. 16. Stealth Assessment Advances in measurement let us administer evidence-based assessments to:  Extract ongoing information from a learner  Make accurate inferences of competencies  React in immediate and helpful ways. Accomplished via automated scoring and machine-based reasoning techniques. When assessment is so seamlessly woven into the fabric of the learning environment that it’s invisible, this is stealth assessment.
  17. 17. Elder Scrolls IVOblivion
  18. 18. Elder Scrolls IV: Oblivion First person 3D RPG set in a medieval world Can be one of many characters (e.g., knight, mage, elf), each who has (or can obtain) various weapons, spells, and tools Primary goal—gain rank & complete quests Quests may include locating a person to obtain info, figuring out a clue for future quests, etc. Multiple mini quests along the way, and a major quest that results in winning the game (100s of hr of game play) Players have the freedom to complete quests in any order
  19. 19. Competency Model Success in Oblivion Cognitive Dispositions Creative Problem Solving Problem Persistence Creativity Solving ReadingAttention Efficiency Novelty Comp Working Background Reasoning Exploratory Reflection Memory Knowledge Skills Behavior
  20. 20. Example ECD Models Competency Model Creative Problem SolvingUnobservables Problem Solving Creativity Efficiency Novelty Evidence Model Scene 1 The Glue Scene 2 Action Model Scene 1 Observables Scene 2 Action Indicators
  21. 21. Action Model w/Indicators Problem: Cross river filled with dangerous fish to get to the cave on the other side. Relevant* Action Novelty Efficiency Swim across the river n = 0.12 e = 0.22 Levitate over the river n = 0.33 e = 0.70 Freeze water with a spell and slide across n = 0.76 e = 0.80 Find a bridge over the river n = 0.66 e = 0.24 Dig a tunnel under the river n = 0.78 e = 0.20 * Relevant refers to any action included in a successful solution.
  22. 22. Indicators Per ActionNovelty: 1 – frequencyEfficiency: Inverse fn (resources, time)Action: Find a bridge over the riverIndicators: Novelty = 1 - 0.34 = 0.66 Efficiency = 1 / [(3 0.4) + (5 0.6)] = 0.24• Resources Used = Weapon (1, fight monster w/sword) + Health (1, damage from monster) + Object (1, magic potion) = 3 resources (weight = 0.4)• Time expended = 5 minutes (weight = 0.6)
  23. 23. Bayes Model—Case 1 CreativeProblemSolving Low 0.60 High 0.40 ProblemSolving CreativityLow 0.64 Low 0.11High 0.36 High 0.89 Efficiency Novelty Low 0.86 Low 0.02 High 0.14 High 0.98 ObservedEfficiency ObservedNovelty 0 to 0.25 1 0 to 0.25 0 0.25 to 0.5 0 0.25 to 0.5 0 0.5 to 0.75 0 0.5 to 0.75 0 0.75 to 1 0 0.75 to 1 1 0.20 0.07 0.78 0.07 Dig a tunnel under the river: e = 0.20; n = 0.78
  24. 24. Wrapping it Up Bayes nets can be used in various ways to improve learning and performance.  Continuously gather evidence for accurate, real-time estimates of comp’s.  Info on competencies used by (a) teachers (to adjust instruction & give good feedback), (b) system (to select new gaming experiences), and/or (c) students (to reflect on how they’re doing). Current estimates of competency levels can be integrated into the game and displayed as progress indicators. This elevates valued competencies to the same level as health and weapons!
  25. 25. Summary To address educational challenges and harness potential of immersive games, I presented an ECD-inspired idea re: • Specifying competencies to be acquired from the game • Defining EMs that link game behaviors to competencies • Updating the learner model regularly Using ECD, stealth assessment, and automated data collection and analysis tools is meant to collect valid evidence of students’ emerging competencies, and reduce teachers’ workload allowing them to focus on fostering student learning. Next steps: adapt content (feedback, difficulty levels, scaffolding, etc.) in game to fit current needs of player, test plug ’n play of models, evaluate learning effects, etc.
  26. 26. Thank you! Questions? Email: vshute@fsu.eduWebsite: http://www.myweb.fsu.edu/vshute
  27. 27. Really Old Days In the Stone Age, kids learned how to start a fire with sticks. Hunting skills were important too. The world in which they lived was fairly simple and flat. Return