1. The document discusses stealth assessment, an ongoing, unobtrusive form of assessment that can evaluate skills like creativity and problem-solving within digital learning environments. 2. It describes a study that used stealth assessment within a physics game called Physics Playground to evaluate students' physics understanding without interrupting gameplay. 3. The results found the stealth assessment of physics understanding was valid and that adaptive sequencing of levels based on students' skills improved learning outcomes more than linear or free-choice gameplay.

1. Stealth Assessment of
Physics Understanding in
Physics Playground
Seyedahmad Rahimi
Assistant Professor of Educational Technology
School of Teaching and Learning (STL)
Institute of Advanced Learning Technologies (IALT)
eMadrid Conference
June 30th
2021

2. 2
The Problem Stealth Assessment Addresses
• Teach ; Stop ; Test ; Stop ; Teach ; Stop; Test; …
• Low ecological validity—generalizability to real-life situations
• Focus on assessment of learning and late or no feedback
• No or little success in assessing 21st-century skills (e.g., creativity)
• Test anxiety
• …
With advances in technologies, learning sciences, AI, and psychometrics we can have assessments that are:
Ongoing, unobtrusive, for learning, and ecologically valid
Shute (2011)

3. 3
ZPD Theory
Vygotsky, 1978
Flow Theory
Csikszentmihalyi, 1997
Flow
Too easy: Bored
Too hard: Frustrated
Difficulty
Skill Level
Theories That Stealth Assessment Supports

4. 4
PP Stealth Assessment & Adaptive Process
(Shute, 2011)
• Physics Understanding
• Calculus
• Creativity
• Persistence
• Problem-solving
• Systems Thinking
• …

5. 5
(Adapted by Shute, 2011 from Mislevy, Steinberg, & Almond, 2003)
Evidence-centered Design framework (ECD)
Evidence
Identification
Evidence
Accumulation
Assessment Design
Diagnostic Inference
Image from Shute & Rahimi, 2021

6. 6
Key Elements of Stealth Assessment
1. A technology-rich learning environment (e.g., a digital game)
2. ECD to design and develop a stealth assessment
3. Formative assessment and feedback
Test or Measurement
Assessment
(Shute, 2009; 2011)

7. 7

8. 8
Physics Understanding Competency Model

9. 9
Q-Matrix

10. 10
Defining Observables

11. 11
Example for Observables

12. 12
Log sample from Learning Locker

13. 13
Proc 4 Message example from Mongo DB
Almond, Shute, Rahimi, & Tingir, 2020

14. PP’s Big Picture—PP’s Architecture
PP Server
Main DB –
Levels, users,
…
Learning
Locker
Mongo DB
Assessment Server
Proc 4
Mongo DB
Login & Load
User Profile
Gameplay +
Send Statements (observables)
Level Selection
via Adaptivity
Algorithm
Update My
Backpack
Load Student’s
Game Data
PP-Unity POST request
POST request
xAPI
PlayerStart.php
PlayerStats.php
End of Each
Level
POST request
Load
Selected
Level
Evidence
Identification
Evidence
Accumulation
Assessment Engine
Filter
Dongle

15. PP’s Big Picture—PP’s Architecture
PP Server
Main DB –
Levels, users,
…
Learning
Locker
Mongo DB
Assessment Server
Proc 4
Mongo DB
Login & Load
User Profile
Gameplay +
Send Statements (observables)
Level Selection
via Adaptivity
Algorithm
Update My
Backpack
Load Student’s
Game Data
PP-Unity POST request
POST request
xAPI
PlayerStart.php
PlayerStats.php
End of Each
Level
POST request
Load
Selected
Level
Evidence
Identification
Evidence
Accumulation
Assessment Engine
Filter
Dongle

16. 16
A recent study using Physics Playground
263 9th to 11th grade students
Four groups:
Adaptive, Linear, Free Choice,
and Control
~ 5 hours of gameplay in six days
Photo: US Dep. Of Education; ED Games Expo 2020, Washington D.C.

17. 17
Order of the levels in linear PP
• Easy
• Medium
• Hard
N1L
• Easy
• Medium
• Hard
ECT • Easy
• Medium
• Hard
ECD
• Easy
• Medium
• Hard
POM • Easy
• Medium
• Hard
COM
• Easy
• Medium
• Hard
POT • Easy
• Medium
• Hard
Equil
• Easy
• Medium
• Hard
N2L • Easy
• Medium
• Hard
N3L
ü Difficulty indices: Game Mechanics (GM) Difficulty & Physics Understanding Difficulty (PU)

18. 18
PP Adaptivity Algorithm—Simplified
Level’s Topic +
Student’s PL
PL < E Easy Level + LS
Start
E < PL < M Easy Level
M < PL < H Medium Level
PL > H Difficult Level
Change Topic
Yes
No
No
No
Yes
Any
unsolved
level?
Any silver
level?
Play that level
End
Yes
Yes
No
No
Topic
Left?
No
No
Yes
End of the Level
Play the Easiest
Level
Yes
Yes

19. 19
Research Questions:
1. Was the stealth assessment of Physics Understanding Valid?
2. Did adaptive sequencing of game levels improve learning compared to a
linear and free choice version of the game?
3. Were the learning supports used in the game effective?
YES!
YES!
NO!
Photo: US Dep. Of Education; ED Games Expo 2020, Washington D.C.

20. 20
Research Questions:
1. Was the stealth assessment of Physics Understanding Valid?
2. Did adaptive sequencing of game levels improve learning compared to a
linear and free choice version of the game?
3. Were the learning supports used in the game effective?
YES!
YES!
NO!
Photo: US Dep. Of Education; ED Games Expo 2020, Washington D.C.
Overall Stealth Assessment of Physics Understanding and
Pretest (r = .36, p < .01)
Posttest (r = .40, p < .01)
Sub facets of Stealth Assessment of Physics Understanding and
Force and motion estimate (r = .32, p < .01)
Linear momentum estimate (r = .30, p < .01)
Energy estimate (r = .35, p < .01)
Torque estimate (r = .21, p < .01).

21. 21
Photo: US Dep. Of Education; ED Games Expo 2020, Washington D.C.

22. 22
Photo: US Dep. Of Education; ED Games Expo 2020, Washington D.C.

23. 23
NSF & IES PI NSF Co-PI NSF Co-PI IES Co-PI IES Co-PI
SMEs
Current
RAs
Former Post doc 1
Former
RAs
Current Post doc

24. Thank you J !
srahimi@ufl.edu
@sahmadrahimi
My Twitter