This document discusses two methods for assessing student engagement and affect in educational games: quantitative field observations and automated detectors. Quantitative field observations involve experts observing students' engagement and affect through peripheral vision during gameplay and noting behaviors like collaboration, boredom, and frustration. Automated detectors use models to assess engagement and affect in real-time from student behavior within software, without sensors. The author's lab has used these methods to compare engagement between games and intelligent tutoring systems, and to study how social behavior influences affect dynamics in games. Both methods provide ecologically valid assessments of student disengagement and experiences of specific emotions during learning.

1. Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games Ryan S.J.d. Baker Assistant Professor of Psychology, Learning Science, and Computer Science Worcester Polytechnic Institute

2. Many ways to assess engagement and affect 2 I’ll discuss two methods our lab uses

3. Quantitative Field Observations (Expert Judgments) Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting Conducted using peripheral vision/side glances Good inter-rater reliability: k 0.6-0.8 Include engaged behaviors (collaboration with other students) and disengaged behaviors (off-task behavior) Include positive affect (delight, engaged concentration) and negative affect (boredom, frustration) Ecologically valid assessments of how much and when Students are disengaged Students experience specific affect 3

4. Automated Detectors Models that assess student engagement and affect in real-time or retrospectively from behavior within software In our approach, no sensors used Improves scalability – lots of data being automatically collected these days Reduces predictive power for some affective states,relative to sensor-based detectors Successful at detecting disengaged behaviors such as off-task behavior, carelessness, gaming the system Successful at detecting engaged concentration and boredom in two learning systems Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008) Used in interventions that improve learning outcomes(Baker et al., 2006) 4

5. Ongoing Project (NSF PSLC) 5 Comparative Analysis Completed for intelligent tutors; in process for serious games Quantitative Field Observation Detector Development Affect Basic Research

6. Use in Research 6

7. How does student affectdiffer between games and ITS?(Rodrigo & Baker, 2011) 7

8. Aplusix .vs. MathBlaster 8 Matched mathematical content between systems Student affect assessed using quantitative field observations with real students in real classrooms

9. Interesting differences in affect 9 Proportions of each affective state shown

10. How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review) 10

11. 11 Chemistry Game (Yaron et al., 2010) Students compete to be first to identify a substance chosen by their opponent Student affect assessed using quantitative field observations with real students in real classrooms

12. Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010) 12 Gaming the System Bored Confused

13. With Social Behavior(Baker, Moore, et al., under review) 13 Gaming the System Off-Task Behavior Bored On-Task Conversation Confused

14. Bottom-Line Field observations and detectors are powerful tools For assessing and understanding student engagement and affect during learning Including in serious games 14