Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games
Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games<br />Ryan S.J.d. Baker<br />Assistant Professor of Psychology, Learning Science, and Computer Science<br />Worcester Polytechnic Institute<br />
Many ways to assess engagement and affect<br />2<br />I’ll discuss two methods our lab uses<br />
Quantitative Field Observations (Expert Judgments)<br />Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting<br />Conducted using peripheral vision/side glances<br />Good inter-rater reliability: k 0.6-0.8<br />Include engaged behaviors (collaboration with other students) and disengaged behaviors (off-task behavior)<br />Include positive affect (delight, engaged concentration) and negative affect (boredom, frustration)<br />Ecologically valid assessments of how much and when <br />Students are disengaged<br />Students experience specific affect<br />3<br />
Automated Detectors<br />Models that assess student engagement and affect in real-time or retrospectively from behavior within software<br />In our approach, no sensors used<br />Improves scalability – lots of data being automatically collected these days<br />Reduces predictive power for some affective states,relative to sensor-based detectors<br />Successful at detecting disengaged behaviors such as off-task behavior, carelessness, gaming the system<br />Successful at detecting engaged concentration and boredom in two learning systems<br />Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008)<br />Used in interventions that improve learning outcomes(Baker et al., 2006)<br />4<br />
Ongoing Project (NSF PSLC)<br />5<br />Comparative Analysis<br />Completed for intelligent tutors; in process for serious games<br />Quantitative Field Observation<br />Detector Development<br />Affect Basic Research<br />
How does student affectdiffer between games and ITS?(Rodrigo & Baker, 2011)<br />7<br />
Aplusix .vs. MathBlaster<br />8<br />Matched mathematical content between systems<br />Student affect assessed using quantitative field observations<br />with real students in real classrooms<br />
Interesting differences in affect<br />9<br />Proportions of each affective state shown<br />
How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review)<br />10<br />
11<br />Chemistry Game (Yaron et al., 2010)<br />Students compete to be first to identify a substance chosen by their opponent<br />Student affect assessed using quantitative field observations<br />with real students in real classrooms<br />
Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010)<br />12<br />Gaming the System<br />Bored<br />Confused<br />
With Social Behavior(Baker, Moore, et al., under review)<br />13<br />Gaming the System<br />Off-Task Behavior<br />Bored<br />On-Task Conversation<br />Confused<br />
Bottom-Line<br />Field observations and detectors are powerful tools <br />For assessing and understanding student engagement and affect during learning<br />Including in serious games<br />14<br />
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.