Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games

253 views
213 views

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
253
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games

  1. 1. Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games<br />Ryan S.J.d. Baker<br />Assistant Professor of Psychology, Learning Science, and Computer Science<br />Worcester Polytechnic Institute<br />
  2. 2. Many ways to assess engagement and affect<br />2<br />I’ll discuss two methods our lab uses<br />
  3. 3. Quantitative Field Observations (Expert Judgments)<br />Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting<br />Conducted using peripheral vision/side glances<br />Good inter-rater reliability: k 0.6-0.8<br />Include engaged behaviors (collaboration with other students) and disengaged behaviors (off-task behavior)<br />Include positive affect (delight, engaged concentration) and negative affect (boredom, frustration)<br />Ecologically valid assessments of how much and when <br />Students are disengaged<br />Students experience specific affect<br />3<br />
  4. 4. Automated Detectors<br />Models that assess student engagement and affect in real-time or retrospectively from behavior within software<br />In our approach, no sensors used<br />Improves scalability – lots of data being automatically collected these days<br />Reduces predictive power for some affective states,relative to sensor-based detectors<br />Successful at detecting disengaged behaviors such as off-task behavior, carelessness, gaming the system<br />Successful at detecting engaged concentration and boredom in two learning systems<br />Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008)<br />Used in interventions that improve learning outcomes(Baker et al., 2006)<br />4<br />
  5. 5. Ongoing Project (NSF PSLC)<br />5<br />Comparative Analysis<br />Completed for intelligent tutors; in process for serious games<br />Quantitative Field Observation<br />Detector Development<br />Affect Basic Research<br />
  6. 6. Use in Research<br />6<br />
  7. 7. How does student affectdiffer between games and ITS?(Rodrigo & Baker, 2011)<br />7<br />
  8. 8. Aplusix .vs. MathBlaster<br />8<br />Matched mathematical content between systems<br />Student affect assessed using quantitative field observations<br />with real students in real classrooms<br />
  9. 9. Interesting differences in affect<br />9<br />Proportions of each affective state shown<br />
  10. 10. How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review)<br />10<br />
  11. 11. 11<br />Chemistry Game (Yaron et al., 2010)<br />Students compete to be first to identify a substance chosen by their opponent<br />Student affect assessed using quantitative field observations<br />with real students in real classrooms<br />
  12. 12. Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010)<br />12<br />Gaming the System<br />Bored<br />Confused<br />
  13. 13. With Social Behavior(Baker, Moore, et al., under review)<br />13<br />Gaming the System<br />Off-Task Behavior<br />Bored<br />On-Task Conversation<br />Confused<br />
  14. 14. Bottom-Line<br />Field observations and detectors are powerful tools <br />For assessing and understanding student engagement and affect during learning<br />Including in serious games<br />14<br />

×