Affective Metacognitive Scaffolding and User Model
Augmentation for Experiental Training Simulators:
A Follow-up Study
Imm...
Overview
- Intro
- ImREAL
- Challenges and Approach
- ETU Simulator
- Simulator Augmentation
- Empirical study
- Methods a...
Intro
- Experiential Training Simulators
- For engaging and motivating learning experiences
- To create situational contex...
4
ImREAL: Challenge
Simulated environment
 Augment existing training simulators by...
 Real-world activity modelling
 E...
Experiential Training Simulator:
EmpowerTheUser - ETU
Medical Interview Training:
 Two Psychiatric Interview Scenarios  ...
ETU: Assess mode
Performance Prediction and Scores
EC-TEL 2013 6
Interview steps (Process)
Interview skills
ETU scores
ETU: Practice mode
EC-TEL 2013 7
Visible Decision scores
Coach: feedback
on decision path
Thinking prompts and scaffolding service
ETU: Practice mode
Meta-cognitive scaffolding (MSS)
EC-TEL 2013 8
ETU: Practice mode
Meta-cognitive scaffolding (MSS)
EC-TEL 2013 9
Prompts are based on MAI-ROC (Schraw, 194)
Prompts are m...
ETU: Practice mode:
Affective Metacognitive Scaffolding
EC-TEL 2013 10
SAM:
Smiley Affect Measurement technology
• Explici...
Empirical Study: Research Questions
• Simulator Augmentation:
• Is the simulation relevant for users?
• Are the services w...
Longterm Methodology:
Berthold et al., EC-TEL 2012
Participants
UT1: 143 TCD medical students, 22.7 years, 54.8% male,
97%...
Methods: Procedure and Measures
EC-TEL 2013 14
Cohort survey (N=95)
Training in the simulator
(N=143/152)
Post-sim survey
...
Results: Log Data
Simulator Usage & Training Times
15
Usage (solid lines) and mean training times
(dashed) of assess and p...
Results
Which aspects are (not) effected?
No differences in ...
 Integration & perception of thinking prompts
 Self-regu...
Results
Relevance for Real-World Interviews
 UT2 students feel better prepared for real-world interviews
 Perception of ...
Results
Learning Experience
Flow (Flow short scale; Rheinberg et al., 2003)
 increase in overall flow, technical flow & 7...
Results
Learning Experience
Workload: Nasa-TLX (Hart & Staveland,1988)
 Lower overall workload in UT2
 UT2 users felt le...
Results
Self-Regulated Learning
Reflection notes and AMSS prompts
• 1092 triggers provided (M=15/student) in UT2, 2001 in ...
Results: Log Data
ETU Scores
ETU: predicted and achieved scores in UT1 and UT2
21
 Higher scores in UT2 for
introduction,...
Results
Affect
 Affective state reported after simulation: Total affective state scales
(TASS, Nicolescu et al., 2006 )
...
Results
Social Networks
Usage: 81% use facebook, 20% twitter, < 5% Flickr, MySpace, linkedIn
 Nobody (of 152) provided SN...
Results
Summary
• Changes from pure (BL) to augmented (UT) simulator
 ↑ Perceived relevance, usability, state motivation
...
Conclusions & Challenges
• Metacognitive scaffolding supports learners on different levels
• Evaluation of user model augm...
Conclusions & Challenges
• (Non)-invasiveness vs. (un)-visibility
 SAM now with triggers and presented in main window
EC-...
Thanks for listening!
EC-TEL 2013 27
Upcoming SlideShare
Loading in …5
×

Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators: A Follow-up Study

624 views
505 views

Published on

The ImREAL project (http://www.imreal-project.eu) is researching how to meaningfully augment and extend existing experiential training simulators. The services developed support self-regulated, goal-, and application-oriented learning in adult training. We present results from a study evaluating a medical interview training simulator that has been augmented by an affective metacognitive scaffolding service and by user modelling exploiting social digital traces. Data from 152 medical students participating in this user trial were compared to the results of a prior trial on an earlier technology version. Findings show that students perceived the learning simulator positively and that the enhanced simulator led to increased feelings of success, less frustration, higher technical flow, and more reflection on learning. Interestingly, this cohort of users proved reluctant to provide their open social IDs to enrich their user models.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
624
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators: A Follow-up Study

  1. 1. Affective Metacognitive Scaffolding and User Model Augmentation for Experiental Training Simulators: A Follow-up Study Immersive Reflective Experience-based Adaptive Learning 8th European Conference on Technology Enhanced Learning (EC-TEL 2013) Gudrun Wesiak, Adam Moore, Christina M. Steiner, Claudia Hauff, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly, Gary Donohoe, Gordon Power, & Owen Conlan
  2. 2. Overview - Intro - ImREAL - Challenges and Approach - ETU Simulator - Simulator Augmentation - Empirical study - Methods and results - Discussion & Conclusions EC-TEL 2013 2
  3. 3. Intro - Experiential Training Simulators - For engaging and motivating learning experiences - To create situational contexts to practice job-relevant skills - Adult learning (Andragogy) - Self-directed, experienced-based, goal-oriented, intrinsically motivating, relevancy-driven (Knowles, 1984) - Influenced by learner’s physical, digital and social world - Should adapt to learner’s previous experiences, knowledge, …. EC-TEL 2013 3
  4. 4. 4 ImREAL: Challenge Simulated environment  Augment existing training simulators by...  Real-world activity modelling  Enriched user modelling  Affective-metacognitive scaffolding service  Develop services for different simulators Real-world experience Close the gap Medical interview training Fig. 2. Example learner interaction screen – the simulated situation is in the Dinner episode where the host has to decide about ordering food for his business guests. The simulated scenario includes four episodes: Greetings (situations embed arriving on time, different norms about greetings, first impression, and use of body language), Dinner (situations embed use of body language and different preferences about food Business simulation & Buddy program Simulated world Real world Forethought Reflection
  5. 5. Experiential Training Simulator: EmpowerTheUser - ETU Medical Interview Training:  Two Psychiatric Interview Scenarios  Mania & Depression  Basic idea: Watch patient video and select appropriate answer  2Training modes: Assess mode and practice mode EC-TEL 2013 5
  6. 6. ETU: Assess mode Performance Prediction and Scores EC-TEL 2013 6 Interview steps (Process) Interview skills ETU scores
  7. 7. ETU: Practice mode EC-TEL 2013 7 Visible Decision scores Coach: feedback on decision path
  8. 8. Thinking prompts and scaffolding service ETU: Practice mode Meta-cognitive scaffolding (MSS) EC-TEL 2013 8
  9. 9. ETU: Practice mode Meta-cognitive scaffolding (MSS) EC-TEL 2013 9 Prompts are based on MAI-ROC (Schraw, 194) Prompts are mapped to different interview situations
  10. 10. ETU: Practice mode: Affective Metacognitive Scaffolding EC-TEL 2013 10 SAM: Smiley Affect Measurement technology • Explicit assessment of current emotional state with SAM • Selection of displayed prompts influenced by users’ emotional state • Sentiment analysis of digital traces (U-SEM) • Mappings of prompts to situations as weighted list
  11. 11. Empirical Study: Research Questions • Simulator Augmentation: • Is the simulation relevant for users? • Are the services well integrated? • Digital Traces: • Usage, privacy & trust EC-TEL 2013 12 • AMSS: Are there effect on… • self-regulated learning? • users’ learning experience? • learning performance? • motivation/affect?
  12. 12. Longterm Methodology: Berthold et al., EC-TEL 2012 Participants UT1: 143 TCD medical students, 22.7 years, 54.8% male, 97% real world interview experience (15.4% psychiatric) UT2: 152 TCD medical students, 22.81 years, 50% male, 99% real-world interview experience (25.6% psychiatric) • 2011 • Pure Simulator Baseline • 2012 • ETU with MSS User Trial 1 • 2013 • AMSS, SA M, U-SEM User Trial 2 Methods EC-TEL 2013 13
  13. 13. Methods: Procedure and Measures EC-TEL 2013 14 Cohort survey (N=95) Training in the simulator (N=143/152) Post-sim survey (N=39/ 40) Real-world interview Post-int survey (N=7/19) • Demographics • Social Network use & attitudes • Log-data (Usage, scores) 2 scenarios, 2 modes • Real-World Relevance • SRL-strategies • Motivation & Affect • Learning Experience • Learning Performance • Real-world relevance • Motivation • Fb on modes, AMSS, SAM, SN
  14. 14. Results: Log Data Simulator Usage & Training Times 15 Usage (solid lines) and mean training times (dashed) of assess and practice mode  UT1: very few users for Mania  UT2: short times for Depression Assess vs. Practice:  Assess: more users and longer times UT1 vs. UT2:  Same no. of users, but increased overall training times (both modes)
  15. 15. Results Which aspects are (not) effected? No differences in ...  Integration & perception of thinking prompts  Self-regulated learning strategies (QSRL)  Motivation to improve skills, to perform good, to apply what has been learned (post-sim & post-int) No difference between UT1 & UT2, but change from Baseline  Perceived relevance of simulation  Usability (SUS)  Overall motivation  Overall workload (decreased!) Deacrease from UT1 to UT2 Motivation to learn more about interviewing (post-sim) EC-TEL 2013 16 Baseline UT1/UT2 UT1=UT2
  16. 16. Results Relevance for Real-World Interviews  UT2 students feel better prepared for real-world interviews  Perception of relevancy & preparedness unchanged after real-world interview 17
  17. 17. Results Learning Experience Flow (Flow short scale; Rheinberg et al., 2003)  increase in overall flow, technical flow & 7 single items i.e. stronger involvement into the task, greater feeling of smoothness and fluency, smooth interplay between software components 18 Flow score means on item basis (left fig.) and overall means (and SE) for the FKS, worry, and tech scales (right fig.)
  18. 18. Results Learning Experience Workload: Nasa-TLX (Hart & Staveland,1988)  Lower overall workload in UT2  UT2 users felt less frustrated & had stronger feelings of success EC-TEL 2013 19 Mean scores (and SE) from the NASA-TX workload scales for the two user trials.
  19. 19. Results Self-Regulated Learning Reflection notes and AMSS prompts • 1092 triggers provided (M=15/student) in UT2, 2001 in UT1 • MAI-prompts: 88.4% of 69 answers positive • Increase in proportion of reflective notes EC-TEL 2013 20 Table 1. Rounded percentages of content types for entries from the note-taking tools. Note: Entries can be coded to more than one type, thus percentages may exceed 100. Users Text entries Position Technical Notes Reflection UT1 50 107 17 57 16 66 UT2 35 86 1 13 7 93
  20. 20. Results: Log Data ETU Scores ETU: predicted and achieved scores in UT1 and UT2 21  Higher scores in UT2 for introduction, eliciting and planning phase  Mainly underestimation of scores in UT2
  21. 21. Results Affect  Affective state reported after simulation: Total affective state scales (TASS, Nicolescu et al., 2006 )  Affect during simulation:  SAM was displayed 352 times (2.23 times / user) – used by 1 person 22 Mean TASS scores on 7 subscales plus overall mean score across all scales. NO Yes
  22. 22. Results Social Networks Usage: 81% use facebook, 20% twitter, < 5% Flickr, MySpace, linkedIn  Nobody (of 152) provided SN-ID in simulator (13 did in cohort survey; 11 fb)  For detailed results: see Moore, 2013 (Umap; LBR) Reasons: EC-TEL 2013
  23. 23. Results Summary • Changes from pure (BL) to augmented (UT) simulator  ↑ Perceived relevance, usability, state motivation  ↓ Overall workload • Changes from MSS (UT1) to AMSS (UT2) simulator  Relevancy: ↑ stronger feeling of preparedness  SRL: ↑ effort (times), feeling of success, reflective notes;  Performance: ↑ ETU-Scores  Learning Experience: ↑ flow ↓ decrease in frustration  Motivation: ↓ to learn more about interviewing EC-TEL 2013 24
  24. 24. Conclusions & Challenges • Metacognitive scaffolding supports learners on different levels • Evaluation of user model augmentation  Low user acceptance for exploiting their social digital traces  Need to increase willingness to share SN-IDs • Study learning activity rather than purely subjective reports  Analysing learners’ real activities with the simulators • (Non)-invasiveness vs. (un)-visibility  SAM, thinking prompts EC-TEL 2013 25
  25. 25. Conclusions & Challenges • (Non)-invasiveness vs. (un)-visibility  SAM now with triggers and presented in main window EC-TEL 2013 26 NO YES
  26. 26. Thanks for listening! EC-TEL 2013 27

×