An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators


Published on

Slides from my presentation at EC-TEL 2012 in Saarbrucken, Germany, 18th September
This paper elaborates on the evaluation of a Metacognitive Scaffolding Service (MSS), which has been integrated into an already existing and mature medical training simulator. The MSS is envisioned to facilitate self- regulated learning (SRL) through thinking prompts and appropriate learning hints enhancing the use of metacognitive strategies. The MSS is developed in the European ImREAL (Immersive Reflective Experience-based Adaptive Learning) project that aims to augment simulated learning environments throughout services that are decoupled from the simulation itself. Results comparing a baseline evaluation of the ‘pure’ simulator (N=131) and a first user trial including the MSS (N=143) are presented. The findings indicate a positive effect on learning motivation and perceived performance with consistently good usability. The MSS and simulator are perceived as an entity by medical students involved in the study. Further steps of development are discussed and outlined.

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Flow / fidelity???
  • An Initial Evaluation of Metacognitive Scaffolding for Experiential Training Simulators

    1. 1. An Initial Evaluation of Metacognitive Scaffolding for Experiential Training SimulatorsMarcel Berthold, Adam Moore, Christina Steiner, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly, Gary Donohoe, Gordon Power, Owen Conlan
    2. 2. Overview• Who? • ImREAL Project • Evaluating Self-Regulated Learning and Metacognitive Scaffolding in Experiential Simulator• How? • Cross cohort comparison • Baseline usage of sim vs scaffolded usage • Initial – no cross correlation / matching • Third year medical students• What (did we find out)? • Performance • SRL behaviour • Scaffolding encounters & effects • Analysis of reflection text New! (Not in paper)
    3. 3. Rationale of ImREAL• • ’ ’• ‘ ’ ‘ ’ •
    4. 4. What do we want to know?1. Is self-regulated learning supported?2. Does the simulator augmentation through the service lead to better learning performance?3. Does the simulator augmentation through the service increase motivation?4. Is the service well integrated in the simulation and learning experience?
    5. 5. Self-Regulated Learning (SRL) • SRL is composed of three cyclic learning phases: Forethought, Learning, Reflection (Zimmermann, 2002) • Good SR learners use appropriate learning strategies and techniques • Good SR learners achieve better learning results and are more motivated to learn (Zimmerman, 2002; Veenmann, 2011)SRL@ET workshop
    6. 6. Pedagogical framework inImREAL• Integrated SRL cycles + Peer experience (Hetzner et al., 2011)
    7. 7. Simulator• Dialogic• Training for diagnostic interviews for psychiatric patients
    8. 8. Reflection Elicitation / Scaffolding• Simulator has pre-existing reflection tool• Can be triggered by learner• Prompts to reflect also added by instructional designer• In practice mode, cohort 2 students had ImREAL Metacognitive Scaffolding Service (MSS)
    9. 9. Scaffolding – Thinking Prompts Phase 1 Trial
    10. 10. Cohorts• Approximately 140 (cohort 1 = 131 / cohort 2 = 143) medical students, on average 21 years old (40% male vs. 60% female, 80% Irish). They performed the simulation as part of their third year medical training at Trinity College, Dublin (cohort 1 in 2011, cohort 2 in 2012).• Cohort 2 additionally has scaffolding prompts in practice mode only• Experience • No experience with ETU simulator • Experienced with interviews (97 %) • limited experience with interviewing psychiatric patients (15 %)
    11. 11. Methods Assessment Log-Data Have you set a learning goal? Questionnaire Reflection Data postings
    12. 12. Methods: Evaluation approach in ImREAL (First User trail)• Project evaluation question o Can SRL be enhanced through Metacognitive Scaffolding Services (MSS)?• Formative evaluation approach• Research foci of ImREAL MSS Integrated ImREAL services• Investigate: Impact on SRL reports, behaviour, qualitative feedback
    13. 13. Results: SRL• No change in SRL compared to baseline• Highest usage of elaboration strategies SRL strategies profil 100 90 80 70 60 50 % 40 30 20 10 0 Self- Time Achievement Memorizing Elaboration Organisation Planning Internal Effort monitoring management motivation strategies strategies strategies strategies attribution strategies strategies strategies strategies First user trial 57.65 70.16 62.68 59.14 60.01 61.22 62.40 59.11 55.78 Baseline Evaluation 54.07 69.80 62.24 58.24 60.91 59.93 60.44 56.87 55.30 SRL strategy scales t20=3.34, p=.003
    14. 14. Results: Learning Experience• Scores are above centre line of the rating scale• Mentioned that prompts were not always presented at appropriate time Learning Experience MSS 5 4 Ratings 3.42 3.33 3.39 3.38 3.45 3.42 3.12 3.27 3.26 3.09 2.97 3 2 1 Items
    15. 15. Results: Motivation• Students were motivated • To perform simulation • Learn about interviews • To apply skills learnt in simulation in real world• State motivation after simulation higher in 1st user trial compared to baseline Motivation 4 3.35 3 Ratings 2.48 2 1 0 Trial Baseline Evaluation First User Trial t118.47=-8.64, p<.001
    16. 16. Results: Scaffolding• Comparison of expected and empirical MSS prompt distribution• More scaffolds in Information Management phase• Less scaffolds on reflection Expected vs. Empirical Distribution of MCSs 800 700 Frequencies 600 500 400 300 200 100 0 Informantion Planning Monitoring Debugging Reflection Phases Management Expected distribution 400.20 571.71 400.20 301.00 343.03 Empirical distribution 469.00 752.00 425.00 301.00 54.00 χ2(4,0.95)= 314.55, p<.001
    17. 17. Methods: Coding reflective text• Positional – time, date, simulator, etc. • “Rapport and Initial inquiry have been completed. Must now concentrate on moving the interview forward”• Technical – material addressing the system, questions or underlying model • Great concept. Has a lot of potential, will be much better when more points to consider and points of information are added to the overview that accurately reflect performance.• Notes – observations of the patient • “Px. unable to quantify length of period she has been like this for.• Reflections – reflective text • it is quite important to deal with the patient with empathy to make them comfortable and also to try and illicit the cause of the bout of depression, in this case. I felt I did not do too bad as far as expressing empathy is concerned, allowing the patient to open up and try and formulate a management plan. More experience in history taking will be the key
    18. 18. Results: Reflective Text• Corpus 1 (from cohort 1): had 358 unique text entries from 81 unique UserIDs giving an average of 4.41 entries per unique UserID • Engaged percentage = 57.9%• Corpus 2 (from cohort 2): had 107 unique text entries into the note- taking tool from 50 unique UserIDs giving an average of 2.14 entries and 3418 views of the MST, resulting in 28 entries, 19 with text from 8 unique UserIDs • Engaged percentage = 35.7%
    19. 19. Results: Reflective Text• percentage breakdown of each type of content within the note-taking tool across the two cohorts: Position Technical Notes ReflectionCohort 1 15 33 25 69Cohort 2 17 57 16 66
    20. 20. Discussion• SRL • No correlation of log-data and SRL reports were observed – needs to be investigated why? • No changes in SRL – long-term process• MSS • Learners seem to need more assistance in effectively processing information by hints to use more organizational, elaborative, summarizing or selective learning strategies • Are they more confident in the reflection phase and so pass the offer of scaffolds• Time to engage • Less availability of simulators in cohort 2 • Less lab time & shorter overall period • Overall engagement and motivation increased • Higher percentage of time being scaffolded
    21. 21. Lessons Learnt• Provide MS at appropriate times• Keep learners longer in simulation• Provide additional services to promote SRL
    22. 22. What did we find out?1. Is self-regulated learning supported? • Yes • Is it improved . . . . Maybe2. Does the simulator augmentation through the service lead to better learning performance? • No! But . . . . Yes!3. Does the simulator augmentation through the service increase motivation? • Yes4. Is the service well integrated in the simulation and learning experience? • Yes
    23. 23. Further Work• Longitudinal study • SRL changes slow • Integration of changes into behaviour• Fuller analysis of competency • Metacognitive / SRL as well as that trained by the simulator • Surfacing of reflection / competency – social?• Affect support • Encouragement / engagement / motivation • Please take our survey!!!
    24. 24. Acknowledgements• The research leading to these results has received funding from the European Communitys Seventh Framework Program (FP7/2007-2013) under grant agreement no 257831 (ImREAL project) and could not be realized without the close collaboration between all ImREAL partners.
    25. 25. Thank-you! Case sensitive!!! /