Download to read offline
Uploaded byDaniele Di Mitri
The Multimodal Tutor - Presentation PhD defence
The document discusses the concept and implementation of a multimodal tutor that utilizes various sensors and analytics for enhanced learning experiences. It highlights the challenges and frameworks for effective data collection and analysis, particularly in training high-stakes skills like cardiopulmonary resuscitation (CPR). The findings suggest that multimodal tutors can provide real-time feedback, supporting learners even in the absence of human instructors.
More Related Content
The Multimodal Tutor - Presentation PhD defence
- 2. The Multimodal Tutor AdaptiveFeedback from Multimodal Experiences 4th September 2020 Daniele Di Mitri
- 3. How do peoplelearn nowadays? The Multimodal Tutor – Motivation 3
- 4. Multimodal Learning Analytics(MMLA) 4 + Learning Analytics approach "measurement, collection, analysis and reporting of data about learners” The Multimodal Tutor – Motivation data from multimodal and multi-sensor interfaces = a more accurate representation of the learning process
- 5. PhD journey I. Exploratorymission • Learning Pulse study II. Map of Multimodality • Literature survey, conceptual model, the big five challenges III. Preparation of the Navy •Visual inspection Tool, Multimodal Pipeline, mistake detection IV. Conquest mission •Real-time feedback with the CPR Tutor 5The Multimodal Tutor – Structure
- 6. “Are you inthe Flow?” (CHAPTER 1) 6Chapter 1 – Learning Pulse
- 7. What is multimodaldata? (CHAPTER 2) 7Chapter 2 – From Signals to Knowledge
- 8. Titel van depresentatie 8 s Input space Hypothesis space
- 9. Multimodal Learning Analytics Model (CHAPTER 2) Titelvan de presentatie 9
- 10. The Big FiveChallenges (CHAPTER 3) 10 Multimodal Feedback Loop Chapter 3 – The Big Five • Multimodal data is messy, most studies stand at the level of data geology • No clear picture how MMLA can support learning • We identify five big challenges.
- 11. Emotiv Empatica Myo Leap Motion Data collection challenge ModalitiesSensors Controllers Multimodal Learning Hub example of serialisation of Myo data in JSON External research – The Multimodal Learning Hub
- 12. Data annotation challenge(CHAPTER 4) Chapter 4 – Read Between the Lines 12
- 13. The Multimodal Pipeline(CHAPTER 5) 13Chapter 5 – The Multimodal Pipeline • To the reduce data manipulation over- head and focus on the data analysis • A technological framework composed by generic solutions for the big five challenges • The aim is to support researchers in setting up experiments more quickly
- 14. Cardiopulmonary Resuscitation (CPR) WhyCPR? • It’s taught singularly to one learner • It is a highly standardized procedure • It has clear and well-defined criteria to measure the quality • It is a highly relevant skill 14Chapter 6 – Detecting CPR Mistakes
- 15. CPR mistake detection(CHAPTER 6) 15 Indicator Ideal value Compression rate 100 to 120 compr./min Compression depth 5 to 6 cm Compression release 0 - 1 cm Arms position Elbows locked Body position Using body weight assessed by the ResusciAnne manikin not measured by the ResusciAnne manikin Chapter 6 – Detecting CPR Mistakes
- 16. CPR Tutor –1st iteration (CHAPTER 6) Hardware setup: • Microsoft Kinect v2 • Myo armband • Laerdal ResusciAnne manikin Dateset collected: • ~5500 chest compressions from 14 experts • Each CC tagged with 5 classes Trained 5 neural networks to classify CPR mistakes Chapter 6 – Detecting CPR Mistakes 16
- 17. CPR Tutor –2nd iteration (CHAPTER 7) Chapter 7 – Real Time Multimodal Feedback 17 Lock your arms! Use your body weight! Release the compression! *Metronome sound 110bpm* Check compression depth!
- 18. Real-time feedback architecture(CHAPTER 7) 18 Sensors CPRTutor C# app SharpFlow Python3 TCP client TCP server Chunk (1 CC – 0.5 sec) Classification ML models ClassRate ClassRelease ClassDepth ArmsLocked BodyWeight Feedback Chapter 7 – Real Time Multimodal Feedback
- 19. Positive effect offeedback (CHAPTER 7) Chapter 7 – Real Time Multimodal Feedback 19 error rates drops shortly after the feedback is fired
- 20. Conclusions 1) Sensors cannotreason about the data they collect. Machine learning & human annotation can help for automatic reasoning. The Multimodal Tutor – Conclusions 20 2) Multimodal Tutors can support the learners when the human instructor is not available. 3) Multimodal Tutors can help closing the feedback loop using MMLA. 4) The Multimodal Tutor is an example that paves the way to human-AI cooperation.
- 21. Thanks for yourattention.
Editor's Notes
- #3 Dear guests, Dear colleagues, Cara Famiglia, good afternoon and welcome to the final presentation of my PhD project: the Multimodal Tutor, adaptive feedback from multimodal experiences.
- #4 how do we people learn nowadays? In time of covid-19 pandemics, the first thing that comes into our mind is people learn using video conference tools or with distance education or e-learning platforms. While this is for current academic education, there are a lot of learning activities the actually take place not behind the laptop or desktop I'm thinking for example about learning how to play a sport or learning how to cook a new recipe this activities require physical interaction that we call multimodal interactions beyond the keyboard the mouse and keyboard in my PC. My PhD project I focused on how to improve this actions this learning activities not mediated by mouse and keyboard. How can we use computers to support these tasks?
- #5 We think we can do it with MMLA. Mixed Reality devices
- #14 A
- #16 2 mistakes are a
- #19 After lots of trials and error we came up with this architecture