Presentation at the LASER Institute 2022

1. Multimodal
Learning Analytics
Xavier Ochoa
New York University
Rationale, Process,
Example and Direction

2. 2
Get the Slides
https://tinyurl.com/mmlaLASER

3. Rationale
01 Why we need MmLA

4. Learning analytics is the
measurement, collection, analysis
and reporting of data about
learners and their contexts, for
purposes of understanding and
optimizing learning and the
environments in which it occurs.

5. 5
Images in public domain. Obtained from Wikimedia Foundation.

6. 6
Images in public domain. Obtained from Wikimedia Foundation.

7. 7
Images in public domain. Obtained from Wikimedia Foundation.

8. Abundant digital
traces of
student actions
Data is there and it is ready to be used

9. Streetlight Effect
in Learning Analytics
Image taken from The Streetlight Effect—Is There Light at the End of the Tunnel?
Åke Lernmark, Diabetes. Apr 2015, 64 (4) 1105-1107

10. We are reaching the
limits of what
clickstream data can
tell us
Maybe it is time to look in a
different place

11. Multimodal Learning
Analytics
MmLA
Going beyond digital traces

12. Process
02 How we do MmLA

13. Multimodal
Learning
Analytics
Analyzing learning traces in
multiple modalities to support
learning process stakeholders

14. The machine
should be closer to
the human
Sense like a human
Perceive like a human
Think like a human
Compute like a machine
Communicate with humans

15. 01
Persiving
Feature Extraction
Analyzing
Fusion
Sensing
Recording
02 03
Feedback
Multimodal
Interfaces
04

16. Sense like a human
Sense the world through multiple modalities

18. Sensors
Cameras
Depth Camera
Eye Tracker
Microphones / Arrays
Position Tracker
Pulse / GSR
Digital Pens
EEG

19. Sensors

20. Sensors
+ Directional audio

21. Sensors

22. Software

23. Perceive like a human
Detect learning traces in those modalites

25. Video – Posture - OpenPose
https://colab.research.google.com/drive/1Ap3MH00-0GGXM2td3twXP7pUp7GIXlO2?usp=sharing

26. Video – Facial Landmarks – MediaPipe
https://colab.research.google.com/drive/1hwMhkRlf70cTnD_nlVCnMwyGIqTXXuY8?usp=sharing

27. Video – Hands – MediaPipe
https://colab.research.google.com/drive/1ZudZqefYiovVBsLuQpCa46MJ1gUQbusP

28. Video – Holistic - MediaPipe
https://colab.research.google.com/drive/18I5T7RJdRG3L6ZigiZ4zYTgDodpsdT5i?usp=sharing

29. Video – Object Detection - YOLO
https://colab.research.google.com/drive/1yKaFjhbbyN8eKnPhjtr7BM_My1apSeBI?usp=sharing

30. Audio – Speech Recognition – NVIDIA
Nemo
https://colab.research.google.com/drive/1k9-q-OgBwjYbuLbNWXkJ40rGt5flztPn?usp=sharing

31. Audio – Speech Diarization – Pyannote
https://colab.research.google.com/drive/1_A5C8xNW2TDDyoPkOO3Fcw-uuCF0lO_c?usp=sharing

32. Feature Extraction

33. Feature Extraction

34. Feature Extraction

35. EZ-MMLA: https://mmla.gse.harvard.edu/
Try it Yourself

36. Compute like a
machine
Being able to handle and simplify large amounts of data

37. Fusion

38. Communicate with
humans
Use multimodal ways to interact

39. Feedback

40. Collaboration
Constructs
Traces
Behavioral
Indicators
Modalities
Sensors
Feature
Extraction
Feedback
Analytics
Collaboration
Feedback
THEORY
TOOL
PEDAGOGICAL
TECHNOLOGICAL
+ Independent + Dependent
CONTEXT

41. Examples
03 MmLA in the Wild

42. RAP System
(Oral Presentation Automatic Feedback)
Ochoa, et al. The RAP System: Automatic Feedback of Oral Presentation Skills using Multimodal Analysis and Low-Cost Sensors
Learning Analytics and Knowledge Conference 2018

43. RAP System
(Oral Presentation Automatic Feedback)
Ochoa, et al. The RAP System: Automatic Feedback of Oral Presentation Skills using Multimodal Analysis and Low-Cost Sensors
Learning Analytics and Knowledge Conference 2018

44. Multimodal Feedback

45. OpenOPAF
Open Source Oral Presentation Feedback

51. Directions
04 Where MmLA is going

52. Multimodal Learning
Analytics is not a
just a sub-field of of
Learning Analytics
It is its natural evolution

53. Tendency to use
non-intrusive
sensors
Make it scalable
and practical

54. Using Artificial
Sensing
Not replacing humans,
but augmenting them

55. MmLA could help us
bridge Learning
Analytics with more
traditional
Educational
Research
Focus on the same learning contexts

56. You could be a
MmLA researcher
and practitioner!
Join SoLAR
CrossMmLA SIG

57. CREDITS: This presentation template was
created by Slidesgo, including icons by Flaticon,
and infographics & images by Freepik.
THANKS!
Xavier Ochoa
xavier.ochoa@nyu.edu
http://wp.nyu.edu/xavier_ochoa
Twitter: @xaoch