Presentation given on the February 1st, 2022 at the "Brown Bag" presentation series organised by the Faculty of NYU Educational Communication and Technology which is part of the Steinhardt School of Culture, Education, and Human Development. https://www.ectstudent.info/news-events/brown-bag-dr-daniele-di-mitri Presentation Abstract: The COVID-19 pandemic forced more than 1.6 billion learners out of school, becoming the most challenging disruption ever endured by the global education systems. In many countries, education institutions decided to move their regular activities online, opting for remote teaching as an emergency solution to continue their education. Meanwhile, physical distancing and learning in isolation heavily challenge learners and hinder their study success. There is a compelling need to make education systems more resilient and less vulnerable to future disruptions in such a critical landscape. In particular, we have to reconsider how digital technologies can support online and hybrid teaching. If digital education technologies such as video conferencing tools and learning management systems have improved to make educational resources more available and education more flexible, the modes of interaction they implement remain essentially unnatural for the learner due to a substantial lack of context. Modern sensor-enabled computer systems allow extending the standard human-computer interfaces and facilitate richer multimodal interaction. Furthermore, advances in AI allow interpreting the data collected from multimodal and multi-sensor devices. These insights can be used to support online teaching and learning in isolation with personalised feedback and adaptation through Multimodal Learning Experiences (MLX). This guest lecture elaborates on existing approaches, architectures, and methodologies. I illustrate use cases that employ multimodal learning analytics applications that can shape the online teaching of the future.

1. Restoring Context in Distance Learning with
Artificial Intelligence and Multimodal
Learning Experiences
Guest presentation at the “ECT Brown Bag” at New York University
by Dr Daniele Di Mitri
February, 1st 2022

2. BSc in Computer Science
MSc in Artificial Intelligence
PhD in Learning Analytics
PostDoc Artificial Intelligence
Former IBM Extreme Blue
Research Leader AI in Education
Daniele DI MITRI, PhD
dimitri@dipf.de
Who am I?
Multimodal Learning Experiences 2

3. What is DIPF?
The DIPF | Leibniz Institute for
Research and Information in
Education
Delivers
- empirical educational research
- digital infrastructure and
- targeted knowledge transfer
contributing to coping with
challenges in education.
DIPF is the biggest Think Tank in
Education in Germany
Multimodal Learning Experiences 3
www.edutec.science
EduTec group at the DIPF

4. Disruptions in education
Multimodal Learning Experiences 4

5. Challenging time for education
Multimodal Learning Experiences 5
Challenging for students
Challenging for teachers

6. Traditional education technologies
Traditional education technologies
• limited modalities of interaction
• portable, flexible
» but still constrained by a screen
• learning activities are mainly designed
around the device
• application of knowledge is missing
• lack of contextual information
• lack of authentic practice
Multimodal Learning Experiences 6

7. The Learning-Practice gap
Multimodal Learning Experiences 7
Learning about a skill Practicing a skill
actual learning requires authentic practice
“I want to learn
how to speak in public”

8. Role of context in learning
• Context is pervasive – can be found in
building, spaces
• Context is dynamic – moves with the learner
• Context is social – it is shapes by the other
people
• Context is multimodal – it relates to how the
learner interact with the world
Multimodal Learning Experiences 8

9. Multimodality in Human Interactions
Communication is a two-way process:
Encoding messages
using multiple modalities like textual,
linguistic, spatial (Kress, 2003)
Decoding messages
capturing through the senses and
reasoning about them (Paivio, 1990).
Kress, G. (2003). Literacy in the new media age. Psychology Press Chicago.
Paivio, A. (1990). Mental representations: A dual coding approach. Oxford University Press.
Multimodal Learning Experiences 9

10. Theoretical groundings of Multimodal Learning
• Embodied learning (EL) learning and skill acquisition as grounded in the body and the
environment it is operating (Robbins & Aydede, 2009)
‒ perception takes place via senses, motor activity, and introspection
‒ same principle of cognitive offloading of Dual Coding Theory (Clark & Paivio 1991)
‒ movements, gestures help to free memory resources
‒ better recall rates for training (Kiefer et al., 2015)
• Deliberate practice (DP) individualized training activities specially to improve specific aspects
of an individual’s performance through repetition (Ericsson & Lehmann, 1996).
‒ Takes place with repeated perception-action cycles
‒ conscious, structured practice required for sustained improvement (Ericsson et al 1993)
‒ requires high cognitive load (Rikers et al., 2004)
‒ Integrated EL in DP would distribute the cognitive load
‒ must be done in an authentic setting (Neelen & Kirschner, 2018)
Multimodal Learning Experiences 10

11. Bloom’s Theory: domains of learning & the 2-sigma problem
Multimodal Learning Experiences 11

12. New technological affordances
Multimodal Learning Experiences 12
Multimodal Learning Experiences
+
learning analytics approach
"measurement, collection, analysis and reporting of
data about learners”
data from multimodal and multi-sensor
interfaces
= Multimodal Learning Analytics
a more accurate representation of the
learning process

13. Multimodal Learning Experiences (MLX)
“any digital-enhanced learning activity using more than two modalities within an authentic learning setting”
Multimodal Learning Experiences 13
(1) Sensors (2) Authentic practice (3) Immersive tech
The 3 pillars of MLX

14. Some examples from related research
Benefits of MLX for teaching and
learning
Multimodal Learning Experiences 14

15. Benefit 1) Create richer online learning experiences
Multimodal Learning Experiences 15
ZoomSense Open Source Real-time
Infrastructure for Zoom Augmentation
https://zoomsense.org/
SenseTheClassroom / MMLA Tool for real-time
emotion recognition in video conferencing tools
(Böttger et al. , upcoming)

16. Benefit 2) Support to psychomotor learning
Multimodal Learning Experiences 16
(Schneider et al, 2015; 2019)
Example: Presentation Trainer

17. Benefit 3) Create premium in-presence learning experiences
Multimodal Learning Experiences 17
Example: Edusense
https://github.com/edusense/edusense

18. Designing MLX systems that works (and that fail gracefully)
Engineering MLX systems
Multimodal Learning Experiences 18

19. How can MLX support learning?
Objective of MLX: unobtrusive tracking during deliberate practice
..however…
- Sensor data is messy and has poor semantic value
- Multimodal data introduces multidimensional complexity
- In my PhD thesis, the Multimodal Tutor (Di Mitri, 2020)
- the Five Big challenges
- the Multimodal Learning Analytics Model
- the Multimodal Pipeline
Multimodal Learning Experiences 19
5. Data
Exploitation
User
1. Data
collection
3. Data
Processing
4. Data
Annotation
2. Data
storing
Multimodal
Feedback
Loop

20. Structure of a generic MLX system
Multimodal Learning Experiences 20

21. EEG
Empatica
Myo
Leap
Motion
Data collection & storing challenges
21
Multimodal Learning Experiences
Modalities Sensors Controllers
Multimodal
Learning
Hub
example data from
EMG sensor
(Schneider et
al, 2018)
Multimodal Learning Experiences

22. Data annotation challenge
Multimodal Learning Experiences
22
Multimodal Learning Experiences

23. Real-time feedback architecture (Di Mitri et al., 2021)
Sensors
CPRTutor
C# app
SharpFlow
Python3
TCP
client
TCP
server
Chunk
(1 CC – 0.5 sec)
Classification ML models
ClassRate
ClassRelease
ClassDepth
ArmsLocked
BodyWeight
Feedback
23

24. Positive effect of real-time feedback
error rates drops shortly
after the feedback is fired
24
Multimodal Learning Experiences

25. Edutex architecture for context recognition (Ciordas-Hertel et al., 2021)
Multimodal Learning Experiences 25
Ciordas-Hertel, G.-P.; Rödling, S.; Schneider, J.; Di Mitri, D.; Weidlich, J.; Drachsler, H. Mobile Sensing with Smart Wearables of the Physical Context of Distance
Learning Students to Consider Its Effects on Learning. Sensors 2021, 21, 6649. https://doi.org/10.3390/s21196649

26. Design Based Research
• First Iteration:
‒ Basic requirement analysis
» what skills needs to be practiced? how?
‒ Part-of-task analysis
» to select a small part of the skill
‒ Prototype development
‒ User tests to test feasibility
» user experience, usability test, F-score, etc.
• Second iteration:
‒ Improve prototype
‒ Involve experts
‒ Evaluations to test learning gains might be possible
Multimodal Learning Experiences 26

27. Design learning activities with analytics in mind
Multimodal Learning Experiences 27
Fola2 board game – Learning Analytics for Learning Activity design – www.fola2.com
Schmitz, Marcel; Scheffel, Maren; Bemelmans, Roger; Drachsler, Hendrik (2020): Fellowship Of The Learning Activity –
Learning Analytics 4 Learning Design. https://doi.org/10.25385/zuyd.9884279

28. MLX systems that we created so far
Application use cases
Multimodal Learning Experiences 28

29. Calligraphy Trainer (Limbu et al, 2019)
29
Multimodal Learning Experiences

30. CPR Tutor (Di Mitri et al 2020)
Multimodal Learning Experiences
Lock your
arms!
Use your
body
weight!
Release
the
compression!
*Metronome
sound 110bpm*
Check
compressio
n depth!
Microsoft
Kinect v2
Simpad
Laerdal
ResusciAnne
Myo armband
30
Multimodal Learning Experiences

31. Many other examples
Multimodal Learning Experiences 31
• Salsa Trainer (Romano et al. 2019)
• Tennis Table Tutor (Asyraaf et al., 2021)
• Mode of transportation tracking (Di Mitri et al., 2020)
• Pilates Trainer (Meik et al., 2021)
• Running Trainer (in the context of Milky-PSY project)
• … more to come …

32. Ongoing research: How to embed MLX in the teaching practice?
Multimodal Learning Experiences 32
• AI systems can support adaptive learning in distance learning setting
• Sensors monitor the behaviour & performance
• MLX allow to restore context
• AI provides automatic feedback
Where do teachers fit in?
• Human-AI hybrid adaptation (Holstein, 2020)
• Combine the strengths of humans and AI
‒ Goal information; perceptual, interpretation, decision
augmentation
• Teachers ”in the loop” can intervene and give feedback
• Any time, any place

33. Human-AI hybrid tutoring model
Multimodal Learning Experiences 33

34. Caveats of MLX – and data centric technologies
1. Privacy and ethical concerns – for which purpose are
student’s data collected?
2. Deviate from original learning goals -> Alignment problem
3. Data intensive technologies can deepen social inequalities –
expensive hardware amplifies the digital devide
4. Deepening of social fragmentation and isolation
5. Threaten the student or teacher’s agency (e.g. no forgiveness,
no learn by your mistakes)
What else?
Multimodal Learning Experiences 34

35. Conclusions
• Multimodal Learning Experiences can improve online/teaching learning
1. by creating richer online learning experiences
2. by supporting to psychomotor learning tasks
3. by creating premium in-presence learning experiences
Restoring context, especially when learning in isolation
• However, they also introduce some risks
• We need more research on how to mitigate these risks and maximise benefits
Multimodal Learning Experiences 35

36. Thanks for listening!
Daniele Di Mitri
dimitri@dipf.de
@dimstudi0
36
Multimodal Learning Experiences