The keynote discusses the integration of artificial intelligence and multimodal learning experiences in online education, emphasizing the need for authentic practice and the role of context in learning. It presents the benefits of multimodal learning, including richer online experiences and enhanced psychomotor skill acquisition but also highlights potential risks such as privacy concerns and social inequalities. The speaker concludes that while multimodal learning can significantly improve educational outcomes, further research is required to address associated challenges.

1. Restoring Context in Online Teaching
with Artificial Intelligence and
Multimodal Learning Experiences
Keynote at the SITE Interactive Conference
28th October 2021
by Dr Daniele Di Mitri

2. Disruptions in education
Multimodal Learning Experiences 2

3. Question time: What about you education institution?
Multimodal Learning Experiences 3

4. Context and Multimodality
Background
Multimodal Learning Experiences 4

5. Actual learning requires authentic practice
Multimodal Learning Experiences 5
Learning about a skill Practicing a skill

6. Traditional education technologies
• Traditional desktop based computers offers limited
modalities of interaction mainly audio/video
• Mobile devices offer more modalities but remain
constraint to small screen
• Learning activities are mainly designed around the
device
• Gap between knowledge acquisition and knowledge
application
• Lack of authentic and deliberate practice
Multimodal Learning Experiences 6

7. Role of context in learning
- Context is pervasive – can be found in building, spaces
- Context is dynamic – moves with the learner
- Context is social – it is shapes by the other people
- Context is multimodal – it relates to how the learner
interact with the world
Multimodal Learning Experiences 7

8. Theoretical groundings of Multimodal Learning
• Embodied learning (EL) learning and skill acquisition as grounded in the body and the
environment it is operating (Robbins & Aydede, 2009)
‒ perception via senses, motor activity, and introspection
‒ same principle of cognitive offloading of Dual Coding Theory (Clark & Paivio 1991)
‒ movements, gestures help to free memory resources
‒ better recall rates for training (Kiefer et al., 2015)
• Deliberate practice (DP) learning
‒ repeated perception-action cycles
‒ conscious, structured practice required for sustained improvement (Ericsson et al 1993)
‒ requires high cognitive load (Rikers et al., 2004)
‒ Integrated EL in DP would distribute the cognitive load
‒ must be done in an authentic setting (Neelen & Kirschner, 2018)
Multimodal Learning Experiences 8

9. Bloom’s theory: domains of learning & the 2-sigma problem
Multimodal Learning Experiences 9

10. New technological affordances
+
learning analytics approach
"measurement, collection, analysis and
reporting of data about learners”
Multimodal Learning Experiences
data from multimodal and multi-sensor
interfaces
= Multimodal Learning Analytics
a more accurate representation of the
learning process
10
Multimodal Learning Experiences

11. Multimodal Learning Experiences (MLX)
“any digital-enhanced learning activity using more than two modalities within an authentic learning setting”
Multimodal Learning Experiences 11
(1) Sensors (2) Authentic practice (3) Immersive tech
The 3 pillars of MLX

12. Some examples from related research
Benefits of MLX for teaching and
learning
Multimodal Learning Experiences 12

13. Benefit 1) Create richer online learning experiences
Multimodal Learning Experiences 13
Open Source Real-time Infrastructure for Zoom Augmentation https://zoomsense.org/
Example: ZoomSense

14. Benefit 2) Support to psychomotor learning
Multimodal Learning Experiences 14
(Schneider et al, 2015; 2019)
Example: Presentation Trainer

15. Benefit 3) Create premium in-presence learning experiences
Multimodal Learning Experiences 15
Example: Edusense
https://github.com/edusense/edusense

16. Designing MLX systems that works (and that fail gracefully)
Engineering MLX systems
Multimodal Learning Experiences 16

17. How can MLX support learning?
Objective of MLX: unobtrusive tracking during deliberate practice
..however…
- Sensor data is messy and has poor semantic value
- Multimodal data introduces multidimensional complexity
- In my PhD thesis, the Multimodal Tutor (Di Mitri, 2020)
- the Five Big challenges
- the Multimodal Learning Analytics Model
- the Multimodal Pipeline
Multimodal Learning Experiences 17
5. Data
Exploitation
User
1. Data
collection
3. Data
Processing
4. Data
Annotation
2. Data
storing
Multimodal
Feedback
Loop

18. Structure of a generic MLX system
Multimodal Learning Experiences 18
• Interaction layer
• the user interface (UI), the sensors and actuators
• Data layer
• collection, storing, annotation, processing
multimodal data
• Feedback layer
• feedback strategy, real-time or retrospective
• Task layer
• task sequence, modelling, constraints etc.

19. EEG
Empatica
Myo
Leap
Motion
Data collection & storing challenges
19
Multimodal Learning Experiences
Modalities Sensors Controllers
Multimodal
Learning
Hub
example data from
EMG sensor
(Schneider et
al, 2018)
Multimodal Learning Experiences

20. Data annotation challenge
Multimodal Learning Experiences
20
Multimodal Learning Experiences

21. Real-time feedback architecture (Di Mitri et al., 2021)
Sensors
CPRTutor
C# app
SharpFlow
Python3
TCP
client
TCP
server
Chunk
(1 CC – 0.5
sec)
Classification ML models
ClassRate
ClassRelease
ClassDepth
ArmsLocked
BodyWeight
Feedback
21

22. Positive effect of real-time feedback
error rates drops shortly
after the feedback is fired
22
Multimodal Learning Experiences

23. Design Based Research
• First Iteration:
‒ Basic requirement analysis -- what needs to be
practice, how can it be practice, usually skills are
complex
‒ Part task analysis -- to select a small part of the
skill
‒ Prototype development ---
‒ User tests to test feasibility, user experience,
usability, F score, etc.
• Second iteration:
‒ Improve prototype
‒ Involve experts (optional to help to improve the
prototype)
‒ Evaluations to test learning gains might be possible
Multimodal Learning Experiences 23

24. Design learning activities with analytics in mind
Multimodal Learning Experiences 24
Fola2 board game – Learning Analytics for Learning Activity design – www.fola2.com

25. MLX systems that we created so far
Application use cases
Multimodal Learning Experiences 25

26. Calligraphy Trainer (Limbu et al, 2019)
26
Multimodal Learning Experiences

27. CPR Tutor (Di Mitri et al 2020)
Multimodal Learning Experiences
Lock your
arms!
Use your
body
weight!
Release
the
compression!
*Metronome
sound 110bpm*
Check
compressio
n depth!
Microsoft
Kinect v2
Simpad
Laerdal
ResusciAnne
Myo armband
27
Multimodal Learning Experiences

28. Edutex architecture (Ciordas-Hertel et al., 2021)
Multimodal Learning Experiences 28

29. Many other examples
Multimodal Learning Experiences 29
• Salsa Trainer (Romano et al. 2019)
• Tennis Table Tutor (Asyraaf et al., 2021)
• Mode of transportation tracking (Di Mitri et al., 2020)
• Pilates Trainer (Meik et al., 2021)
• … more to come …

30. Immersive Learning for Psychomotor Skills
Multimodal Learning Experiences 30
• Multimodal Immersive Learning with Artificial Intelligence for
Psychomotor Skills
• BMBF project with DFKI, RWTH Aachen, TH Köln, DSHS
• Two main use cases
‒ Running use case
‒ Assembly Human-Robot collaboration
• https://milki-psy.de/

31. Research direction: how to include the teacher in the loop?
Multimodal Learning Experiences 31

32. Caveats of MLX – and data centric technologies
1. Privacy and ethical concerns – for which purpose are
student’s data collected?
2. Deviate from original learning goals -> Alignment
problem
3. Data intensive technologies can deepen social
inequalities – expensive hardware amplifies the digital
devide
4. Deepening of social fragmentation and isolation
5. Threaten the student or teacher’s agency (e.g. no
forgiveness, no learn by your mistakes)
What else?
Multimodal Learning Experiences 32

33. Conclusions
• Multimodal Learning Experiences can improve online/teaching learning
1. by creating richer online learning experiences
2. by supporting to psychomotor learning tasks
3. by creating premium in-presence learning experiences
Restoring context, especially when learning in isolation
• However, they also introduce some risks
• We need more research on how to mitigate these risks and maximise benefits
Multimodal Learning Experiences 33

34. Thanks for listening!
Daniele Di Mitri
dimitri@dipf.de
@dimstudi0
34
Multimodal Learning Experiences

35. Q&A time
Multimodal Learning Experiences 35