SlideShare a Scribd company logo

The Multimodal Learning Analytics Pipeline

Presented the 25th May 2019 at the conference Artificial Intelligence and Adaptive Education (AIAED'19) Beijing, China. Abstract: We introduce the Multimodal Learning Analytics Pipeline, a generic approach for collecting and exploiting multimodal data to support learning activities across physical and digital spaces. The MMLA Pipeline facilitates researchers in setting up their multimodal experiments, reducing setup and configuration time required for collecting meaningful datasets. Using the MMLA Pipeline, researchers can decide to use a set of custom sensors to track different modalities, including behavioural cues or affective states. Hence, researchers can quickly obtain multimodal sessions consisting of synchronised sensor data and video recordings. They can analyse and annotate the sessions recorded and train machine learning algorithms to classify or predict the patterns investigated.

1 of 15
Download to read offline
The Multimodal Learning Analytics Pipeline
The Multimodal
Learning Analytics
Pipeline
Daniele Di Mitri1, Jan Schneider2,
Marcus Specht3, Hendrik Drachsler2
1 Open University of The Netherlands, The Netherlands
2 German Institute for International Educational Research, Germany
3 Delft University of Technology, The Netherlands
• We introduce the Multimodal Learning
Analytics Pipeline (MMLA Pipeline)
• a generic approach for collecting and exploiting
multimodal data to support learning activities across
physical and digital spaces
• Using Internet of Things devices, Wearable sensors,
Signal Processing and facilitating Machine Learning
Introduction
1. Psychomotor learning
• coordination between body and mind
2. Multimodal Learning
• Modes: text, image, speech, haptic
• Modalities: speaking, gesturing, moving,
facial expressions, physiological signals
3. Embodied communication
• People communicate using whole body
Relevant Theories of Learning
Multimodal Learning Analytics
(MMLA)
Learning Analytics approach
Measurement, collection, analysis and
reporting of data about learners
+
Data from multiple modalities
=
More accurate representation
of the learning process!
• Problem: multimodal data is complex
• multi-dimensional, have different format, support
• It’s noisy, messy,
• Difficult to store, synchronize, annotate, and exploit
• Solution: MMLA Pipeline
• Support researchers in setting up experiments
much more quickly
• Standard tools over tailor-made solutions
• Reduce data manipulation over-head
• Focus on analysis
Enabling technological advances

Recommended

Real-time Multimodal Feedback with the CPR Tutor
Real-time Multimodal Feedback with the CPR TutorReal-time Multimodal Feedback with the CPR Tutor
Real-time Multimodal Feedback with the CPR TutorDaniele Di Mitri
 
Multimodal Tutor for CPR presented at AIME'19
Multimodal Tutor for CPR presented at AIME'19Multimodal Tutor for CPR presented at AIME'19
Multimodal Tutor for CPR presented at AIME'19Daniele Di Mitri
 
Read Between The Lines: an Annotation Tool for Multimodal Data
Read Between The Lines: an Annotation Tool for Multimodal DataRead Between The Lines: an Annotation Tool for Multimodal Data
Read Between The Lines: an Annotation Tool for Multimodal DataDaniele Di Mitri
 
Multimodal Tutor - Adaptive feedback from multimodal experience capturing
Multimodal Tutor - Adaptive feedback from multimodal experience capturingMultimodal Tutor - Adaptive feedback from multimodal experience capturing
Multimodal Tutor - Adaptive feedback from multimodal experience capturingDaniele Di Mitri
 
Workshop: Multimodal Tutor
Workshop: Multimodal TutorWorkshop: Multimodal Tutor
Workshop: Multimodal TutorDaniele Di Mitri
 
SITE Interactive kenyote 2021
SITE Interactive kenyote 2021SITE Interactive kenyote 2021
SITE Interactive kenyote 2021Daniele Di Mitri
 
Learning Pulse - paper presentation at LAK17
Learning Pulse - paper presentation at LAK17Learning Pulse - paper presentation at LAK17
Learning Pulse - paper presentation at LAK17Daniele Di Mitri
 
The Multimodal Tutor - Presentation PhD defence
The Multimodal Tutor - Presentation PhD defenceThe Multimodal Tutor - Presentation PhD defence
The Multimodal Tutor - Presentation PhD defenceDaniele Di Mitri
 

More Related Content

What's hot

Predicting the future with social media
Predicting the future with social mediaPredicting the future with social media
Predicting the future with social mediaPeter Wlodarczak
 
Multimedia data mining using deep learning
Multimedia data mining using deep learningMultimedia data mining using deep learning
Multimedia data mining using deep learningPeter Wlodarczak
 
Deep Learning of High-Level Representations
Deep Learning of High-Level RepresentationsDeep Learning of High-Level Representations
Deep Learning of High-Level RepresentationsHamid Eghbal-zadeh
 
PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...
PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...
PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...webwinkelvakdag
 
Deep Learning Projects - Anomaly Detection Using Deep Learning
Deep Learning Projects - Anomaly Detection Using Deep LearningDeep Learning Projects - Anomaly Detection Using Deep Learning
Deep Learning Projects - Anomaly Detection Using Deep LearningDezyreAcademy
 
Introduction to Few shot learning
Introduction to Few shot learningIntroduction to Few shot learning
Introduction to Few shot learningRidge-i, Inc.
 
Applying Soft Computing Techniques in Information Retrieval
Applying Soft Computing Techniques in Information RetrievalApplying Soft Computing Techniques in Information Retrieval
Applying Soft Computing Techniques in Information RetrievalIJAEMSJORNAL
 
Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021Vincenzo Lomonaco
 
Continual Learning over Small non-i.i.d. Batches of Natural Video Streams
Continual Learning over Small non-i.i.d. Batches of Natural Video StreamsContinual Learning over Small non-i.i.d. Batches of Natural Video Streams
Continual Learning over Small non-i.i.d. Batches of Natural Video StreamsGabriele Graffieti
 
Separating Gesture Detection and Application Control Concerns with a Multimo...
Separating Gesture Detection and Application Control Concerns with a Multimo...Separating Gesture Detection and Application Control Concerns with a Multimo...
Separating Gesture Detection and Application Control Concerns with a Multimo...Luís Fernandes
 
Greenfoot in Problem-solving and Artificial Intelligence
Greenfoot in Problem-solving and Artificial IntelligenceGreenfoot in Problem-solving and Artificial Intelligence
Greenfoot in Problem-solving and Artificial IntelligenceScott Turner
 
On Collective Reinforcement Learning
On Collective Reinforcement LearningOn Collective Reinforcement Learning
On Collective Reinforcement LearningGianluca Aguzzi
 
Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...
Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...
Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...Gabriele Graffieti
 
Fractional step discriminant pruning
Fractional step discriminant pruningFractional step discriminant pruning
Fractional step discriminant pruningVasileiosMezaris
 
Architecture for Integrating Real Objects with Virtual Academic Communities
Architecture for Integrating Real Objects with VirtualAcademic CommunitiesArchitecture for Integrating Real Objects with VirtualAcademic Communities
Architecture for Integrating Real Objects with Virtual Academic Communitiesgalex68
 

What's hot (16)

Predicting the future with social media
Predicting the future with social mediaPredicting the future with social media
Predicting the future with social media
 
Multimedia data mining using deep learning
Multimedia data mining using deep learningMultimedia data mining using deep learning
Multimedia data mining using deep learning
 
Deep Learning of High-Level Representations
Deep Learning of High-Level RepresentationsDeep Learning of High-Level Representations
Deep Learning of High-Level Representations
 
PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...
PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...
PREDICTING STOCK PRICE MOVEMENTS BASED ON NEWSPAPER ARTICLES USING A NOVEL DE...
 
Deep Learning Projects - Anomaly Detection Using Deep Learning
Deep Learning Projects - Anomaly Detection Using Deep LearningDeep Learning Projects - Anomaly Detection Using Deep Learning
Deep Learning Projects - Anomaly Detection Using Deep Learning
 
Introduction to Few shot learning
Introduction to Few shot learningIntroduction to Few shot learning
Introduction to Few shot learning
 
Applying Soft Computing Techniques in Information Retrieval
Applying Soft Computing Techniques in Information RetrievalApplying Soft Computing Techniques in Information Retrieval
Applying Soft Computing Techniques in Information Retrieval
 
Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021Continual Learning with Deep Architectures - Tutorial ICML 2021
Continual Learning with Deep Architectures - Tutorial ICML 2021
 
Continual Learning over Small non-i.i.d. Batches of Natural Video Streams
Continual Learning over Small non-i.i.d. Batches of Natural Video StreamsContinual Learning over Small non-i.i.d. Batches of Natural Video Streams
Continual Learning over Small non-i.i.d. Batches of Natural Video Streams
 
Separating Gesture Detection and Application Control Concerns with a Multimo...
Separating Gesture Detection and Application Control Concerns with a Multimo...Separating Gesture Detection and Application Control Concerns with a Multimo...
Separating Gesture Detection and Application Control Concerns with a Multimo...
 
Greenfoot in Problem-solving and Artificial Intelligence
Greenfoot in Problem-solving and Artificial IntelligenceGreenfoot in Problem-solving and Artificial Intelligence
Greenfoot in Problem-solving and Artificial Intelligence
 
On Collective Reinforcement Learning
On Collective Reinforcement LearningOn Collective Reinforcement Learning
On Collective Reinforcement Learning
 
Designing and eXperiencing Smart Objects based Learning Scenarios: an approac...
Designing and eXperiencing Smart Objects based Learning Scenarios: an approac...Designing and eXperiencing Smart Objects based Learning Scenarios: an approac...
Designing and eXperiencing Smart Objects based Learning Scenarios: an approac...
 
Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...
Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...
Self-supervised Learning for Next-Generation Industry-level Autonomous Drivin...
 
Fractional step discriminant pruning
Fractional step discriminant pruningFractional step discriminant pruning
Fractional step discriminant pruning
 
Architecture for Integrating Real Objects with Virtual Academic Communities
Architecture for Integrating Real Objects with VirtualAcademic CommunitiesArchitecture for Integrating Real Objects with VirtualAcademic Communities
Architecture for Integrating Real Objects with Virtual Academic Communities
 

Similar to The Multimodal Learning Analytics Pipeline

2013 05 17 (uned) emadrid mtotschnig grados apertura mooc
2013 05 17 (uned) emadrid mtotschnig grados apertura mooc2013 05 17 (uned) emadrid mtotschnig grados apertura mooc
2013 05 17 (uned) emadrid mtotschnig grados apertura mooceMadrid network
 
Multimedia in Education
Multimedia in EducationMultimedia in Education
Multimedia in EducationAce Matilac
 
Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...
Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...
Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...Daniele Di Mitri
 
Evaluation of Linked Data tools for Learning Analytics
Evaluation of Linked Data tools for Learning AnalyticsEvaluation of Linked Data tools for Learning Analytics
Evaluation of Linked Data tools for Learning AnalyticsHendrik Drachsler
 
Jpc era5 sketchup pres
Jpc era5 sketchup presJpc era5 sketchup pres
Jpc era5 sketchup presJohn Parker
 
Universal Design for Learning ED554
Universal Design for Learning ED554Universal Design for Learning ED554
Universal Design for Learning ED554Steven Knight
 
Lessons Learned from a Digital Tool Criticism Workshop
Lessons Learned from a Digital Tool Criticism WorkshopLessons Learned from a Digital Tool Criticism Workshop
Lessons Learned from a Digital Tool Criticism WorkshopMarijn Koolen
 
Concept of instructional media
Concept of instructional mediaConcept of instructional media
Concept of instructional mediaStephen Ndawula
 
UDL Presentation- APP4ChungH
UDL Presentation- APP4ChungHUDL Presentation- APP4ChungH
UDL Presentation- APP4ChungHhchung723
 
Six dimensions of Learning Analytics
Six dimensions of Learning AnalyticsSix dimensions of Learning Analytics
Six dimensions of Learning AnalyticsHendrik Drachsler
 
De zes dimensies van learning analytics
De zes dimensies van learning analyticsDe zes dimensies van learning analytics
De zes dimensies van learning analyticsSURF Events
 
Lewis tech unit iste technology standards met
Lewis tech unit iste technology standards metLewis tech unit iste technology standards met
Lewis tech unit iste technology standards metlaurajelewis
 
LAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCH
LAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCHLAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCH
LAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCHThieme Hennis
 
OU Live (synchronous online tutorials) pedagogy
OU Live (synchronous online tutorials) pedagogyOU Live (synchronous online tutorials) pedagogy
OU Live (synchronous online tutorials) pedagogyMandy Honeyman
 

Similar to The Multimodal Learning Analytics Pipeline (20)

Critical Thinking in Information Literacy: what teachers can do in digital era?
Critical Thinking in Information Literacy: what teachers can do in digital era?Critical Thinking in Information Literacy: what teachers can do in digital era?
Critical Thinking in Information Literacy: what teachers can do in digital era?
 
2013 05 17 (uned) emadrid mtotschnig grados apertura mooc
2013 05 17 (uned) emadrid mtotschnig grados apertura mooc2013 05 17 (uned) emadrid mtotschnig grados apertura mooc
2013 05 17 (uned) emadrid mtotschnig grados apertura mooc
 
Multimedia in Education
Multimedia in EducationMultimedia in Education
Multimedia in Education
 
Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...
Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...
Guest Lecture: Restoring Context in Distance Learning with Artificial Intelli...
 
Evaluation of Linked Data tools for Learning Analytics
Evaluation of Linked Data tools for Learning AnalyticsEvaluation of Linked Data tools for Learning Analytics
Evaluation of Linked Data tools for Learning Analytics
 
Jpc era5 sketchup pres
Jpc era5 sketchup presJpc era5 sketchup pres
Jpc era5 sketchup pres
 
Multimedia
MultimediaMultimedia
Multimedia
 
Universal Design for Learning ED554
Universal Design for Learning ED554Universal Design for Learning ED554
Universal Design for Learning ED554
 
Dixit_resume
Dixit_resumeDixit_resume
Dixit_resume
 
Lessons Learned from a Digital Tool Criticism Workshop
Lessons Learned from a Digital Tool Criticism WorkshopLessons Learned from a Digital Tool Criticism Workshop
Lessons Learned from a Digital Tool Criticism Workshop
 
Concept of instructional media
Concept of instructional mediaConcept of instructional media
Concept of instructional media
 
UDL Presentation- APP4ChungH
UDL Presentation- APP4ChungHUDL Presentation- APP4ChungH
UDL Presentation- APP4ChungH
 
Six dimensions of Learning Analytics
Six dimensions of Learning AnalyticsSix dimensions of Learning Analytics
Six dimensions of Learning Analytics
 
De zes dimensies van learning analytics
De zes dimensies van learning analyticsDe zes dimensies van learning analytics
De zes dimensies van learning analytics
 
Lewis tech unit iste technology standards met
Lewis tech unit iste technology standards metLewis tech unit iste technology standards met
Lewis tech unit iste technology standards met
 
Hub DE summit Sydney
Hub DE summit SydneyHub DE summit Sydney
Hub DE summit Sydney
 
The NeDiMAH workgroup on Space and Time
The NeDiMAH workgroup on Space and TimeThe NeDiMAH workgroup on Space and Time
The NeDiMAH workgroup on Space and Time
 
Kreijns english
Kreijns englishKreijns english
Kreijns english
 
LAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCH
LAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCHLAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCH
LAK15 - SCALABILITY AND FLEXIBILITY THROUGH OPEN RESEARCH
 
OU Live (synchronous online tutorials) pedagogy
OU Live (synchronous online tutorials) pedagogyOU Live (synchronous online tutorials) pedagogy
OU Live (synchronous online tutorials) pedagogy
 

More from Daniele Di Mitri

SenseTheClassroom Live at EC-TEL 2022
SenseTheClassroom Live at EC-TEL 2022SenseTheClassroom Live at EC-TEL 2022
SenseTheClassroom Live at EC-TEL 2022Daniele Di Mitri
 
MOBIUS: Smart Mobility Tracking with Smartphone Sensors
MOBIUS: Smart Mobility Tracking with Smartphone SensorsMOBIUS: Smart Mobility Tracking with Smartphone Sensors
MOBIUS: Smart Mobility Tracking with Smartphone SensorsDaniele Di Mitri
 
The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...
The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...
The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...Daniele Di Mitri
 
Sensors for Learning workshop
Sensors for Learning workshopSensors for Learning workshop
Sensors for Learning workshopDaniele Di Mitri
 
Multimodal Machines #JTELSS17 workshop
Multimodal Machines #JTELSS17 workshopMultimodal Machines #JTELSS17 workshop
Multimodal Machines #JTELSS17 workshopDaniele Di Mitri
 
Digital Learning Projection - poster for #LAK17
Digital Learning Projection - poster for #LAK17Digital Learning Projection - poster for #LAK17
Digital Learning Projection - poster for #LAK17Daniele Di Mitri
 
Digital Learning Projection - Learning state estimation from multimodal learn...
Digital Learning Projection - Learning state estimation from multimodal learn...Digital Learning Projection - Learning state estimation from multimodal learn...
Digital Learning Projection - Learning state estimation from multimodal learn...Daniele Di Mitri
 
Academic writing in LaTeX
Academic writing in LaTeX Academic writing in LaTeX
Academic writing in LaTeX Daniele Di Mitri
 
Visual Learning Pulse - Final Thesis presentation
Visual Learning Pulse - Final Thesis presentationVisual Learning Pulse - Final Thesis presentation
Visual Learning Pulse - Final Thesis presentationDaniele Di Mitri
 
Research project MAI2 - Final Presentation Group 4
Research project MAI2  - Final Presentation Group 4Research project MAI2  - Final Presentation Group 4
Research project MAI2 - Final Presentation Group 4Daniele Di Mitri
 
Lifelong Topic Modelling presentation
Lifelong Topic Modelling presentation Lifelong Topic Modelling presentation
Lifelong Topic Modelling presentation Daniele Di Mitri
 
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...Daniele Di Mitri
 
Battlecode2014 - final presentation - group n.2
Battlecode2014 - final presentation - group n.2Battlecode2014 - final presentation - group n.2
Battlecode2014 - final presentation - group n.2Daniele Di Mitri
 
Inclusive & open learning analytics
Inclusive & open learning analyticsInclusive & open learning analytics
Inclusive & open learning analyticsDaniele Di Mitri
 
(IT) Slides della presentazione della tesi di Laurea
(IT) Slides della presentazione della tesi di Laurea(IT) Slides della presentazione della tesi di Laurea
(IT) Slides della presentazione della tesi di LaureaDaniele Di Mitri
 
Claim Your Voice - Presentation of the results of the campign for VET students
Claim Your Voice - Presentation of the results of the campign for VET studentsClaim Your Voice - Presentation of the results of the campign for VET students
Claim Your Voice - Presentation of the results of the campign for VET studentsDaniele Di Mitri
 
Obessu views on school resource
Obessu views on school resourceObessu views on school resource
Obessu views on school resourceDaniele Di Mitri
 
Obessu’s inputs on «opening up education»
Obessu’s inputs on «opening up education»Obessu’s inputs on «opening up education»
Obessu’s inputs on «opening up education»Daniele Di Mitri
 
European politicaldebates presentation
European politicaldebates presentationEuropean politicaldebates presentation
European politicaldebates presentationDaniele Di Mitri
 

More from Daniele Di Mitri (20)

SenseTheClassroom Live at EC-TEL 2022
SenseTheClassroom Live at EC-TEL 2022SenseTheClassroom Live at EC-TEL 2022
SenseTheClassroom Live at EC-TEL 2022
 
MOBIUS: Smart Mobility Tracking with Smartphone Sensors
MOBIUS: Smart Mobility Tracking with Smartphone SensorsMOBIUS: Smart Mobility Tracking with Smartphone Sensors
MOBIUS: Smart Mobility Tracking with Smartphone Sensors
 
The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...
The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...
The Multimodal Tutor - short pitch presentation at JTELSS 2018 in Durrës, Alb...
 
Sensors for Learning workshop
Sensors for Learning workshopSensors for Learning workshop
Sensors for Learning workshop
 
Multimodal Machines #JTELSS17 workshop
Multimodal Machines #JTELSS17 workshopMultimodal Machines #JTELSS17 workshop
Multimodal Machines #JTELSS17 workshop
 
Visual Learning Pulse
Visual Learning PulseVisual Learning Pulse
Visual Learning Pulse
 
Digital Learning Projection - poster for #LAK17
Digital Learning Projection - poster for #LAK17Digital Learning Projection - poster for #LAK17
Digital Learning Projection - poster for #LAK17
 
Digital Learning Projection - Learning state estimation from multimodal learn...
Digital Learning Projection - Learning state estimation from multimodal learn...Digital Learning Projection - Learning state estimation from multimodal learn...
Digital Learning Projection - Learning state estimation from multimodal learn...
 
Academic writing in LaTeX
Academic writing in LaTeX Academic writing in LaTeX
Academic writing in LaTeX
 
Visual Learning Pulse - Final Thesis presentation
Visual Learning Pulse - Final Thesis presentationVisual Learning Pulse - Final Thesis presentation
Visual Learning Pulse - Final Thesis presentation
 
Research project MAI2 - Final Presentation Group 4
Research project MAI2  - Final Presentation Group 4Research project MAI2  - Final Presentation Group 4
Research project MAI2 - Final Presentation Group 4
 
Lifelong Topic Modelling presentation
Lifelong Topic Modelling presentation Lifelong Topic Modelling presentation
Lifelong Topic Modelling presentation
 
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
Word2Vec: Learning of word representations in a vector space - Di Mitri & Her...
 
Battlecode2014 - final presentation - group n.2
Battlecode2014 - final presentation - group n.2Battlecode2014 - final presentation - group n.2
Battlecode2014 - final presentation - group n.2
 
Inclusive & open learning analytics
Inclusive & open learning analyticsInclusive & open learning analytics
Inclusive & open learning analytics
 
(IT) Slides della presentazione della tesi di Laurea
(IT) Slides della presentazione della tesi di Laurea(IT) Slides della presentazione della tesi di Laurea
(IT) Slides della presentazione della tesi di Laurea
 
Claim Your Voice - Presentation of the results of the campign for VET students
Claim Your Voice - Presentation of the results of the campign for VET studentsClaim Your Voice - Presentation of the results of the campign for VET students
Claim Your Voice - Presentation of the results of the campign for VET students
 
Obessu views on school resource
Obessu views on school resourceObessu views on school resource
Obessu views on school resource
 
Obessu’s inputs on «opening up education»
Obessu’s inputs on «opening up education»Obessu’s inputs on «opening up education»
Obessu’s inputs on «opening up education»
 
European politicaldebates presentation
European politicaldebates presentationEuropean politicaldebates presentation
European politicaldebates presentation
 

Recently uploaded

Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023stephizcoolio
 
Artificial Intelligence and its Impact on Society.pptx
Artificial Intelligence and its Impact on Society.pptxArtificial Intelligence and its Impact on Society.pptx
Artificial Intelligence and its Impact on Society.pptxVighnesh Shashtri
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)UNCResearchHub
 
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdfIIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdfAustraliaChapterIIBA
 
SABARI PRIYAN's self introduction as a reference
SABARI PRIYAN's self introduction as a referenceSABARI PRIYAN's self introduction as a reference
SABARI PRIYAN's self introduction as a referencepriyansabari355
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensKondapi V Siva Rama Brahmam
 
Lies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix EnigmaLies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix EnigmaAdrian Sanabria
 
AWS Identity and access management for users
AWS Identity and access management for usersAWS Identity and access management for users
AWS Identity and access management for usersStephenEfange3
 
Industry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptxIndustry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptxMdRafiqulIslam403212
 
Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...
Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...
Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...Cyber Security Experts
 
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...Thibaud Le Douarin
 
SABARI PRIYAN's self introduction as reference
SABARI PRIYAN's self introduction as referenceSABARI PRIYAN's self introduction as reference
SABARI PRIYAN's self introduction as referencepriyansabari355
 
Oppotus - Malaysians on Malaysia 4Q 2023.pdf
Oppotus - Malaysians on Malaysia 4Q 2023.pdfOppotus - Malaysians on Malaysia 4Q 2023.pdf
Oppotus - Malaysians on Malaysia 4Q 2023.pdfOppotus
 
data analytics and tools from in2inglobal.pdf
data analytics  and tools from in2inglobal.pdfdata analytics  and tools from in2inglobal.pdf
data analytics and tools from in2inglobal.pdfdigimartfamily
 
Big Data - large Scale data (Amazon, FB)
Big Data - large Scale data (Amazon, FB)Big Data - large Scale data (Amazon, FB)
Big Data - large Scale data (Amazon, FB)CUO VEERANAN VEERANAN
 

Recently uploaded (17)

Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023Soil Health Policy Map Years 2020 to 2023
Soil Health Policy Map Years 2020 to 2023
 
2.pptx
2.pptx2.pptx
2.pptx
 
Artificial Intelligence and its Impact on Society.pptx
Artificial Intelligence and its Impact on Society.pptxArtificial Intelligence and its Impact on Society.pptx
Artificial Intelligence and its Impact on Society.pptx
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)
 
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdfIIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
IIBA Adl - Being Effective on Day 1 - Slide Deck.pdf
 
SABARI PRIYAN's self introduction as a reference
SABARI PRIYAN's self introduction as a referenceSABARI PRIYAN's self introduction as a reference
SABARI PRIYAN's self introduction as a reference
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample Screens
 
Electricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptxElectricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptx
 
Lies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix EnigmaLies and Myths in InfoSec - 2023 Usenix Enigma
Lies and Myths in InfoSec - 2023 Usenix Enigma
 
AWS Identity and access management for users
AWS Identity and access management for usersAWS Identity and access management for users
AWS Identity and access management for users
 
Industry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptxIndustry 4.0 in IoT Transforming the Future.pptx
Industry 4.0 in IoT Transforming the Future.pptx
 
Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...
Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...
Web 3.0 in Data Privacy and Security | Data Privacy |Blockchain Security| Cyb...
 
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
Generative AI Rennes Meetup with OVHcloud - WAICF highlights & how to deploy ...
 
SABARI PRIYAN's self introduction as reference
SABARI PRIYAN's self introduction as referenceSABARI PRIYAN's self introduction as reference
SABARI PRIYAN's self introduction as reference
 
Oppotus - Malaysians on Malaysia 4Q 2023.pdf
Oppotus - Malaysians on Malaysia 4Q 2023.pdfOppotus - Malaysians on Malaysia 4Q 2023.pdf
Oppotus - Malaysians on Malaysia 4Q 2023.pdf
 
data analytics and tools from in2inglobal.pdf
data analytics  and tools from in2inglobal.pdfdata analytics  and tools from in2inglobal.pdf
data analytics and tools from in2inglobal.pdf
 
Big Data - large Scale data (Amazon, FB)
Big Data - large Scale data (Amazon, FB)Big Data - large Scale data (Amazon, FB)
Big Data - large Scale data (Amazon, FB)
 

The Multimodal Learning Analytics Pipeline

  • 2. The Multimodal Learning Analytics Pipeline Daniele Di Mitri1, Jan Schneider2, Marcus Specht3, Hendrik Drachsler2 1 Open University of The Netherlands, The Netherlands 2 German Institute for International Educational Research, Germany 3 Delft University of Technology, The Netherlands
  • 3. • We introduce the Multimodal Learning Analytics Pipeline (MMLA Pipeline) • a generic approach for collecting and exploiting multimodal data to support learning activities across physical and digital spaces • Using Internet of Things devices, Wearable sensors, Signal Processing and facilitating Machine Learning Introduction
  • 4. 1. Psychomotor learning • coordination between body and mind 2. Multimodal Learning • Modes: text, image, speech, haptic • Modalities: speaking, gesturing, moving, facial expressions, physiological signals 3. Embodied communication • People communicate using whole body Relevant Theories of Learning
  • 5. Multimodal Learning Analytics (MMLA) Learning Analytics approach Measurement, collection, analysis and reporting of data about learners + Data from multiple modalities = More accurate representation of the learning process!
  • 6. • Problem: multimodal data is complex • multi-dimensional, have different format, support • It’s noisy, messy, • Difficult to store, synchronize, annotate, and exploit • Solution: MMLA Pipeline • Support researchers in setting up experiments much more quickly • Standard tools over tailor-made solutions • Reduce data manipulation over-head • Focus on analysis Enabling technological advances
  • 7. Graphical overview Task model 3rd party sensors or API 2. Data storing Dashboards Physiological sensor data Motoric sensor data (D) Historical reports (B) Predictions (C) Patterns 5. Data exploitation 1. Data collection Processed data store 4. Data processing Predictions Model fitting (A) Corrective feedback Intelligent Tutors (A) Evaluation Expert reports 3. Data annotation (B,C) Prediction models (D) (D) B,C (B,C) RESEARCHPRODUCTION correctionsawareness orchestration adaptation
  • 8. Current prototypes 1. Multimodal Learning Hub (Schneider et al., 2018) data collection, data storing 2. Visual Inspection Tool (Di Mitri et al., 2019) data annotation
  • 9. MLT data format example of serialisation of Myo data in JSON MLT = Meaningful Learning Experience example annotation.json
  • 10. A. Corrective feedback: hardcoded rules e.g. “if sensor value is x then y”; (non- adaptive) B. Classification/Prediction: estimation of the learning labels (adaptive) C. Pattern identification mining of recurrent sensor values D. Historical reports: visualizations and analytics dashboard Exploitation strategies
  • 11. Real world applications 3. CPR Tutor 2. Presentation Trainer1. Calligraphy trainer
  • 12. • All cases are Multimodal Tutors • intelligent tutoring systems that use multimodal data • The MMLA Pipeline supports • the collection, analysis, annotation and exploitation of multimodal data • Current prototype • optimized for individual learning • recording of ~10 minutes, retrospective feedback • Future prototypes • real-time multimodal feedback • collaborative learning • longer recorded sessions Evidence of potential impact
  • 13. • The MMLA Pipeline is a generic and useful approach for researchers • Flexible and extensible applications • Can be used with a range of sensor applications • The different components are Open Source • Available for demo! Summary
  • 14. • Schneider, J., Di Mitri, D., Limbu, B., & Drachsler, H. (2018). Multimodal Learning Hub: A Tool for Capturing Customizable Multimodal Learning Experiences, 1, 45– 58. http://doi.org/10.1007/978-3-319-98572-5_4 • Di Mitri D, Schneider J, Specht M, Drachsler H. From signals to knowledge: A conceptual model for multimodal learning analytics. J Comput Assist Learn. 2018;34:338–349. https://doi.org/10.1111/jcal.12288 • Di Mitri D., Schneider J., Specht M., Drachsler H. (2019) Read Between the Lines: An Annotation Tool for Multimodal Data for Learning. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge - LAK19 (pp. 51–60). New York, NY, USA: ACM. http://doi.org/10.1145/3303772.3303776 • Di Mitri D. (2018) Multimodal Tutor for CPR. In: Penstein Rosé C. et al. (eds) Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science, vol 10948. Springer, Cham. http://doi.org/10.1007/978-3-319-93846-2_96 • Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018) The Big Five: Addressing Recurrent Multimodal Learning Data Challenges. In R. Martinez- Maldonado et al. (Eds.), Proceedings of the Second Multimodal Learning Analytics Across (Physical and Digital) Spaces (CrossMMLA), Vol. 2163. CEUR Proceedings. http://ceur-ws.org/Vol-2163/#paper6 Useful references