SlideShare a Scribd company logo
RESEARCH POSTER PRESENTATION DESIGN © 2015
www.PosterPresentations.com
TimeDomain
Features
FrequencyDomain
Features
Simulations enable a risk-free way for healthcare practitioners to learn and enhance their skills.
Such simulations should be tailored to the needs of the individual learners. However, most
current simulations do not consider the learner’s existing skills, expertise, and ability to handle
cognitive load. We propose an end-to-end framework for trauma simulation that actively
classifies participant's cognitive load and level of expertise and dynamically modulates the
complexity of a simulation by altering symptom severity of a patient using augmented reality
(AR) to ensure an optimum amount of cognitive load is maintained. A deep learning approach
was taken to utilize electrocardiogram (ECG) and galvanic skin response (GSR) collected using
wearable devices. The model was developed to classify high vs low cognitive load, as well as
level of expertise, and validated using “leave one out” scheme. Finally, the intelligent
dynamically adaptive AR simulation was successfully tested in real time on two participants of
each class.
Abstract
Motivation
Data Pre-processing & Feature Extraction
Experiment Setup
Result and Analysis
Conclusion
We were able to accurately estimate cognitive load and expertise in real-time and use the
classifications to alter a simulation with augmented reality to maintain optimal cognitive load.
Pritam Sarkar1, Kyle Ross1, Dirk Rodenburg3, Aaron Ruberto4, Paul Hungler2, Dan Howes4, Adam Szulewski4, Ali Etemad1
1Dept. of Electrical and Computer Engineering, 2Dept. Of Chemical Engineering, 3Faculty of Engineering and Applied Science,
4Dept. of Emergency Medicine Queen’s University
A Deep Learning Approach for AR-based Adaptive Simulation using Wearables
Cognitive Load
Learning Rate
Microsoft HoloLens ECG Shimmer Sensor
ProposedDynamic SimulationFramework
Data Collection
Data Visualization
DataCollection
AR-based Adaptive
Simulation
ECG based
Convolutional
Neural Network
Feature
Extraction from
ECG
FusedDecision
(Probability)
Feature
Extraction GSR
GSR based Neural
Network
ECG
GSR
❑ Many learning systems are one size fits all
regarding levels of expertise.
❑ The development of expertise is characterized by
the ability to execute increasingly complex tasks
with lowered cognitive load.
❑ To be effective, simulations should match
complexity with the cognitive abilities of the
learner.
❑ Goal: To develop a learning system that
dynamically adjusts complexity to match learner
expertise and cognitive capacity.
GSR Shimmer Sensor
TSNE plot of GSR feature-space without (left) and with (right) baseline correction
TSNE plot of ECG feature-space without (left) and with (right) baseline correction
Expertise Cognitive
Load
Testing (Offline) 91% 74%
Realtime Testing
2 participantsof each class
were successfully tested to
classify level of expertise and
cognitive load.
We used an augmented reality overlayon top of a mannequinto
change the difficulty of the simulationthrough patient symptomology
Participants during simulation
anne u n
stra tor
View from HoloLens during simulation
RR peaks for ECG feature extraction
SCR response for GSR feature extraction
Features
Publications
➢ Conference: Classification of Expertise and Cognitive Load in Adaptive AR-Based Simulation
using Deep Learning, Affective Computing & Intelligent Interaction (ACII), 2019
➢ Journal: Toward Dynamically Adaptive Simulation: Multimodal Classification of User
Expertise using Wearable Devices, Sensors 2019, 19, 4270
➢ Expert participants (Experienced in emergency medicine)
➢ Nov e part pants (Queen’s 4th year medical students)
(Left) Plot of the likelihood for expertise vs
cognitive load. The results show an inverse
relationship between level of expertise and
cognitive load, where the top left corner shows
high level of expertise and low cognitive load,
and the bottom right corner indicates low
expertise and high cognitive load.
Reach us at: pritam.sarkar@queensu.ca, 12kjr1@queensu.ca

More Related Content

Similar to A Deep Learning Approach for AR-based Adaptive Simulation using Wearables​

Iaetsd recognition of emg based hand gestures
Iaetsd recognition of emg based hand gesturesIaetsd recognition of emg based hand gestures
Iaetsd recognition of emg based hand gestures
Iaetsd Iaetsd
 
i2164-2591-8-6-4 (1).pdf presentation cm
i2164-2591-8-6-4 (1).pdf presentation cmi2164-2591-8-6-4 (1).pdf presentation cm
i2164-2591-8-6-4 (1).pdf presentation cm
ahsamjutt1234
 
MAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTOR
MAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTORMAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTOR
MAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTOR
cscpconf
 
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATIONCAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
gerogepatton
 
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATIONCAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
ijaia
 
Review on classification based on artificial
Review on classification based on artificialReview on classification based on artificial
Review on classification based on artificial
ijasa
 
Using Artificial Neural Networks to Detect Multiple Cancers from a Blood Test
Using Artificial Neural Networks to Detect Multiple Cancers from a Blood TestUsing Artificial Neural Networks to Detect Multiple Cancers from a Blood Test
Using Artificial Neural Networks to Detect Multiple Cancers from a Blood Test
StevenQu1
 
Capsule Network Performance with Autonomous Navigation
Capsule Network Performance with Autonomous Navigation Capsule Network Performance with Autonomous Navigation
Capsule Network Performance with Autonomous Navigation
gerogepatton
 
Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...
Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...
Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...
IRJET Journal
 
BLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATION
BLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATIONBLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATION
BLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATION
IRJET Journal
 
researchpaper_2023_Lungs_Cancer.pdfdfgdgfhdf
researchpaper_2023_Lungs_Cancer.pdfdfgdgfhdfresearchpaper_2023_Lungs_Cancer.pdfdfgdgfhdf
researchpaper_2023_Lungs_Cancer.pdfdfgdgfhdf
AvijitChaudhuri3
 
Analysis and Classification of Skin Lesions Using 3D Volume Reconstruction
Analysis and Classification of Skin Lesions Using 3D Volume ReconstructionAnalysis and Classification of Skin Lesions Using 3D Volume Reconstruction
Analysis and Classification of Skin Lesions Using 3D Volume Reconstruction
IOSR Journals
 
Deep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRX
Deep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRXDeep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRX
Deep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRX
BioInformaticsGRX
 
A COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTION
A COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTIONA COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTION
A COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTION
IJCSEA Journal
 
Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​
Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​
Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​
Pritam Sarkar
 
Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...
Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...
Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...
CSCJournals
 
heart final last sem.pptx
heart final last sem.pptxheart final last sem.pptx
heart final last sem.pptx
rakshashadu
 
Performance Comparision of Machine Learning Algorithms
Performance Comparision of Machine Learning AlgorithmsPerformance Comparision of Machine Learning Algorithms
Performance Comparision of Machine Learning Algorithms
Dinusha Dilanka
 
0 eeg based 3 d visual fatigue evaluation using cnn
0 eeg based 3 d visual fatigue evaluation using cnn0 eeg based 3 d visual fatigue evaluation using cnn
0 eeg based 3 d visual fatigue evaluation using cnn
Hoopeer Hoopeer
 
Ensemble learning approach for multi-class classification of Alzheimer’s stag...
Ensemble learning approach for multi-class classification of Alzheimer’s stag...Ensemble learning approach for multi-class classification of Alzheimer’s stag...
Ensemble learning approach for multi-class classification of Alzheimer’s stag...
TELKOMNIKA JOURNAL
 

Similar to A Deep Learning Approach for AR-based Adaptive Simulation using Wearables​ (20)

Iaetsd recognition of emg based hand gestures
Iaetsd recognition of emg based hand gesturesIaetsd recognition of emg based hand gestures
Iaetsd recognition of emg based hand gestures
 
i2164-2591-8-6-4 (1).pdf presentation cm
i2164-2591-8-6-4 (1).pdf presentation cmi2164-2591-8-6-4 (1).pdf presentation cm
i2164-2591-8-6-4 (1).pdf presentation cm
 
MAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTOR
MAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTORMAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTOR
MAMMOGRAPHY LESION DETECTION USING FASTER R-CNN DETECTOR
 
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATIONCAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
 
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATIONCAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
CAPSULE NETWORK PERFORMANCE WITH AUTONOMOUS NAVIGATION
 
Review on classification based on artificial
Review on classification based on artificialReview on classification based on artificial
Review on classification based on artificial
 
Using Artificial Neural Networks to Detect Multiple Cancers from a Blood Test
Using Artificial Neural Networks to Detect Multiple Cancers from a Blood TestUsing Artificial Neural Networks to Detect Multiple Cancers from a Blood Test
Using Artificial Neural Networks to Detect Multiple Cancers from a Blood Test
 
Capsule Network Performance with Autonomous Navigation
Capsule Network Performance with Autonomous Navigation Capsule Network Performance with Autonomous Navigation
Capsule Network Performance with Autonomous Navigation
 
Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...
Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...
Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detec...
 
BLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATION
BLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATIONBLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATION
BLOOD TISSUE IMAGE TO IDENTIFY MALARIA DISEASE CLASSIFICATION
 
researchpaper_2023_Lungs_Cancer.pdfdfgdgfhdf
researchpaper_2023_Lungs_Cancer.pdfdfgdgfhdfresearchpaper_2023_Lungs_Cancer.pdfdfgdgfhdf
researchpaper_2023_Lungs_Cancer.pdfdfgdgfhdf
 
Analysis and Classification of Skin Lesions Using 3D Volume Reconstruction
Analysis and Classification of Skin Lesions Using 3D Volume ReconstructionAnalysis and Classification of Skin Lesions Using 3D Volume Reconstruction
Analysis and Classification of Skin Lesions Using 3D Volume Reconstruction
 
Deep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRX
Deep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRXDeep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRX
Deep Learning aplicado al diagnostico de Alzheimer | BioInformaticsGRX
 
A COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTION
A COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTIONA COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTION
A COMPARATIVE STUDY OF BACKPROPAGATION ALGORITHMS IN FINANCIAL PREDICTION
 
Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​
Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​
Computer-Aided Diagnosis using Class-Weighted Deep Neural Network​
 
Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...
Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...
Evolutionary Algorithm for Optimal Connection Weights in Artificial Neural Ne...
 
heart final last sem.pptx
heart final last sem.pptxheart final last sem.pptx
heart final last sem.pptx
 
Performance Comparision of Machine Learning Algorithms
Performance Comparision of Machine Learning AlgorithmsPerformance Comparision of Machine Learning Algorithms
Performance Comparision of Machine Learning Algorithms
 
0 eeg based 3 d visual fatigue evaluation using cnn
0 eeg based 3 d visual fatigue evaluation using cnn0 eeg based 3 d visual fatigue evaluation using cnn
0 eeg based 3 d visual fatigue evaluation using cnn
 
Ensemble learning approach for multi-class classification of Alzheimer’s stag...
Ensemble learning approach for multi-class classification of Alzheimer’s stag...Ensemble learning approach for multi-class classification of Alzheimer’s stag...
Ensemble learning approach for multi-class classification of Alzheimer’s stag...
 

Recently uploaded

Seminar on Distillation study-mafia.pptx
Seminar on Distillation study-mafia.pptxSeminar on Distillation study-mafia.pptx
Seminar on Distillation study-mafia.pptx
Madan Karki
 
integral complex analysis chapter 06 .pdf
integral complex analysis chapter 06 .pdfintegral complex analysis chapter 06 .pdf
integral complex analysis chapter 06 .pdf
gaafergoudaay7aga
 
BRAIN TUMOR DETECTION for seminar ppt.pdf
BRAIN TUMOR DETECTION for seminar ppt.pdfBRAIN TUMOR DETECTION for seminar ppt.pdf
BRAIN TUMOR DETECTION for seminar ppt.pdf
LAXMAREDDY22
 
Computational Engineering IITH Presentation
Computational Engineering IITH PresentationComputational Engineering IITH Presentation
Computational Engineering IITH Presentation
co23btech11018
 
Design and optimization of ion propulsion drone
Design and optimization of ion propulsion droneDesign and optimization of ion propulsion drone
Design and optimization of ion propulsion drone
bjmsejournal
 
Applications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdfApplications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdf
Atif Razi
 
Properties Railway Sleepers and Test.pptx
Properties Railway Sleepers and Test.pptxProperties Railway Sleepers and Test.pptx
Properties Railway Sleepers and Test.pptx
MDSABBIROJJAMANPAYEL
 
spirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptxspirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptx
Madan Karki
 
Certificates - Mahmoud Mohamed Moursi Ahmed
Certificates - Mahmoud Mohamed Moursi AhmedCertificates - Mahmoud Mohamed Moursi Ahmed
Certificates - Mahmoud Mohamed Moursi Ahmed
Mahmoud Morsy
 
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
IJECEIAES
 
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.pptUnit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
KrishnaveniKrishnara1
 
artificial intelligence and data science contents.pptx
artificial intelligence and data science contents.pptxartificial intelligence and data science contents.pptx
artificial intelligence and data science contents.pptx
GauravCar
 
132/33KV substation case study Presentation
132/33KV substation case study Presentation132/33KV substation case study Presentation
132/33KV substation case study Presentation
kandramariana6
 
CEC 352 - SATELLITE COMMUNICATION UNIT 1
CEC 352 - SATELLITE COMMUNICATION UNIT 1CEC 352 - SATELLITE COMMUNICATION UNIT 1
CEC 352 - SATELLITE COMMUNICATION UNIT 1
PKavitha10
 
AI assisted telemedicine KIOSK for Rural India.pptx
AI assisted telemedicine KIOSK for Rural India.pptxAI assisted telemedicine KIOSK for Rural India.pptx
AI assisted telemedicine KIOSK for Rural India.pptx
architagupta876
 
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
shadow0702a
 
Mechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdfMechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdf
21UME003TUSHARDEB
 
Comparative analysis between traditional aquaponics and reconstructed aquapon...
Comparative analysis between traditional aquaponics and reconstructed aquapon...Comparative analysis between traditional aquaponics and reconstructed aquapon...
Comparative analysis between traditional aquaponics and reconstructed aquapon...
bijceesjournal
 
哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样
哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样
哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样
insn4465
 
An improved modulation technique suitable for a three level flying capacitor ...
An improved modulation technique suitable for a three level flying capacitor ...An improved modulation technique suitable for a three level flying capacitor ...
An improved modulation technique suitable for a three level flying capacitor ...
IJECEIAES
 

Recently uploaded (20)

Seminar on Distillation study-mafia.pptx
Seminar on Distillation study-mafia.pptxSeminar on Distillation study-mafia.pptx
Seminar on Distillation study-mafia.pptx
 
integral complex analysis chapter 06 .pdf
integral complex analysis chapter 06 .pdfintegral complex analysis chapter 06 .pdf
integral complex analysis chapter 06 .pdf
 
BRAIN TUMOR DETECTION for seminar ppt.pdf
BRAIN TUMOR DETECTION for seminar ppt.pdfBRAIN TUMOR DETECTION for seminar ppt.pdf
BRAIN TUMOR DETECTION for seminar ppt.pdf
 
Computational Engineering IITH Presentation
Computational Engineering IITH PresentationComputational Engineering IITH Presentation
Computational Engineering IITH Presentation
 
Design and optimization of ion propulsion drone
Design and optimization of ion propulsion droneDesign and optimization of ion propulsion drone
Design and optimization of ion propulsion drone
 
Applications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdfApplications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdf
 
Properties Railway Sleepers and Test.pptx
Properties Railway Sleepers and Test.pptxProperties Railway Sleepers and Test.pptx
Properties Railway Sleepers and Test.pptx
 
spirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptxspirit beverages ppt without graphics.pptx
spirit beverages ppt without graphics.pptx
 
Certificates - Mahmoud Mohamed Moursi Ahmed
Certificates - Mahmoud Mohamed Moursi AhmedCertificates - Mahmoud Mohamed Moursi Ahmed
Certificates - Mahmoud Mohamed Moursi Ahmed
 
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...
 
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.pptUnit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
Unit-III-ELECTROCHEMICAL STORAGE DEVICES.ppt
 
artificial intelligence and data science contents.pptx
artificial intelligence and data science contents.pptxartificial intelligence and data science contents.pptx
artificial intelligence and data science contents.pptx
 
132/33KV substation case study Presentation
132/33KV substation case study Presentation132/33KV substation case study Presentation
132/33KV substation case study Presentation
 
CEC 352 - SATELLITE COMMUNICATION UNIT 1
CEC 352 - SATELLITE COMMUNICATION UNIT 1CEC 352 - SATELLITE COMMUNICATION UNIT 1
CEC 352 - SATELLITE COMMUNICATION UNIT 1
 
AI assisted telemedicine KIOSK for Rural India.pptx
AI assisted telemedicine KIOSK for Rural India.pptxAI assisted telemedicine KIOSK for Rural India.pptx
AI assisted telemedicine KIOSK for Rural India.pptx
 
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...
 
Mechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdfMechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdf
 
Comparative analysis between traditional aquaponics and reconstructed aquapon...
Comparative analysis between traditional aquaponics and reconstructed aquapon...Comparative analysis between traditional aquaponics and reconstructed aquapon...
Comparative analysis between traditional aquaponics and reconstructed aquapon...
 
哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样
哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样
哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样
 
An improved modulation technique suitable for a three level flying capacitor ...
An improved modulation technique suitable for a three level flying capacitor ...An improved modulation technique suitable for a three level flying capacitor ...
An improved modulation technique suitable for a three level flying capacitor ...
 

A Deep Learning Approach for AR-based Adaptive Simulation using Wearables​

  • 1. RESEARCH POSTER PRESENTATION DESIGN © 2015 www.PosterPresentations.com TimeDomain Features FrequencyDomain Features Simulations enable a risk-free way for healthcare practitioners to learn and enhance their skills. Such simulations should be tailored to the needs of the individual learners. However, most current simulations do not consider the learner’s existing skills, expertise, and ability to handle cognitive load. We propose an end-to-end framework for trauma simulation that actively classifies participant's cognitive load and level of expertise and dynamically modulates the complexity of a simulation by altering symptom severity of a patient using augmented reality (AR) to ensure an optimum amount of cognitive load is maintained. A deep learning approach was taken to utilize electrocardiogram (ECG) and galvanic skin response (GSR) collected using wearable devices. The model was developed to classify high vs low cognitive load, as well as level of expertise, and validated using “leave one out” scheme. Finally, the intelligent dynamically adaptive AR simulation was successfully tested in real time on two participants of each class. Abstract Motivation Data Pre-processing & Feature Extraction Experiment Setup Result and Analysis Conclusion We were able to accurately estimate cognitive load and expertise in real-time and use the classifications to alter a simulation with augmented reality to maintain optimal cognitive load. Pritam Sarkar1, Kyle Ross1, Dirk Rodenburg3, Aaron Ruberto4, Paul Hungler2, Dan Howes4, Adam Szulewski4, Ali Etemad1 1Dept. of Electrical and Computer Engineering, 2Dept. Of Chemical Engineering, 3Faculty of Engineering and Applied Science, 4Dept. of Emergency Medicine Queen’s University A Deep Learning Approach for AR-based Adaptive Simulation using Wearables Cognitive Load Learning Rate Microsoft HoloLens ECG Shimmer Sensor ProposedDynamic SimulationFramework Data Collection Data Visualization DataCollection AR-based Adaptive Simulation ECG based Convolutional Neural Network Feature Extraction from ECG FusedDecision (Probability) Feature Extraction GSR GSR based Neural Network ECG GSR ❑ Many learning systems are one size fits all regarding levels of expertise. ❑ The development of expertise is characterized by the ability to execute increasingly complex tasks with lowered cognitive load. ❑ To be effective, simulations should match complexity with the cognitive abilities of the learner. ❑ Goal: To develop a learning system that dynamically adjusts complexity to match learner expertise and cognitive capacity. GSR Shimmer Sensor TSNE plot of GSR feature-space without (left) and with (right) baseline correction TSNE plot of ECG feature-space without (left) and with (right) baseline correction Expertise Cognitive Load Testing (Offline) 91% 74% Realtime Testing 2 participantsof each class were successfully tested to classify level of expertise and cognitive load. We used an augmented reality overlayon top of a mannequinto change the difficulty of the simulationthrough patient symptomology Participants during simulation anne u n stra tor View from HoloLens during simulation RR peaks for ECG feature extraction SCR response for GSR feature extraction Features Publications ➢ Conference: Classification of Expertise and Cognitive Load in Adaptive AR-Based Simulation using Deep Learning, Affective Computing & Intelligent Interaction (ACII), 2019 ➢ Journal: Toward Dynamically Adaptive Simulation: Multimodal Classification of User Expertise using Wearable Devices, Sensors 2019, 19, 4270 ➢ Expert participants (Experienced in emergency medicine) ➢ Nov e part pants (Queen’s 4th year medical students) (Left) Plot of the likelihood for expertise vs cognitive load. The results show an inverse relationship between level of expertise and cognitive load, where the top left corner shows high level of expertise and low cognitive load, and the bottom right corner indicates low expertise and high cognitive load. Reach us at: pritam.sarkar@queensu.ca, 12kjr1@queensu.ca