SlideShare a Scribd company logo
1 of 2
Download to read offline
Behavioral and Physiological Signals
Based Deep Multimodal Approach for
Mobile Emotion Recognition
Abstract
With the rapid development of mobile and wearable devices, it is increasingly
possible to access users’ affective data in a more unobtrusive manner. On this
basis, researchers have proposed various systems to recognize user’s
emotional states. However, mos
learning techniques and a limited number of signals, leading to systems that
either do not generalize well or would frequently lack sufficient information for
emotion detection in realistic scenarios. In this
attention-based LSTM system that uses a combination of sensors from a
smartphone (front camera, microphone, touch panel) and a wristband
(photoplethysmography, electrodermal activity, and infrared thermopile
sensor) to accurately determine user’s emotional states. We evaluated the
proposed system by conducting a user study with 45 participants. Using
collected behavioral (facial expression, speech, keystroke) and physiological
Behavioral and Physiological Signals
Based Deep Multimodal Approach for
Mobile Emotion Recognition
With the rapid development of mobile and wearable devices, it is increasingly
possible to access users’ affective data in a more unobtrusive manner. On this
basis, researchers have proposed various systems to recognize user’s
emotional states. However, most of these studies rely on traditional machine
learning techniques and a limited number of signals, leading to systems that
either do not generalize well or would frequently lack sufficient information for
emotion detection in realistic scenarios. In this paper, we propose a novel
based LSTM system that uses a combination of sensors from a
smartphone (front camera, microphone, touch panel) and a wristband
(photoplethysmography, electrodermal activity, and infrared thermopile
determine user’s emotional states. We evaluated the
proposed system by conducting a user study with 45 participants. Using
collected behavioral (facial expression, speech, keystroke) and physiological
Behavioral and Physiological Signals-
Based Deep Multimodal Approach for
With the rapid development of mobile and wearable devices, it is increasingly
possible to access users’ affective data in a more unobtrusive manner. On this
basis, researchers have proposed various systems to recognize user’s
t of these studies rely on traditional machine
learning techniques and a limited number of signals, leading to systems that
either do not generalize well or would frequently lack sufficient information for
paper, we propose a novel
based LSTM system that uses a combination of sensors from a
smartphone (front camera, microphone, touch panel) and a wristband
(photoplethysmography, electrodermal activity, and infrared thermopile
determine user’s emotional states. We evaluated the
proposed system by conducting a user study with 45 participants. Using
collected behavioral (facial expression, speech, keystroke) and physiological
(blood volume, electrodermal activity, skin temperature) affective responses
induced by visual stimuli, our system was able to achieve an average
accuracy of 89.2 percent for binary positive and negative emotion
classification under leave-one-participant-out cross-validation. Furthermore,
we investigated the effectiveness of different combinations of data signals to
cover different scenarios of signal availability.

More Related Content

Similar to Behavioral and Physiological Signals-Based Deep Multimodal Approach for Mobile Emotion Recognition.pdf

Affective analysis in machine learning using AMIGOS with Gaussian expectatio...
Affective analysis in machine learning using AMIGOS with  Gaussian expectatio...Affective analysis in machine learning using AMIGOS with  Gaussian expectatio...
Affective analysis in machine learning using AMIGOS with Gaussian expectatio...
International Journal of Reconfigurable and Embedded Systems
 
PPT on mind reading computer
 PPT on mind reading computer PPT on mind reading computer
PPT on mind reading computer
Anjali Agarwal
 
Iaetsd an effective alarming model for danger and activity
Iaetsd an effective alarming model for danger and activityIaetsd an effective alarming model for danger and activity
Iaetsd an effective alarming model for danger and activity
Iaetsd Iaetsd
 

Similar to Behavioral and Physiological Signals-Based Deep Multimodal Approach for Mobile Emotion Recognition.pdf (20)

Blue eyes technology New Version 2017
Blue eyes technology New Version 2017Blue eyes technology New Version 2017
Blue eyes technology New Version 2017
 
Emotion Detection Using Noninvasive Low-cost Sensors
Emotion Detection Using Noninvasive Low-cost SensorsEmotion Detection Using Noninvasive Low-cost Sensors
Emotion Detection Using Noninvasive Low-cost Sensors
 
A survey on bio-signal analysis for human-robot interaction
A survey on bio-signal analysis for human-robot interactionA survey on bio-signal analysis for human-robot interaction
A survey on bio-signal analysis for human-robot interaction
 
Kinect Sensor based Indian Sign Language Detection with Voice Extraction
Kinect Sensor based Indian Sign Language Detection with Voice ExtractionKinect Sensor based Indian Sign Language Detection with Voice Extraction
Kinect Sensor based Indian Sign Language Detection with Voice Extraction
 
Abstract.docx
Abstract.docxAbstract.docx
Abstract.docx
 
Affective analysis in machine learning using AMIGOS with Gaussian expectatio...
Affective analysis in machine learning using AMIGOS with  Gaussian expectatio...Affective analysis in machine learning using AMIGOS with  Gaussian expectatio...
Affective analysis in machine learning using AMIGOS with Gaussian expectatio...
 
Io t research_niit_durgapur
Io t research_niit_durgapurIo t research_niit_durgapur
Io t research_niit_durgapur
 
Data Analytics Project proposal: Smart home based ambient assisted living - D...
Data Analytics Project proposal: Smart home based ambient assisted living - D...Data Analytics Project proposal: Smart home based ambient assisted living - D...
Data Analytics Project proposal: Smart home based ambient assisted living - D...
 
Georgetown Innovation Center for Biomedical Informatics Symposium Precision ...
 Georgetown Innovation Center for Biomedical Informatics Symposium Precision ... Georgetown Innovation Center for Biomedical Informatics Symposium Precision ...
Georgetown Innovation Center for Biomedical Informatics Symposium Precision ...
 
Ieeepro techno solutions 2013 ieee embedded project - child activity recog...
Ieeepro techno solutions   2013 ieee embedded project  - child activity recog...Ieeepro techno solutions   2013 ieee embedded project  - child activity recog...
Ieeepro techno solutions 2013 ieee embedded project - child activity recog...
 
SOPRANO
SOPRANOSOPRANO
SOPRANO
 
Ecis final paper-june2017_two way architecture between iot sensors and cloud ...
Ecis final paper-june2017_two way architecture between iot sensors and cloud ...Ecis final paper-june2017_two way architecture between iot sensors and cloud ...
Ecis final paper-june2017_two way architecture between iot sensors and cloud ...
 
Human activity recognition updated 1 - Copy.pptx
Human activity recognition updated 1 - Copy.pptxHuman activity recognition updated 1 - Copy.pptx
Human activity recognition updated 1 - Copy.pptx
 
Wearable sensor-based human activity recognition with ensemble learning: a co...
Wearable sensor-based human activity recognition with ensemble learning: a co...Wearable sensor-based human activity recognition with ensemble learning: a co...
Wearable sensor-based human activity recognition with ensemble learning: a co...
 
Health monitoring catalogue based on human activity classification using mac...
Health monitoring catalogue based on human activity  classification using mac...Health monitoring catalogue based on human activity  classification using mac...
Health monitoring catalogue based on human activity classification using mac...
 
Depression Screening in Humans With AI and Deep Learning Techniques.pdf
Depression Screening in Humans With AI and Deep Learning Techniques.pdfDepression Screening in Humans With AI and Deep Learning Techniques.pdf
Depression Screening in Humans With AI and Deep Learning Techniques.pdf
 
PPT on mind reading computer
 PPT on mind reading computer PPT on mind reading computer
PPT on mind reading computer
 
Journal Publication
Journal PublicationJournal Publication
Journal Publication
 
PhD research publications
PhD research publicationsPhD research publications
PhD research publications
 
Iaetsd an effective alarming model for danger and activity
Iaetsd an effective alarming model for danger and activityIaetsd an effective alarming model for danger and activity
Iaetsd an effective alarming model for danger and activity
 

More from OKOKPROJECTS

More from OKOKPROJECTS (20)

Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...
Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...
Distributed State Estimation With Deep Neural Networks for Uncertain Nonlinea...
 
Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...
Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...
Distributed Inference in Resource-Constrained IoT for Real-Time Video Surveil...
 
DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...
DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...
DLTIF Deep Learning-Driven Cyber Threat Intelligence Modeling and Identificat...
 
DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...
DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...
DGSSC A Deep Generative Spectral-Spatial Classifier for Imbalanced Hyperspect...
 
Digital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdf
Digital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdfDigital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdf
Digital Restoration of Cultural Heritage With Data-Driven Computing A Survey.pdf
 
Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...
Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...
Dependable Intrusion Detection System for IoT A Deep Transfer Learning Based ...
 
DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...
DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...
DendroMap Visual Exploration of Large-Scale Image Datasets for Machine Learni...
 
Dense Nested Attention Network for Infrared Small Target Detection.pdf
Dense Nested Attention Network for Infrared Small Target Detection.pdfDense Nested Attention Network for Infrared Small Target Detection.pdf
Dense Nested Attention Network for Infrared Small Target Detection.pdf
 
Detection of Small Moving Targets in Cluttered Infrared Imagery.pdf
Detection of Small Moving Targets in Cluttered Infrared Imagery.pdfDetection of Small Moving Targets in Cluttered Infrared Imagery.pdf
Detection of Small Moving Targets in Cluttered Infrared Imagery.pdf
 
DeepTx Deep Learning Beamforming With Channel Prediction.pdf
DeepTx Deep Learning Beamforming With Channel Prediction.pdfDeepTx Deep Learning Beamforming With Channel Prediction.pdf
DeepTx Deep Learning Beamforming With Channel Prediction.pdf
 
DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...
DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...
DeHIN A Decentralized Framework for Embedding Large-Scale Heterogeneous Infor...
 
DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...
DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...
DefQ Defensive Quantization Against Inference Slow-Down Attack for Edge Compu...
 
Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...
Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...
Deep-Learning-Driven Proactive Maintenance Management of IoT-Empowered Smart ...
 
Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...
Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...
Deep-Distributed-Learning-Based POI Recommendation Under Mobile-Edge Networks...
 
DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...
DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...
DeepCog A Trustworthy Deep Learning-Based Human Cognitive Privacy Framework i...
 
DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...
DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...
DeepCrowd A Deep Model for Large-Scale Citywide Crowd Density and Flow Predic...
 
D2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdf
D2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdfD2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdf
D2Net Deep Denoising Network in Frequency Domain for Hyperspectral Image.pdf
 
Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...
Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...
Decentralized Federated Learning for Industrial IoT With Deep Echo State Netw...
 
Cyber Code Intelligence for Android Malware Detection.pdf
Cyber Code Intelligence for Android Malware Detection.pdfCyber Code Intelligence for Android Malware Detection.pdf
Cyber Code Intelligence for Android Malware Detection.pdf
 
CSKG4APT A Cybersecurity Knowledge Graph for Advanced Persistent Threat Organ...
CSKG4APT A Cybersecurity Knowledge Graph for Advanced Persistent Threat Organ...CSKG4APT A Cybersecurity Knowledge Graph for Advanced Persistent Threat Organ...
CSKG4APT A Cybersecurity Knowledge Graph for Advanced Persistent Threat Organ...
 

Recently uploaded

Salient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functionsSalient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functions
KarakKing
 

Recently uploaded (20)

21st_Century_Skills_Framework_Final_Presentation_2.pptx
21st_Century_Skills_Framework_Final_Presentation_2.pptx21st_Century_Skills_Framework_Final_Presentation_2.pptx
21st_Century_Skills_Framework_Final_Presentation_2.pptx
 
Google Gemini An AI Revolution in Education.pptx
Google Gemini An AI Revolution in Education.pptxGoogle Gemini An AI Revolution in Education.pptx
Google Gemini An AI Revolution in Education.pptx
 
Tatlong Kwento ni Lola basyang-1.pdf arts
Tatlong Kwento ni Lola basyang-1.pdf artsTatlong Kwento ni Lola basyang-1.pdf arts
Tatlong Kwento ni Lola basyang-1.pdf arts
 
SOC 101 Demonstration of Learning Presentation
SOC 101 Demonstration of Learning PresentationSOC 101 Demonstration of Learning Presentation
SOC 101 Demonstration of Learning Presentation
 
Single or Multiple melodic lines structure
Single or Multiple melodic lines structureSingle or Multiple melodic lines structure
Single or Multiple melodic lines structure
 
Python Notes for mca i year students osmania university.docx
Python Notes for mca i year students osmania university.docxPython Notes for mca i year students osmania university.docx
Python Notes for mca i year students osmania university.docx
 
Accessible Digital Futures project (20/03/2024)
Accessible Digital Futures project (20/03/2024)Accessible Digital Futures project (20/03/2024)
Accessible Digital Futures project (20/03/2024)
 
How to Add New Custom Addons Path in Odoo 17
How to Add New Custom Addons Path in Odoo 17How to Add New Custom Addons Path in Odoo 17
How to Add New Custom Addons Path in Odoo 17
 
Food safety_Challenges food safety laboratories_.pdf
Food safety_Challenges food safety laboratories_.pdfFood safety_Challenges food safety laboratories_.pdf
Food safety_Challenges food safety laboratories_.pdf
 
Basic Intentional Injuries Health Education
Basic Intentional Injuries Health EducationBasic Intentional Injuries Health Education
Basic Intentional Injuries Health Education
 
Jamworks pilot and AI at Jisc (20/03/2024)
Jamworks pilot and AI at Jisc (20/03/2024)Jamworks pilot and AI at Jisc (20/03/2024)
Jamworks pilot and AI at Jisc (20/03/2024)
 
How to setup Pycharm environment for Odoo 17.pptx
How to setup Pycharm environment for Odoo 17.pptxHow to setup Pycharm environment for Odoo 17.pptx
How to setup Pycharm environment for Odoo 17.pptx
 
Interdisciplinary_Insights_Data_Collection_Methods.pptx
Interdisciplinary_Insights_Data_Collection_Methods.pptxInterdisciplinary_Insights_Data_Collection_Methods.pptx
Interdisciplinary_Insights_Data_Collection_Methods.pptx
 
80 ĐỀ THI THỬ TUYỂN SINH TIẾNG ANH VÀO 10 SỞ GD – ĐT THÀNH PHỐ HỒ CHÍ MINH NĂ...
80 ĐỀ THI THỬ TUYỂN SINH TIẾNG ANH VÀO 10 SỞ GD – ĐT THÀNH PHỐ HỒ CHÍ MINH NĂ...80 ĐỀ THI THỬ TUYỂN SINH TIẾNG ANH VÀO 10 SỞ GD – ĐT THÀNH PHỐ HỒ CHÍ MINH NĂ...
80 ĐỀ THI THỬ TUYỂN SINH TIẾNG ANH VÀO 10 SỞ GD – ĐT THÀNH PHỐ HỒ CHÍ MINH NĂ...
 
How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17
 
General Principles of Intellectual Property: Concepts of Intellectual Proper...
General Principles of Intellectual Property: Concepts of Intellectual  Proper...General Principles of Intellectual Property: Concepts of Intellectual  Proper...
General Principles of Intellectual Property: Concepts of Intellectual Proper...
 
Unit 3 Emotional Intelligence and Spiritual Intelligence.pdf
Unit 3 Emotional Intelligence and Spiritual Intelligence.pdfUnit 3 Emotional Intelligence and Spiritual Intelligence.pdf
Unit 3 Emotional Intelligence and Spiritual Intelligence.pdf
 
Salient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functionsSalient Features of India constitution especially power and functions
Salient Features of India constitution especially power and functions
 
Graduate Outcomes Presentation Slides - English
Graduate Outcomes Presentation Slides - EnglishGraduate Outcomes Presentation Slides - English
Graduate Outcomes Presentation Slides - English
 
On National Teacher Day, meet the 2024-25 Kenan Fellows
On National Teacher Day, meet the 2024-25 Kenan FellowsOn National Teacher Day, meet the 2024-25 Kenan Fellows
On National Teacher Day, meet the 2024-25 Kenan Fellows
 

Behavioral and Physiological Signals-Based Deep Multimodal Approach for Mobile Emotion Recognition.pdf

  • 1. Behavioral and Physiological Signals Based Deep Multimodal Approach for Mobile Emotion Recognition Abstract With the rapid development of mobile and wearable devices, it is increasingly possible to access users’ affective data in a more unobtrusive manner. On this basis, researchers have proposed various systems to recognize user’s emotional states. However, mos learning techniques and a limited number of signals, leading to systems that either do not generalize well or would frequently lack sufficient information for emotion detection in realistic scenarios. In this attention-based LSTM system that uses a combination of sensors from a smartphone (front camera, microphone, touch panel) and a wristband (photoplethysmography, electrodermal activity, and infrared thermopile sensor) to accurately determine user’s emotional states. We evaluated the proposed system by conducting a user study with 45 participants. Using collected behavioral (facial expression, speech, keystroke) and physiological Behavioral and Physiological Signals Based Deep Multimodal Approach for Mobile Emotion Recognition With the rapid development of mobile and wearable devices, it is increasingly possible to access users’ affective data in a more unobtrusive manner. On this basis, researchers have proposed various systems to recognize user’s emotional states. However, most of these studies rely on traditional machine learning techniques and a limited number of signals, leading to systems that either do not generalize well or would frequently lack sufficient information for emotion detection in realistic scenarios. In this paper, we propose a novel based LSTM system that uses a combination of sensors from a smartphone (front camera, microphone, touch panel) and a wristband (photoplethysmography, electrodermal activity, and infrared thermopile determine user’s emotional states. We evaluated the proposed system by conducting a user study with 45 participants. Using collected behavioral (facial expression, speech, keystroke) and physiological Behavioral and Physiological Signals- Based Deep Multimodal Approach for With the rapid development of mobile and wearable devices, it is increasingly possible to access users’ affective data in a more unobtrusive manner. On this basis, researchers have proposed various systems to recognize user’s t of these studies rely on traditional machine learning techniques and a limited number of signals, leading to systems that either do not generalize well or would frequently lack sufficient information for paper, we propose a novel based LSTM system that uses a combination of sensors from a smartphone (front camera, microphone, touch panel) and a wristband (photoplethysmography, electrodermal activity, and infrared thermopile determine user’s emotional states. We evaluated the proposed system by conducting a user study with 45 participants. Using collected behavioral (facial expression, speech, keystroke) and physiological
  • 2. (blood volume, electrodermal activity, skin temperature) affective responses induced by visual stimuli, our system was able to achieve an average accuracy of 89.2 percent for binary positive and negative emotion classification under leave-one-participant-out cross-validation. Furthermore, we investigated the effectiveness of different combinations of data signals to cover different scenarios of signal availability.