Emotion Recognition
F e l i c i t o u s C o m p u t i n g I n s t i t u t e visit to
Deb * Mal * Ya Sin * Ha
Means of Emotion Recognition
Speech
Gestures
Facial Feature
Heartbeat
Skin Conductance
Brain Imagery
Movement Features
Han...
Active User Participation
Speech
Gestures
Facial Feature
Heartbeat
Skin Conductance
Brain Imagery
Movement Features
Hand W...
Passive User Participation
Speech
Gestures
Facial Feature
Heartbeat
Skin Conductance
Brain Imagery
Movement Features
Hand ...
Active Passive
Transition
Other two Key Factors for the Transition:
1. Device invisibility
2. Sensor distance
Over time us...
Device invisibility
Instead of
An invisible tracker/sensor device sits in
the background and do work for the users
Problem...
Sensor Distance
• Facial Feature
• Heartbeat
• Skin Conductance
• Brain Imagery
• Movement and
Gestures
• Bio Chemical
• E...
Passive, Ambient Sensors
• Facial Feature
• Eye & pupil
• Movement and
Gestures
needs
• Focus
• Facing to camera
• A degre...
Movements and Gestures: A scenario
Situations where body movements and gestures are crucial:
1. A Post Traumatic Stress Di...
HaiXiu - 害羞
Records gestures
and movement
Comes up with
unique feature set
Trains a Neural Net
for later detection
Continu...
HaiXiu - 害羞
• Microsoft Kinect™ for movement detection
• Rather than discreet affective states, our target
is to detect Ar...
Feature Set for Arousal level detection
Kinect gives us 20 Different Joint position data
We Calculate:
1. Minimum coordina...
Training the Neural Net
Initially we took 20 movement features (without the position features) and
told 2 subjects to walk...
Challenges
1. Short working range of Kinect : .8m to 4.0m
2. Shorter than the range needed in practical scenarios
3. Data ...
Next Step
1. Introducing the Position CoOrdinates
2. Fine tune the Arousal level recognizer
3. A Robust Gesture recognitio...
Integrated Emotion detection
1. Every one of the modes of recognition have their merits
2. There are a plethora of existin...
Thank You
Upcoming SlideShare
Loading in …5
×

HaiXiu: Emotion Recognition from Movements

486 views

Published on

This gives an overview of HaiXiu, a software that detects arousal levels from body movements and how it connects to the other means of emotion detection techniques. The presentation was given at Microsoft Research Asia, Beijing.

Published in: Technology, Health & Medicine
1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
486
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
15
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

HaiXiu: Emotion Recognition from Movements

  1. 1. Emotion Recognition F e l i c i t o u s C o m p u t i n g I n s t i t u t e visit to Deb * Mal * Ya Sin * Ha
  2. 2. Means of Emotion Recognition Speech Gestures Facial Feature Heartbeat Skin Conductance Brain Imagery Movement Features Hand Writing Dance Valance Arousal High High Low Low Anger LoveSadness Joy Social Presence Bio Chemical Eye & pupil
  3. 3. Active User Participation Speech Gestures Facial Feature Heartbeat Skin Conductance Brain Imagery Movement Features Hand Writing Dance Valance Arousal High High Low Low Anger LoveSadness Joy Social Presence Bio Chemical Eye & pupil •Users usually have to consciously take part in it •Chances of suppressing emotive cues •Chances of showing inverted affective state •Tend to be biased
  4. 4. Passive User Participation Speech Gestures Facial Feature Heartbeat Skin Conductance Brain Imagery Movement Features Hand Writing Dance Valance Arousal High High Low Low Anger LoveSadness Joy Social Presence Bio Chemical Eye & pupil • Users do not need to take part actively • Users have much less control over the results • Less attention and control means less bias
  5. 5. Active Passive Transition Other two Key Factors for the Transition: 1. Device invisibility 2. Sensor distance Over time users tend to get familiarized •The actions slowly gets more passive •Controlling tendency decreases •Faking of emotion decreases
  6. 6. Device invisibility Instead of An invisible tracker/sensor device sits in the background and do work for the users Problem of devices with direct contact: • In various situations users will not have the mindset to actively engage (Trauma, Sadness, Forlorn) • Sensor Distance is important in these situations MoodScope
  7. 7. Sensor Distance • Facial Feature • Heartbeat • Skin Conductance • Brain Imagery • Movement and Gestures • Bio Chemical • Eye & pupil With External Attachments • No body attachments • Large Sensor Distance • Can be operated more passively • Much more unconscious participation • Bias can be more minimized Without Attachments Modes of Passive Recognizers
  8. 8. Passive, Ambient Sensors • Facial Feature • Eye & pupil • Movement and Gestures needs • Focus • Facing to camera • A degree of attachment to sensors In many cases where: 1. the face is not visible 2. there is no provision for attaching sensors to body 3. there is no speech input The movement and gesture detection is much more feasible to detect affect
  9. 9. Movements and Gestures: A scenario Situations where body movements and gestures are crucial: 1. A Post Traumatic Stress Disorder (PTSD) patient pacing in the room. 2. A schizophrenic patient at an asylum is going impatient and angry and doing frivolous, jerky movements. 3. A patient of Chronic Depression is seen pacing slowly, hands in pocket, head drooping. An Automated system that detects emotive states in such situations, can even save lives.
  10. 10. HaiXiu - 害羞 Records gestures and movement Comes up with unique feature set Trains a Neural Net for later detection Continuous Emotion Detection
  11. 11. HaiXiu - 害羞 • Microsoft Kinect™ for movement detection • Rather than discreet affective states, our target is to detect Arousal and Valence Levels in continuous space. • This model of continuous affective level detection can be implemented with other continuous affective spaces. e.g: Plutchik’s Emotion Wheel, PAD model • Presently HaiXiu detects only Arousal levels. Work is going on to include the Valence level. Valance Arousal High High Low Low Anger LoveSadness Joy
  12. 12. Feature Set for Arousal level detection Kinect gives us 20 Different Joint position data We Calculate: 1. Minimum coordinates for X , Y and Z axis (Relative to spine) 2. Maximum coordinates for X , Y and Z axis (Relative to spine) 3. Speed = Δs/Δt 4. Peak Acceleration = Δu/Δt 5. Peak Deceleration = - Δu/Δt 6. Average Acceleration = (Σ (Δu/Δt))/f 7. Average Deceleration = - (Σ (Δu/Δt))/f 8. Jerk Index = (Σ (Δa/Δt))/f Δt = 0.2 second; f = total time / Δt
  13. 13. Training the Neural Net Initially we took 20 movement features (without the position features) and told 2 subjects to walk in various arousal levels. We measured Speed, Accel, Decel, JerkIndex for upper body joints. Type: Bipolar Feedforward ANN Layers: 3 (20 : 6: 1) Learning: Backpropagation Learning Sample Size: 34 Walks (in different arousal levels) of 2 subjects Error Limit of learned Net: 0.0956 Detection The ANN outputs one variable for Arousal Level The output range is from -1 (totally relaxed) to +1 (Very Aroused)
  14. 14. Challenges 1. Short working range of Kinect : .8m to 4.0m 2. Shorter than the range needed in practical scenarios 3. Data not consistent enough for precise movement feature Calculation 4. Fault Tolerance in case of recording and detection is needed. 5. Kinect does not follow BVH format thus available gesture databases in BVH can not be natively used without a converter module (less efficiency)
  15. 15. Next Step 1. Introducing the Position CoOrdinates 2. Fine tune the Arousal level recognizer 3. A Robust Gesture recognition module 4. Building a Valence recognizer module 5. Getting more test data with more number of subjects 6. Multiple Kinect integration for better recognition 7. A slightly better user interface Valance Arousal High High Low Low Anger LoveSadness Joy
  16. 16. Integrated Emotion detection 1. Every one of the modes of recognition have their merits 2. There are a plethora of existing facial expression detectors like “affectiva” 3. Speech based emotion recognition has also been extensively done 4. MoodScope has changed the smartphone based affect detection 5. Powerful tools like AmbientDynamix makes integration of various sensor inputs ease for processing and using in small devices like a smartphone +
  17. 17. Thank You

×