The document proposes a dense optical flow-based approach to emotion recognition from videos. It extracts dense optical flow features from labeled training videos to train an SVM classifier. During testing, it determines facial movements from unlabeled videos using optical flow and classifies the emotion. It achieved 82-90% accuracy on a dataset of 372 videos across 6 people expressing 4 emotions. Future work includes combining classifiers focusing on different facial regions and testing robustness to distance and camera angle.

1. Dense Optical Flow Based
Emotion Recognition
By Anthony Lowhur (under Professor Chuah)

2. Outline of talk
• Motivation
• Limitations of existing methods
• Overview of our Dense Optical Flow-Based ER
(DOF-ER) approach
• Experimental Evaluation

3. Why emotion recognition?
• Need more effective human-computer interaction
methods in more intelligent systems
• Video Games which depend on users’ emotions
• Educational packages which depend on users’ emotions
• Healthcare applications for treating patients with PTSD or
depressions.
• Counseling and therapy applications that would help one
to optimize psychologicalogical treatment in patients
• Musical applications to autonomously play the right music
for the user at a certain moment.

4. Limitation of Existing Methods
• Still image emotion classifiers suffers from
weaknesses such as bad lighting, the subject’s
unique appearance, and unique facial expression.
• Other classifiers assume the presence of a fully
frontal face location with previous knowledge of
global facial landmark location.

5. Dense Optical Flow-Based Emotional Recognition
Consists of Training and Testing Phase:
During Training
• we extract dense optical flow-based features from labeled videos of users’
different emotions.
• Use extracted features to train a SVM-based classifier.
During Testing
• Using a sliding window of 15 frames, determine if large displacement of facial
movements can be identified from a video of a user’s unknown emotion
• Then, extract dense optical flow-based features and feed into trained SVM
classifier to determine the emotion type
Training Phase
Testing Phase

6. Dense Optical Flow
• Uses Gunner Farneback's optical flow algorithm to track the
facial movements.
• Accomplishes this by checking the surrounding of each
optical flow point in the grid.
• Relies on the contrast on each frame where if there is a
difference in contrast, that indicates that there are
movement.

7. Advantages of our approach
• Analyzing facial movements in videos as oppose to still
images.
• This method is less vulnerable to numerous factors such
as lighting and different appearance of each people
• Our approach makes it easier to classify emotions since
people generally have similar facial movements and
expression patterns for each emotion

8. Details of Evaluation Experiment
l We collected facial data from all of our subjects
l Perform dense optical flow analysis on the
images
l Train the emotion classifier with SVM with the
optical flow point data
l Predict someone’s emotion by recording them
and analyzing dense optical flow movement.

9. Data Collection
l Asked 6 volunteers to allow us to take pictures of them
with four posed emotions.
l Produced own dataset of 372 videos of expressing each
emotion from 6 different people.
l Emotion labels are neutral, happy, surprised, and angry
l More emotions can be easily added by adding more
video samples
l Data was diverse enough so that each emotion had
different facial movements.

10. Evaluation Results
• Successfully recognized the emotions of a large
majority of the test subjects.
• With a large and diverse dataset, the classifier
achieved an accuracy range of 82-90% with
cross-validation.
• The confusion matrix tells us that the classifier
has a tendency to over guess neutral due to small
facial movements in other emotions. However
sensitive mode overcomes this.

11. (a) Neutral (b) Smiling
(c) Angry (d) Surprised

12. Amount of flow points
l We tested to see how our normal 26,912 optical
flow point emotion classifier compares to one with
10,000 optical flow points.
l The one with 26,912 flow points seemed to have
better performance with an accuracy of 81.75%
compared to the other classifier's 74.375%
l

13. Flow Points Distribution with 26912 flows

14. Generic vs Per-User Classifier
• Also evaluate differences between a per-user and
generic classifier
• Per-user classifier can achieve an accuracy in the
range of 81-87% compared to the 81.25% with
generic classifier.
l

15. Future Work
• Investigate if using a combination of different
classifiers with each classifier focusing on facial
movements in a particular region e.g. eyebrows or
mouth) gives higher accuracy.
• Investigate the robustness of our scheme when a
user’s facial expression is captured at different
distance from the camera e.g. near or far views,
front or side view of a user’s face.

16. Video of emotion classifier
https://www.youtube.com/watch?v=PM_k_zACWLE

17. Packages used
l OpenCV Python
l Computer Vision tasks such as haar cascade and
dense optical flow
l Scikit-Learn
l Machine Learning tasks such as Support Vector
Machines
l Numpy, matplotlib, etc
l Work with matrices and analyze data
l LibSVM (will add more info once I used it)

18. Acknowledgement
l We would like to thank all of our subjects who
allowed us to take emotion video samples from
them.