This document discusses facial expression recognition and the challenges that remain. It provides an overview of the current state-of-the-art techniques for facial expression recognition, which still struggle with accuracy when tested on naturalistic data rather than posed images. The document outlines a proposed pipeline for facial expression recognition that combines deep learning techniques for feature fusion and representation learning to help address these challenges and improve recognition accuracy on real-world data. Samples of datasets used for training and evaluating facial expression recognition systems are also presented.

1. Facial Expression
Recognition in the Wild
Adil Khan
Machine Learning & Knowledge Representation Lab
Institute of AI & Robotics
Innopolis University

2. About ME!
Associate Prof. Innopolis University (Russia)
Head of Machine Learning & Knowledge
Representation Lab
Machine Learning, Computer Vision, Pervasive Computing

3. Why FER?
We are emotional beings! Happy → Pizza BUT Sad → Ice-cream

4. Why FER?
Most Effective Way to Understand Emotions:
A. Mehrabian, Nonverbal Communication. New Brunswick, NJ, USA: Aldine, 2007.

5. Two Types: Macro vs. Micro

6. Anger in Two Types!

7. FER: An Extensively Researched Area!
Facial expression recognition using
radial encoding of local Gabor features
and classifier synthesis
Compound facial expressions of
emotion
Rapid perceptual integration of facial
expression and emotional body
language
Simulationist models of face-based
emotion recognition
Emotion recognition in human
computer interaction
Real-time Mobile Facial Expression
Recognition System

8. What Challenges Still Remain?

9. State-of-the-Art When Tested on Naturalistic Data!
FER System Accuracy on ND
AH-ASM 50 ± 4.5%
W-BPNN 62 ± 3.1%
LDN-SVM 64 ± 2.9%
CLM-SVM 60 ± 3.5%
RLBP-NN 64 ± 1.7%
PHOG-SVM 59 ± 5.6%

10. Samples from Naturalistic Dataset

11. Samples from Available Datasets for Training
Static Facial Expressions in the Wild (SFEW) Dataset

12. State-of-the-Art FER is Not Ready for The Real World!

13. FER Pipeline
Pre-Processing Face Detection Facial Components
Feature Extraction Classification Recognized
Expression Class
Image or
Video

14. Journey Begins With Data!
https://link.springer.com/article/10.1007/s11042-016-4321-2

15. We Combine but Combine Intelligently!
http://www.araya.org/wp/wp-content/uploads/2016/10/gan.png

16. Features: Are Very Important; Already Got Some!
Two broad categories:
1. Geometric Features
 ASM, AAM
2. Appearance based features
 GF, LBP, LDA, HOGs, etc.

17. Feature Fusion Has Been Tried; Works to Some
Extent
1. Simple concatenation
2. Spectral Embedding
3. Genetic Algorithms
4. Multi Kernel Learning
5. Ensemble of classifiers
6. etc.
Fusion
Shape Info Texture Info
Fused Feature Vector

18. We use Deep Learning for Feature Fusion

19. Still Variance Might Creep In!

20. Two Level is Good, but Too Naive/Heuristic!
http://uclab.khu.ac.kr/resources/publication/J_201.pdf

21. We combine: One Network for Both Tasks!
A Deep Network
Cluster learning at the first layers
Representation Learning within each cluster at the subsequent layers
Classification at the last layer
But have to figure out how to backprop

22. We Test on the REAL Data!
The biggest challenge, but we are trying
We collect our own real, spontaneous expression data
1. Online resources
2. Offline resources

23. Machine learning and Knowledge
Representation Lab
a.khan@innopolis.ru