Automated analysis of emotions and affects is an inherently challenging problem with a broad range of applications in Human-Computer Interaction (HCI), health informatics, assistive technologies and multimedia retrieval. Understanding human's specific and basic emotions and reacting accordingly can improve HCI. Besides, giving machines skills to understand human's emotions when interacting with other humans can help humans with a socio-affective intelligence. In this talk, I will present deep neural networks-based approaches to analyse speech and face images to analyse human emotions and mental states.

1. A l i c e O T H M A N I , P h D
M a î t r e d e c o n f é r e n c e s à l ’ U P E C
E m a i l : a l i c e . o t h m a n i @ u - p e c . f r
Artificial Emotional Intelligence: Human Emotions
Understanding for Human-Computer Interaction and
for the diagnosis of mental states

2. Artificial Emotional Intelligence
Artificial Emotional Intelligence (EI) or
affective computing is the science of
recognizing, interpreting, processing, and/or
simulating human affects

3. Artificial Emotional Intelligence
Affect describes the experience of a
human's emotion resulting from an
interaction with stimuli.

4. Artificial Emotional Intelligence
Humans express an affect through facial,
vocal, or gestural behaviors.

5. Artificial Emotional Intelligence
▪ Giving machines skills of emotional
intelligence is an important key to
improve Human-Computer Interaction
(HCI) or to understand human's
interactions with other humans.
▪ Understand human's emotions when
interacting with other humans can help
humans with a socio-affective
intelligence.

6. Artificial Emotional Intelligence
Artificial Emotional Intelligence is an
inherently challenging problem with a
broad range of applications in Human-
Computer Interaction (HCI), health
informatics, assistive technologies and
multimedia retrieval.

7. Artificial Emotional Intelligence
In this talk, I will present deep neural
networks-based approaches :
▪ to analyse speech
▪ and face images
to analyse human emotions and mental
states.

8. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images
Amine Djerghri, Ahmed Rachid Hazourli, Alice Othmani: Deep Multi-Facial patches Aggregation Network for Expression
Classification from Face Images. CoRR abs/1909.10305 (2019)

9. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images

10. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images
Multi-Facial Patches based Convolutional Neural Networks
(MFP-CNN). For each image, the face is detected and
aligned. Facial landmarks are extracted from each aligned
face. Facial patches are extracted around facial landmarks
and they are fed each to a sub-network.

11. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images
The structure of each sub-network for each patch. The
input of each sub-network is a facial patch and the output
is fused in the concatenate layer.

12. Conditional Generative Adversarial Network for
Facial Expression generation
cGAN consists of two 'adversarial' models
▪ Generator G: that captures the data distribution,
▪ Discriminator D: that estimates the probability that a
sample come from the training data rather than G.
The generator maximizes the log-probability of labeling real
and fake images correctly while the generator minimizes it.

13. Conditional Generative Adversarial Network for
Facial Expression generation
The loss function of the Generator
The loss function of the Discriminator
Number of
training images The adversarial
loss
The pixel-wise
MSE loss
The perceptual
loss

14. Conditional Generative Adversarial Network for
Facial Expression generation
Structure of the generator G and the discriminator D
Structure of the generator G and the discriminator D

15. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images
The generated seven facial expression images for an input
image from Ck+ dataset using conditional Generative
Adversarial Network

16. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images
Patches generation for data augmentation using
Transformation Functions (TFs).

17. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images

18. Deep Multi-Facial patches Aggregation Network for
Expression Classification
from Face Images
Confusion matrix of the
MFP-CNN of Experience
4: 80% for training and
20% for testing of the
augmented CK+ with
cGAN and the
transformation functions.

19. Clinical Depression and Affect Recognition with
EmoAudioNet
Emna Rejaibi, Daoud Kadoch, Kamil Bentounes, Romain Alfred, Mohamed Daoudi, Abdenour Hadid, Alice Othmani:
Clinical Depression and Affect Recognition with EmoAudioNet. In arXiv:1911.00310
(2019)

20. Clinical Depression and Affect Recognition with
EmoAudioNet
The Objectives:
▪ Automatic analysis of emotions and affects from speech

21. Depression is a Mental Disorder affecting
Behavior
Treatment: If diagnosed on time, the patient might regain
mental stability
Statistics: 300M People
Affected by Depression in
2017
Symptoms: Permanent lack
of interest, sadness, fatigue,
and isolation leading to
suicide

22. Clinical Depression and Affect Recognition with
EmoAudioNet
⅓ of depressed people correctly identified
Patient Health Questionnaire PHQ-8
Beck Depression Inventory BDI-II
Self-Assessed Tests
Depression Diagnosis Patient Health
Questionnaire PHQ
▪ Multiple-choice self report
questionnaire
▪ Score (from 0 to 23) is assigned
to describe Major Depressive
Disorder (MDD) severity level

23. Clinical Depression and Affect Recognition with
EmoAudioNet
Good Parametric representation
Short-time spectral
analysis
visual representation of the
spectrum of frequencies
Spectrogram
Mel frequency cepstral
coefficients (MFCCs)

24. Clinical Depression and Affect Recognition with
EmoAudioNet

25. Clinical Depression Dataset
30%
[0..23]
PHQ-8 Scores
0: Non-
Depression
1: Depression
PHQ-8 Binary
DAIC-WOZ Corpus
46% 54%
[0..23]
PHQ-8 Scores
[0..23]
PHQ-8 Scores
PHQ

26. Affect Recognition Dataset
RECOLA Corpus
▪ Multimodal Corpus of affective interactions in French
▪ Six annotators measured emotion continuously on two
dimensions :
▪ Valence: the level of pleasantness (a continuum
from negative to positive)
▪ Arousal: the intensity and the level of automatic
activation created by an event (varies from low
(calm) to high (excited).

27. Results of EmoAudioNet on Affect recognition
Experience Development Test
Accuracy CC RMSE Accuracy CC RMSE
MFCC-based
CNN
81.93% 0.8130 0.1501 70.23% 0.6981 0.2065
Spectogram-
based CNN
80.20% 0.8157 0.1314 75.65% 0.7673 0.2099
EmoAudioNet 94.49% 0.9521 0.0082 89.30% 0.9069 0.1229
EmoAudioNet
(*)
95.16% 0.9555 0.07895 90.37% 0.9156 0.1180
RECOLA Dataset results for prediction of arousal. The results
obtained for the development and the test sets in term of three metrics:
the Pearson’s Coefficient Correlation (CC) and the Root Mean Square
Error (RMSE).
(*) EmoAudioNet is pre-trained on LibriSpeech dataset and fine-tuned on
RECOLA dataset.

28. Results of EmoAudioNet on Affect recognition
Experience Development Test
Accuracy CC RMSE Accuracy CC RMSE
MFCC-based
CNN
83.37% 0.8289 0.1405 71.12% 0.6965 0.2082
Spectogram-
based CNN
78.32% 0.7984 0.1446 73.81% 0.7598 0.2132
EmoAudioNet 95.42% 0.9568 0.0625 91.44% 0.9221 0.1118
EmoAudioNet
(*)
87.55% 0.9028 0.1222 83.07% 0.8624 0.1508
RECOLA Dataset results for prediction of valence. The results
obtained for the development and the test sets in term of three metrics:
the Pearson’s Coefficient Correlation (CC) and the Root Mean Square
Error (RMSE).
(*) EmoAudioNet is pre-trained on LibriSpeech dataset and fine-tuned on
RECOLA dataset.

29. Results of EmoAudioNet on clinical depression recognition
Experience Development Test
Accuracy CC RMSE Accuracy CC RMSE
EmoAudioNet 81.96% 0.601 0.394 73.25% 0.482 0.467
EmoAudioNet
(*)
86.21% 0.642 0.60 74.13% 0.507 0.47
EmoAudioNet Performances on the development and test sets in the
Depression Assessment Task.
(*) EmoAudioNet is pre-trained on LibriSpeech dataset and fine-tuned on
RECOLA dataset.

30. Conclusion and Future works
▪ Automatic analysis of emotions and affects from
face and speech
▪ Challenging problem because of in-the-wild
environmental conditions
▪ The small size of the existing datasets and and
the high number of parameters to learn
▪ Lack of generalizability (different databases
acquired in different conditions and in realworld
scenarios)

31. E m a i l : a l i c e . o t h m a n i @ u - p e c . f r