Automatic Recognition of Emotions in Speech: Models and Methods. Аndreas wendemuth

130.Oct. 2014YAC - Automatic recognition of emotions in speech – Andreas Wendemuth
Automatic Recognition of Emotions in Speech:  
 
models and methods 

Prof. Dr. Andreas Wendemuth
Univ. Magdeburg, Germany
Chair of Cognitive Systems
Institute for Information Technology and Communications
YAC / Yandex, 30. October 2014, Moscow

Recorded speech starts as an acoustic signal. For decades, appropriate
methods in acoustic speech recognition and natural language processing
have been developed which aimed at the detection of the verbal content of
that signal, and its usage for dictation, command purposes, and assistive
systems. These techniques have matured to date. As it shows, they can be
utilized in a modified form to detect and analyse further affective
information which is transported by the acoustic signal: emotional content, 
intentions, and involvement in a situation. Whereas words and phonemes are
the unique symbolic classes for assigning the verbal content, finding
appropriate descriptors for affective information is much more difficult.  
We describe the corresponding technical steps for software-supported affect
annotation and for automatic emotion recognition, and we report on the
data material used for evaluation of these methods.
Further, we show possible applications in companion systems and in dialog
control.
Abstract

1. Affective Factors in Man-Machine-Interaction
2. Speech and multimodal sensor data – what they reveal  
3. Discrete or dimensional affect description
4. software-supported affect annotation 
5. Corpora
6. Automatic emotion recognition 
7. Applications in companion systems and in dialog control
Contents

Affective Factors in Man-Machine-Interaction

Affective Terms - Disambiguation
Emotion [Becker 2001]
• short-time affect
• bound to specific events
Mood [Morris 1989]
medium-term affect•
• not bound to specific events
Personality [Mehrabian 1996]
• long-term stable
• represents individual characteristics

Emotion: the PAD-space
• Dimensions:
• pleasure / valence (p),
• arousal (a) and
• dominance (d)
• values each from -1.0 bis 1.0
• “neutral” at center
• defines octands, e.g. (+p+a+d)
Siegert et al. 2012 Cognitive Behavioural Systems. COST

Correlation of emotion and mood
In order to make it measurabble, there has to be an empirical correlation of
moods to PAD space (emotion octands). [Mehrabian 1996]
Moods for octands in PAD space
PAD mood PAD mood
+++ Exuberant
++- Dependent
+-+ Relaxed
+- - Docile
- - - Bored
- -+ Disdainful
-+- Anxious
-++ Hostile

Personality and PAD-space
Unique personality model: Big Five [Allport and Odbert 1936]
5 strong independent factors
[Costa and McCrae 1985] presented the five-factor personality
inventory
deliberately applicable to non-clinical environments
•
•
•
•
Neuroticism
Extraversion
openness
agreeableness
conscientiousness
•
•
•
•
•
• measurable by questionnaires (NEO FFI test)
• Mehrabian showed a relation between the Big Five Factors (from Neo-FFI,
scaled to [0,1]) and PAD-space. E.g.:
• P := 0.21 · extraversion +0.59 · agreeableness +0.19 · neuroticism
(other formulae available for arousal and dominance)

5. Corpora
Contents

•  Speech (Semantics)
•  Non-semantic utterances („hmm“, „aehhh“)
•  Nonverbals (laughing, coughing, swallowing,…)
•  Emotions in speech 
Interaction modalities –  
what a person „tells“

Discourse Particles
Especially the intonation reveals details about the speakers attitude but
is influenced by semantic and grammatical information.

investigate discourse particles (DPs)

•  can’t be inflected but emphasized
•  occurring at crucial communicative points
•  have specific intonation curves (pitch-contours)
•  thus may indicate specific functional meanings
Siegert et al. 2013 WIRN Vietri

The Role of Discourse Particles for Human Interaction
J. E. Schmidt [2001] presented an empirical study where he could
determine seven form-function relations of the DP “hm”:
Name idealised
pitch-contour
Description
DP-A attention
DP-T thinking
DP-F finalisation signal
DP-C confirmation
DP-D decline∗
DP-P positive
assessment
DP-R request to respond

The Role of Discourse Particles for Human Interaction
•  [Kehrein and Rabanus, 2001] examined different conversational styles
and confirmed the form-function relation.
•  [Benus et al., 2007] investigated the occurrence frequency of specific
backchannel words for American English HHI.
•  [Fischer et al., 1996]: the number of partner-oriented signals is
decreasing while the number of signals indicating a task-oriented or
expressive function is increasing
•  Research Questions
•  Are DPs occurring within HCI?
•  Which meanings can be determined?
•  Which form-types are occurring?

•  Speech (Semantics)
•  Non-semantic utterances („hmm“, „aehhh“)
•  Nonverbals (laughing, coughing, swallowing,…)
•  Emotions in speech 
•  Eye contact / direction of sight
•  General Mimics
•  Face expressions (Laughing, angryness,..) 
•  Hand gesture, arm gesture
•  Head posure, body posure
•  Bio-signals (blushing, paleness, shivering, frowning…)
•  Pupil width
•  Haptics: Direct operation of devices (keyboard, mouse, touch)
•  Handwriting, drawing, sculpturing, …
Interaction modalities – what a person „tells“ with other modalities

•  Indirect expression (pauses, idleness, fatigueness)
•  Indirect content (humor, irony, sarcasm)
•  Indirect intention (hesitation, fillers, discourse particles)
What speech can (indirectly) reveal

•  Recognizing speech, mimics, gestures, poses, haptics, bio-signals: indirect information
•  Many (most) modalities need data-driven recognition engines
•  Unclear categories (across modalities?)
•  Robustness of recognition in varying / mobile environments
Technical difficulties

So, really, you have raw data.
Now you (hopefully) have recorded (multimodal)
data with (reliable) emotional content 
 
 
 
 
 
 
 
but what does it convey?
 
 
Actually, you have a (speech) signal,

5. Corpora
Contents

transcriptions (intended things which happened)
(Speech: „Nice to see you“; Mimics: „eyes open, lip corners up“; … )
Now you need:
and
annotations (unintended events, or the way how it happened).
Speech: heavy breathing, fast, happy; Mimics: smile, happiness; …
Both processes require
labelling: tagging each recording chunk with marks, which
correspond to the relevant transcription / annotation categories

•  Trained transcribers / annotators with high intra- and interpersonal reliability (kappa
measures)
•  Time aligned (synchronicity!), simultaneous presentation of all modalities to the transcriber /
annotator
•  Selection of (known) categories for the transcriber / annotator
•  Labelling
How to transcribe / annotate?

Clear (?) modal units of investigation / categories e.g.:
•  Speech: phonemes, syllables, words
•  Language: letters, syllables, words
•  Request: content! (orgin city, destination city, day, time)
•  Dialogues: turn, speaker, topic
•  Situation Involvement: object/subject of attention, diectics, active/passive participant
•  Mimics: FACS (Facial Action Coding System) -> 40 action units
•  Big 5 Personality Traits (OCEAN)
•  Sleepiness (Karolinska Scale)
•  Intoxication (Blood Alcohol Percentage)
Categories:

•  Unclear (?) modal categories e.g.:
•  Emotion: ???
•  Cf.: Disposition: Domain-Specific …. ?
•  Cf.: Level of Interest (?)
Categories:

Categorial Models of human emotion ... 

... which can be utilized for automatic emotion recognition
•  Two-Class models,
e.g. (not) cooperative
•  Base Emotions [Ekman, 1992]
(Angriness, Disgust, Fear,
Joy, Sadness, Surprise, Neutral)
•  VA(D) Models
(Valence (Pleasure) Arousal Dominance)
•  Geneva Emotion Wheel
[Scherer, 2005]
2
3

Categorial Models of human emotion (2): 
enhanced listings 

Siegert et al. 2011 ICME
2
4
• sadness,
• contempt,
• surprise,
• interest,
• hope,
• relief,
• joy,
• helplessness,
• confusion

Categorial Models of human emotion (3): 
Self-Assessment Manikins [Bradley, Lang, 1994]
 

Böck et al. 2011 ACII
2
5

5. Corpora
Contents

•  (having fixed the modalities and categories)
•  Examples; EXMARaLDA, FOLKER, ikannotate
EXMARaLDA: „Extensible Markup Language for Discourse Annotation“, www.exmaralda.org/‎, Hamburger Zentrum für Sprachkorpora (HZSK) und
SFB 538 ‘Multilingualism’, seit 2001/ 2006
FOLKER: „Forschungs- und Lehrkorpus Gesprochenes Deutsch“ - Transkriptionseditor, http://agd.ids-mannheim.de/folker.shtml, Institute for
German Language, Uni Mannheim, seit 2010  
[Schmidt, Schütte, 2010]
Transcription / annotation tools

ikannotate - A Tool for Labelling, Transcription, and Annotation of Emotionally Coloured
Speech (2011)
•  Otto von Guericke University - Chair of Cognitive Systems + Dept. of Psychosomatic Medicine
and Psychotherapy
ü Written in QT4 based on C++
ü Versions for Linux, Windows XP and higher, and Mac OS X
ü Sources and binaries are available on demand
ü Handles different output formats, especially, XML and TXT
ü Processes MP3 and WAV files
ü According to conversation analytic system of transcription
(GAT) (version 1 and 2) [Selting et.al., 2011]
http://ikannotate.cognitive-systems-magdeburg.de/
ikannotate tool

Screenshots of ikannotate (I)

Screenshots of ikannotate (II)

5. Corpora
Contents

•  Overview: http://emotion-research.net/wiki/Databases (not complete)
•  Contains information on: Identifier, URL, Modalities, Emotional content,
Emotion elicitation methods, Size, Nature of material, Language
•  Published overviews: Ververidis & Kotropoulos 2006, Schuller et al. 2010,
Appendix of [Pittermann et al.2010]*
•  Popular corpora (listed on website above):
Emo-DB: Berlin Database of Emotional Speech 2005
SAL: Sensitive Artificial Listener (Semaine 2010)
(not listed on website above):
eNTERFACE (2005)
LMC: LAST MINUTE (2012)
Table Talk (2013)
Audio-Visual Interest Corpus (AVIC) (ISCA 2009)
•  Ververidis, D. & Kotropoulos, C. (2006). “Emotional speech recognition: Resources, features, and methods”. Speech Commun 48 (9), pp.
1162–1181.
•  Schuller, B.; Vlasenko, B.; Eyben, F.; Wollmer, M.; Stuhlsatz, A.; Wendemuth, A. & Rigoll, G. (2010). “Cross-Corpus Acoustic Emotion
Recognition: Variances and Strategies” IEEE Trans. Affect. Comput. 1 (2), pp. 119–131.
•  Pittermann, J.; Pittermann, A. & Minker, W. (2010). Handling Emotions in Human-Computer Dialogues. Amsterdam, The Netherlands:
Springer.
Corpora of affective speech (+other modalities)

•  Burkhardt, et al., 2005: A Database of German Emotional Speech,
•  Proc. INTERSPEECH 2005, Lisbon, Portugal, 1517-1520.
•  7 emotions: anger, boredom, disgust, fear, joy, neutral, sadness
•  10 professional German actors, 5f, 494 phrases
•  Perception test with 20 subjects: 84.3% mean acc.
•  http://pascal.kgw.tu-berlin.de/emodb/index-1280.html
Example 1: Berlin Database of Emotional Speech (EMO-DB)

Example 2: LAST MINUTE Corpus
Setup
Non-acted, emotions evoked by story:
task solving with difficulties (barriers)
Groups
N = 130, balanced in age, gender,
education
Duration 56:02:14
Sensors 13
Max. Video Bandwidth 1388x1038 25Hz
Biopsychological data heart beat, respiration, skin reductance
Questionnaires sociodemographic, psychometric
Interviews yes (73 subjects)
Language German
Available upon request at roesner@ovgu.de and joerg.frommer@med.ovgu.de
Frommer et al. 2012 LREC

5. Corpora
Contents

•  Remember, now you have transcribed/annotated data with fixed
categories (across modalities?) and modalities.
•  You want to use that data to construct unimodal or multimodal
data-driven recognition engines
•  Once you have these engines, you can automatically determine the
categories in yet unkown data.
Data-driven recognition engines

•  It’s Pattern Recognition
• 
•  Knowledge
Sources
• 
A Uniﬁed View on data driven recognition
Schuller 2012 Cognitive Behavioural Systems
COST
( ){ }Llyx ll ,...,1, ==Λ
Capture
Pre-
processing
Feature
extraction
Feature
reduction
Classiﬁcation
Regression
Decoding
U f(x') xx' y=κrf(x)
Feature
generation /
selection
multi-
layered multi-
layered
once
Dictionary
Interaction
Grammar
Production
Model
( ) κΩ→→ xxf
Encoding
Learner
Optimisation

Audio Features
Böck et al. 2013 HCII
Facial Action Units
•  MFCCs with Delta and Acceleration
•  Prosodic features
•  Formants and corresponding
bandwidths
•  Intensity
•  Pitch
•  Jitter
• For acoustic feature extraction: Hidden Markov Toolkit
(HTK) and phonetic analysis software PRAAT (
http://www.praat.org)

What is the current state of affect recognition?
Table : Overview of reported results, #C: Number of Classes, eNT: eNTERFACE,
VAM: Vera am Mittag, SAL: Sensitive Artificial Listener, LMC: LAST MINUTE.
Comparing the results on acted emotional data and naturalistic
interactions:
•  recognition performance decreases
•  too much variability within the data
Database Result #C Comment Reference
emoDB
(acted)
91.5% 2 6552 acoustic features and GMMs Schuller et al., 2009
eNT
(primed)
74.9% 2 6552 acoustic features, GMMs Schuller et al., 2009
VAM
(natural)
76.5% 2 6552 acoustic features with GMMs Schuller et al., 2009
SAL
(natural)
61.2% 2 6552 acoustic features with GMMs Schuller et al., 2009
LMC
(natural)
80% 2 pre-classification of visual, acoustic
and gestural features, MFN
Krell et al.,2013
Siegert et al. 2013 ERM4HCI

User-group / temporal specific affect recognition
SuccessRates [stress/nostress](testedon LASTMINUTEcorpus):
•  72% utilizing (few) group-specific (young/old+male/female)
audio features [Siegert et al., 2013]
•  71% utilizing audio-visual features and a linear filter as decision level
fusion [Panning et al., 2012]
•  80% using facial expressions, gestural analysis and acoustic features
with Markov Fusion Networks [Krell et al., 2013]
Approaches2&3integrate their classifiers of longer temporal sequences.
Siegert et al. 2013 ERM4HCI, workshop ICMI 2013

Classification Engines – Cross-Modalities
• Classification based
on audio feature
• Preselection of
relevant video
sequences
• Manual annotation of
Action Units and
classification of facial
expressions

Further:
• preclassification of
the sequences
• Dialog act
representation models
Böck et al. 2013 HCII, Friesen et al. 2014 LREC

5. Corpora
Contents

•  Remember, now you have transcribed/annotated data with fixed
categories (across modalities?) and modalities (maybe a corpus).
•  You also have a categories classifier trained on these data, i.e.  
domain specific / person specific.
Now we use categorized information in applications:
Usage of multimodal information

•  Disambiguation (saying and pointing)
•  Person‘s choice (talking is easier than typing)
•  „Real“ information (jokes from a blushing person?)
•  Robustness (talking obscured by noise, but lipreading works)
•  Higher information content (multiple congruent modalities)m
•  Uniqueness (reliable emotion recognition only from multi-
modalities)
Why more modalities help understanding what a
person wants to „tell“

Companion Technology
Applica'on

/

Dialog-‐
Management

Input signalSpeech

Gesture

Touch

Physiolog.

Sensor

Devices
Mul'modal

Components

Output signal
Multimodal
Adaptive
Individualised
Interaction
Management
User
Weber et al. 2012 SFB TRR
62

Recognition of critical dialogue courses
•  On basis of linguistic content
•  in combination with multi-modal emotion recognition
Development of empathy-promoting dialogue strategies
•  Motivation of the user
•  Prevent abandonment of the dialogue in problem-prone situations
Emotional and dialogic conditions in user behavior

Take home messages / outlook
Emotion / Affect recognition:
• Data driven, automatic pattern recognition
• Categorisation, Annotation tools
• Temporal emotion train dependent on mood and
personality
• Outlook:
Emo'on-‐categorial
Appraisal-‐Model
Use in Man-Machine-Interaction:
• Early detection / counteraction of adverse dialogs
• Outlook:
use
in
call
centers
and
companion
technology

… thank you! 
 
 
 
 
 
 
 
 
www.cogsy.de

Automatic Recognition of Emotions in Speech: Models and Methods. Аndreas wendemuth

Recommended

Recommended

More Related Content

Similar to Automatic Recognition of Emotions in Speech: Models and Methods. Аndreas wendemuth

Similar to Automatic Recognition of Emotions in Speech: Models and Methods. Аndreas wendemuth (20)

More from Yandex

More from Yandex (20)

Recently uploaded

Recently uploaded (20)

Automatic Recognition of Emotions in Speech: Models and Methods. Аndreas wendemuth