SSHOC Kick-Off Meeting. Utrecht, The Netherlands, 11-12 March 2019 Where Social Sciences and Humanities Meet: Investigating emotional expressions of people with dementia Khiet Truong, Dirk Heylen, Deniece Nazareth, Gerben Westerhof - University of Twente

1. Investigating emotional expressions of people with
dementia
Khiet Truong, Dirk Heylen
Deniece Nazareth, Gerben Westerhof
Human Media Interaction
Psychology, Health & Technology

2. Multidisciplinary
2
Narrative
psychology
Dementia
Emotion
Computer
Science
Data
Science
Affective
Computing

3. Team
UT: Dept. of Psychology, Health &
Technology
Prof.dr. Gerben Westerhof
Deniece Nazareth, MSc
UT: Human Media Interaction
Prof.dr. Dirk Heylen
Dr. Khiet Truong
Deniece Nazareth, MSc
UMCG
Dr. Miriam Kunz
eScience Center Amsterdam
Dr. Ben de Vries
Dr. Vincent van Hees 3

4. Emotional Expressions of People
with Dementia
• How does dementia influence
emotional expression?
• Can we automatically recognize
these emotional expressions
through verbal (words) and
nonverbal (voice, face)
information?

5. Beyond state-of-the-art
• Some is known about how elderly recognize emotion (how
does dementia affect how people perceive emotion)
➤ Learn more about spontaneous emotion production in
healthy elderly and elderly with dementia
• There is a lot of research about automatic emotion recognition
in healthy adults
➤ Investigate and develop emotion recognition models and
tools for healthy elderly and elderly with dementia
➤ Develop a publicly available multimodal corpus with
spontaneous emotional expressions of elderly
5

6. Multimodal corpus of emotional expressions
Phase 1: healthy elderly
Phase 2: elderly with dementia + control group
(matched healthy elderly)
6
✓

7. 7
Lifestory books & Reminiscence Therapy
Recalling past memories
Visit 1
Autobiographical
memory test (AMT)
Visit 2
Life story book

8. Visit 1: Autobiographical memory test
(Williams & Broadbent, 1986)
• Recalling specific autobiographical memories
- A specific event that happened once in a specific place on a
specific date that did not last longer than a day
8
Autobio-
graphical
memory test
(AMT)
Word 1 Happy
“I received my diploma in March 2014 at the university and
my grandmother was there.”

9. Visit 1: Autobiographical memory test
(Williams & Broadbent, 1986)
• Recalling specific autobiographical memories
- A specific event that happened once in a specific place on a
specific date that did not last longer than a day
9
Autobio-
graphical
memory test
(AMT)
Word 1 Happy
“I have always been a happy kid.”

10. Visit 2: Lifestory book
10
“Could you tell me more about this?”
“What were you feeling back then?”
“How did people around you react?”
“How was life for you after this happened?”

11. Visit 2: Lifestory book - SAM
How did you feel at that moment when it happened?
How did you feel when you told me about the event?
How are you feeling now?
11

12. Visit 2: IAPS pictures
12

13. Procedure
13
Informed
Consent
Personality
Test
AMT MOCA
V1
Demographic
Information
Lifestory book IAPS
V2

14. 14
“Well, after giving birth, eh, she did not cry and it scared me, but it is
normal, but I did not know that yet. So she was picked up by her legs
and she got a slap on her butt and then she started to cry of course.”

15. 15
Participants
23 participants: 65 – 85 years old
11 male, 12 female
22 born in Netherlands, 1 born in Indonesia
All are living in region of Twente
Recruited via regional newspaper
Data
V1: ~1 hour (audio only)
V2: ~1,5 hours (audiovisual + heart rate + electrodermal activity +
accelerometer)

16. Beyond state-of-the-art
➤ Learn more about spontaneous emotion production in
healthy elderly and elderly with dementia
➤ Investigate and develop emotion recognition models and
tools for healthy elderly and elderly with dementia
➤ Develop a publicly available multimodal corpus with
spontaneous emotional expressions of elderly
16

17. Processing and Analyses
1. When is the speaker emotional, and is this a
positive or negative emotion? (labelling task)
2. What verbal and nonverbal features to extract to
recognize these emotions automatically?
3. How can we make this data (and tools) available
and useful for scholars both in computer science
and social sciences / humanities as well?
17

18. 2. Can we use verbal and nonverbal features
to recognize these emotions automatically?
18
Word-level transcriptions: Dutch
ASR Kaldi
Prosody: Praat
Facial
expressions:
openFace

19. 3. How can we make this data (and tools) available and useful
for scholars both in computer science and social sciences /
humanities as well?
19

20. 1. When is the speaker emotional, and is this a positive
or negative emotion? (labelling task)
• Challenging: labelling spontaneous emotions is not
straightforward!
• Sad memory: Her divorce felt 2-sided – disappointment but it
also set her free
20

21. Synchronization
21

22. Lessons learned so far
• Elicitation via IAPS pictures gives rather artificial
results
• Blended emotions
• This natural setting of recalling memories calls for
novel coding schemes of emotion
22

23. Next steps
• Developing a coding scheme for valence (positive
vs negative emotions)
- Taking into account that these are memories,
lifestories
- Taking into account that emotions can be blended
• How to make data accessible for other
researchers?
23

24. Khiet Truong, Dirk Heylen
{k.p.truong, d.k.j.heylen}@utwente.nl
http://hmi.ewi.utwente.nl
Human Media Interaction
Psychology, Health & Technology
Thank you