Affect in recommender systems

Affect in recommender systems

Marko Tkalčič
University of Ljubljana

Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..


Presentation overview

 I: LDOS presentation & Motivation
 II: What are emotions?
 III: Emotion in recsys – related work
 IV: Role of emotions in the MM consumption chain
 V: Affect in the decision-making stage
 Conclusions

 Note: some material is not ours ... Fair use ...


Part I: LDOS group at UL FE and underlying assumption


LDOS group at UL FE
 University of Ljubljana
– Faculty of electrical engineering
• LDOS (Digital signal processing laboratory)
– Approx 15 members
 Relevant people

Head: prof. Jurij Tasič

Andrej Košir

Marko Tkalčič

Ante Odić
Matevž Kunaver Tomaž Požrl


LDOS work on recommender systems
 2002-2009: public movie datasets
– CBR Basic RecSys
– CF
 2009-2012
– Emotions Affective
– Context Computing
 2012 – Affective
– Decision making (affective + cognitive attributes) RecSys
• Ajzen model
• Kahneman/Tversky model
• ...

Decision making
Modeling in RecSys


Underlying presumption of our work
 Recommender System = predictor of users‘ decision making
 Decision making: EMOTIONS DO INFLUENCE

 (c) Dilbert.com


PART II : What is affect/emotions/mood/personality


PART II : What are affect/emotions/mood/personality
 Not well defined (wikipedia):
Psychophysiological
expressions
– Emotion = subjective, conscious experience
Biological
reactions
– Affect = experience of emotion (interchangable)
Mental
states
– Emotion vs. Mood:
• Emotion = high arousal, short term
• Mood = low arousal, long term

– Personality = accounts for the individual differences in the users’
emotional, interpersonal, experiential, attitudinal and motivational styles [John and Srivastava, 1999]

CHANGES FIXED

emotion mood personality

Time (duration)


Overview of emotions/moods
 Several definitions
 We take simple models, easy to incorporate in computers:
– Basic emotions
– Dimensional model
– Circumplex model


Basic emotions
 Discrete classes model
 Different sets
 Charles Darwin: Expression of emotions in man and animal
 Paul Ekman definition (6 + neutral):
– Happiness
– Anger
– Fear
– Sadness
– Disgust
– Surprise

 (c) Paul Ekman


Dimensional model
 Three dimensions
– Valence (positive vs. Negative)
– Arousal (high vs. Low)
– Dominance (power(less) over emotions)


Dimensional model
 Three dimensions
– Valence
– Arousal
– Dominance

– (c) Lang, P. J. (1980)

 Each emotive state is a point in the VAD space


Circumplex model
 Maps basic emotions dimensional model (Posner et al.)
Arousal
high

joy
anger
surprise

disgust

fear
Valence
neutral
negative positive
sadness

low


How to detect emotions
 Emotions are characterized:
– psychophysiological expressions,
– biological reactions
– mental states

SENSORS !!!


How to detect emotions?
 Explicit vs. Implicit
 Explicit
– Questionnaires (SAM)


How to detect emotions?
 Explicit vs. Implicit
 Explicit
– Questionnaires (SAM)
 Implicit:
– Work done in the affective computing community
– Different modalities (sources):
• Facial actions (video)
• Physiological signals ( GSR, EEG)
• Voice
• Posture
• ...
– ML techniques
• Classification (basic emotions)
• Regression (dimensional model)


Lie To Me
 Part1.avi
 (c) 20th Century Fox
 Main Character Cal Lightman = Paul Ekman

Defined the FACS
(Facial Action Coding System)


LDOS Experiment
 2 datasets:
– Posed (Cohn-Kanade dataset)
– Spontaneous (LDOS-PerAff-1 dataset)
 Input: Video streams of facial expressions as responses to visual stimuli
 Output: emotive states as distinct classes

Gabor features kNN

Emotive
state


Results and conclusions
 Posed dataset: accuracy = 92 %
 Spontaneous dataset: accuracy = 62%
 Reasons for bad results:
– Weak learning supervision
– Non optimal video acquisition (face rotation, occlusions, changing lightning ...)
– Non extreme facial expressions
 Upcoming paper: IEEE Transactions on Multimedia


Personality
 Definition:
 User personality accounts for the individual differences in the users’
emotional, interpersonal, experiential, attitudinal and motivational styles [John and
Srivastava, 1999]
 Ever-lasting
 Several models
 Five Factor Model (FFM or Big5):
 Openness (inventive/curious vs. consistent/cautious)
 Conscientiousness (efficient/organized vs. easy-going/careless)
 Extraversion (outgoing/energetic vs. solitary/reserved)
 Agreeableness (friendly/compassionate vs. cold/unkind)
 Neuroticism (sensitive/nervous vs. secure/confident)
 How to measure?
 Questionnaires:
 International Personality Item Pool ( http://ipip.ori.org/ )


LDOS PerAff-1 dataset
 Emotive responses
 Ratings
 Personality data
 Videos of facial expressions
 50 users, 70 items, sparsity=0
 http://slavnik.fe.uni-lj.si/markot/Main/LDOS-PerAff-1


PART III : Related work on emotions in recsys


PART III : Related work on emotions in recsys
 Emotions and personality
 Scattered work


Emotions in recsys

Gonzalez, 2007 ? Emotions as context in recsys?

User affective feedback from
Arapakis et al., 2009
automatic facial expression analysis


Emotions in recsys


Arapakis, 2009

Tkalčič et al., 2010 Affective user model


Emotions in recsys


Arapakis, 2009


find an appropriate musical score that
Kaminskas, Ricci
would reinforce the affective state induced
2011
by the touristic attraction.


Emotions in recsys


Arapakis, 2009


find an appropriate musical score that
Kaminskas, Ricci
would reinforce the affective state induced
2011
by the touristic attraction.

Lops et al, 2012 Ongoing work: emotion detection in the phase of
presentation of the recommendations for generating
unexpected and seredipitous recommendations


Personality in recsys

Nunes et al., 2007 Personality as ???

Personality-based user similarity measure
Tkalčič et al., 2009
For the cold start problem

Rong Hu and Personality-based user similarity measure
Pearl Pu, For the cold start problem

Dennis and Masthoff, Adapting persuasive (learning)
2012 Technologies to personality traits


PART IV: Emotions in the MM consumption chain


PART IV: Emotions in the MM consumption chain

 Scattered work on emotions in RecSys

 Unifying framework (too ambitious?)


The proposed framework - 1
time

choice

Give Give
recommendations content

Content application

Entry stage Consumption stage Exit stage


time

Entry mood Exit mood

choice

Detect
Give Give
entry
mood

Content application
• Context
• Decision making
• Influence
• Diversification


time

Entry mood Content-induced affective state

choice

Detect
Give Give
entry Observe user
mood

Content application
• Affective tagging
• Affective user profiles



time

Entry mood Content-induced affective state Exit mood

choice

Detect Detect
Give Give
entry Observe user exit
mood mood

Content application

• Implicit feedback
• Evaluation metrics



time


choice

Detect Detect
Give Give
mood mood

Content application
• Context
• Decision making • Affective tagging
• Affective user profiles • Implicit feedback
• Influence • Evaluation metrics
• Diversification


PART V: Affect in the decision making step


PART V: Affect in the decision making step
 Stage 2 and 3 are straightforward
 Stage 1 is interesting = new research avenues

time


choice

Detect Detect
Give Give
mood mood

Content application
• Context
• Decision making • Affective tagging
• Affective user profiles • Implicit feedback
• Influence • Evaluation metrics
• Diversification


From data-centric to user-centric
 The community is problem-solving oriented
– „The existing datasets are real, why building synthetic ones?“ (??, RecSys
2011)
 The data-centric approach is still rooted in the research community:
– „It‘s about music, not about recommenders“ (?? at RecSys 2011)
 Solving existing problems is only a part of research ...

 ... the other part is generating new knowledge (on how the world works) ...

 ... which in turn generates new problems ...

 ... which in turn opens new publishing/funding/citing possibilities


General user modeling framework
 Data-centric = uses data that
– Is available (genres, actors, directors ...)
– Easy to acquire (rating, „liking“ ...)
 But NOT necessarily data that carry information

Controlled variables

USER MODEL
Prediction accuracy

?
Uncontrolled variables


LET‘S MOVE FORWARD
 Try new models!
 Generate new kind of data!
 Find out how the world really works!

 Model DECISION MAKING:
– Ajzen model (Andrej‘s talk)
– Kahneman model


System 1 / System 2
 (c) Kahneman, 2003


Decision Making Modeling in RecSys

Personality Content
detection Affective stimuli metadata
detection
Emotion
detection

System 1 model System 2 model

Aggregation

Decision prediction


SoA Modeling in RecSys

Personality Content
detection Affective stimuli metadata
detection
Emotion
detection

System 1 model System 2 model

Aggregation

Decision prediction


Conclusions

 RecSys = decision making predictor

 Assumption = emotions do influence

 Scattered work  Unifying framework

 Our wish = Focus on stage 1: decision making:
– System 1 / System 2 modeling


Future work

Improve models

Generate dataset

Validate

Affect in recommender systems

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

Similar to Affect in recommender systems

Similar to Affect in recommender systems (8)

Recently uploaded

Recently uploaded (20)

Affect in recommender systems