Affective User Modeling
Upcoming SlideShare
Loading in...5
×
 

Affective User Modeling

on

  • 281 views

Slides from my presentation at the Mei:CogSci event in Ljubljana

Slides from my presentation at the Mei:CogSci event in Ljubljana

Statistics

Views

Total Views
281
Views on SlideShare
275
Embed Views
6

Actions

Likes
0
Downloads
2
Comments
0

3 Embeds 6

http://www.linkedin.com 4
http://www.slashdocs.com 1
https://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Affective User Modeling Affective User Modeling Presentation Transcript

  • Affective User Modeling @MEi:CogSciUniverza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Marko Tkalčič marko.tkalcic@fe.uni-lj.si http://ldos.fe.uni-lj.si/markot
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. User modeling Prediction of users behavior Why? – Product recommendation
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Amazon
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Netflix
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Recommender systems
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Recommender systems DB Recommender System
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Recommender systems Feedback Knowledge DB Recommender System
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example XXX [genre = A] YYY [genre = B] ZZZ [genre = C] XYY [genre = B] XXY [genre = C] User profile: A: 0 B: 0 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example XXX [genre = A] YYY [genre = B] ZZZ [genre = C] YYY XYY [genre = B] XXY [genre = C] User profile: A: 0 B: 0 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example XXX [genre = A] YYY [genre = B] ZZZ [genre = C] YYY R=5 XYY [genre = B] XXY [genre = C] User profile: A: 0 B: 5 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example YYY [genre = B] XYY [genre = B] ZZZ [genre = C] XXX [genre = A] XXY [genre = C] User profile: A: 0 B: 5 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example YYY [genre = B] XYY [genre = B] ZZZ [genre = C] XYY XXX [genre = A] XXY [genre = C] User profile: A: 0 B: 5 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example YYY [genre = B] XYY [genre = B] ZZZ [genre = C] XYY R=3 XXX [genre = A] XXY [genre = C] User profile: A: 0 B: 4 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example YYY [genre = B] XYY [genre = B] ZZZ [genre = C] XXX [genre = A] XXY [genre = C] User profile: A: 0 B: 4 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example YYY [genre = B] XYY [genre = B] ZZZ [genre = C] ZZZ XXX [genre = A] XXY [genre = C] User profile: A: 0 B: 4 C: 0
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Traditional user modeling In movie recommender systems – Netflix example YYY [genre = B] XYY [genre = B] ZZZ [genre = C] ZZZ R=5 XXX [genre = A] XXY [genre = C] User profile: A: 0 B: 4 C: 5
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Context-aware user modeling Users have different preferences in different contexts ????? User profile: User profile: User profile: Context = alone Context = friends Context = children A: 0 A: 5 A: 1 B: 4 B: 2 B: 5 C: 5 C: 3 C: 1
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Context-aware user modeling Users have different preferences in different contexts ZZZ [genre = C] XXY [genre = C] YYY [genre = B] XYY [genre = B] XXX [genre = A] Context = alone User profile: User profile: User profile: Context = alone Context = friends Context = children A: 0 A: 5 A: 1 B: 4 B: 2 B: 5 C: 5 C: 3 C: 1
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. General user modeling framework  Data-centric = uses data that – Is available (genres, actors, directors ...) – Easy to acquire (rating, „liking“ ...)  But NOT necessarily data that carry information Controlled variables USER MODEL Selected MM itemsHuge MM DB  Prediction accuracy Uncontrolled variables
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. General user modeling framework Data-centric = uses data that – Is available (genres, actors, directors ...) – Easy to acquire (rating, „liking“ ...) But NOT necessarily data that carry information Controlled variables USER MODEL Prediction accuracy ? Uncontrolled variables
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. It is not so simple! Bounded rationality theory [Daniel Kahnemann (nobel prize for economics 2002)] Decision making = rational + emotional
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Need for affective user modeling! (Tkalčič et al., 2010)  Affective + generic variables >  Generic) variables Controlled variables = generic + affective variables USER MODEL Uncontrolled variables
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Overview of emotions Emotions are complex human experiences Evolutionary based Several definitions We take with simple models, easy to incorporate in computers: – Basic emotions – Dimensional model – Circumplex model
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Basic emotions Discrete classes model Different sets Darwin: Expression of emotions in man and animal Ekman definition (6 + neutral): – Happiness – Anger – Fear – Sadness – Disgust – Surprise
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Dimensional model Three dimensions – Valence – Arousal – Dominance Each emotive state is a point in the VAD space
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Circumplex model Maps basic emotions dimensional model Arousal high joy anger surprise disgust fear Valence neutral negative positive sadness low
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. How to detect emotions? Explicit vs. Implicit Explicit – Questionnaires (SAM) Implicit: – Work done in the affective computing community – Different modalities (sources): • Facial actions (video) • Physiological signals ( GSR, EEG) • Voice • Posture • ... – ML techniques • Classification (basic emotions) • Regression (dimensional model)
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Emotion detection from videos of facial expressions Problem statement: – Explicit affective labeling has drawbacks: • Annoying • Time consuming • Potentially inaccurate in real applications Proposed solution: – Implicit affective labeling through emotion detection from facial video – Aggregation of emotions detected from several users
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Experiment 2 datasets: – Posed (Kanade Cohn) – Spontaneous (LDOS-PerAff-1) Input: Video streams of facial expressions as responses to visual stimuli Output: emotive states as distinct classes Gabor features kNN Emotive state
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Results and conclusions Posed dataset: accuracy = 92 % Spontaneous dataset: accuracy = 62% Reasons for bad results: – Weak learning supervision – Non optimal video acquisition (face rotation, occlusions, changing lightning ...) – Non extreme facial expressions
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. The Affective User Modeling framework Problem statement: – Research is done in a scattered fashion – Researchers do not benefit from each other‘s work Goal: – Researchers to identify their position – To benefit from each other‘s work – To establish affective user modeling as a (sub)field?
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..The proposed framework - 1 time choice Give Give recommendations content Content applicationEntry stage Consumption stage Exit stage
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. The proposed framework - 2 time Entry mood Exit mood choice Detect Give Give entry recommendations content mood• Context Content application• Decision making• Influence• Diversification• Decision making profile Entry stage Consumption stage Exit stage
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. The proposed framework - 3 timeEntry mood Content-induced affective state choice Detect Give Give entry Observe user recommendations content mood Content application • Affective tagging • Affective user profiles Entry stage Consumption stage Exit stage
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. The proposed framework - 4 timeEntry mood Content-induced affective state Exit mood choice Detect Detect Give Give entry Observe user exit recommendations content mood mood Content application • Implicit feedback • Evaluation metrics (user satisfaction) Entry stage Consumption stage Exit stage
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. The proposed framework - 5 time Entry mood Content-induced affective state Exit mood choice Detect Detect Give Give entry Observe user exit recommendations content mood mood• Context Content application• Decision making• Influence • Affective tagging • Implicit feedback• Diversification • Affective user profiles • Evaluation metrics• Decision making profile (user satisfaction) Entry stage Consumption stage Exit stage
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Profiling in CBR systems Item Profile (md) Id 1 Title Girl Genre Erotic User Profile (up) Item Profile (md) Id 1 Id 2 Action 80 Title Basketball Erotic 60 Genre Sport Sport 95 Still life 35 … … Item Profile (md) Id 3 Title Kitchen Genre Still life
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Proposed solution We propose tu use AFFECTIVE METADATA Multimedia content ELICITS (induces) emotions Underlying assumption: users differ in their preferences for emotions
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Affective modeling  Emotion description models – Basic emotions (Ekman: anger, fear, joy, disgust, surprise,sadness) – Dimensional model (VAD - valence-arousal-dominance)  We aggregated the emotive responses of many users to a single image: – First two statistical moments of V, A and D – Item profile  The user profile is the result of the training an ML classifier Arousal high Valence mean <=4.23 >4.23 joy anger surprise Class = 0 Valence mean disgust <=6.71 >6.71 fear Dominance Valence Class = 1 neutral meannegative positive <=5.92 >5.92 sadness Valence mean Class = 0 <=5.21 <=5.21 Class = 1 Class = 0 low
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Experiment IAPS Image Stimuli generic metadata affective metadata Consumed Metadata EMOTION Item (Item Profile) INDUCTION Explicit Machine User Profile Rating Learning Ground Predicted Truth Ratings Ratings Confusion Matrix
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Results Pearson chi-square statistical significance test to compare the confusion matrices Scalar measures P, R, F Generic+affective metadata > generic metadata Avg(v) best feature (71% of users) SVM best classifier
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Latent factorsUsers with personality properties: Users latent factors space- Extraversion U21- Agreeableness- Conscientousness- Neuroticism- openness U12 U11 U22 Users – items Matrix Latent factors rating matrix factorization Items latent factors space I21Images with affective properties:- Valence I12 I11- Arousal- Dominance I22
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Latent factors - results
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. A personality-based user similarity measure Collaborative filtering recommender (CFR) systems: – Similar users have similar preferences – Rating-based similarity measures Which content should I watch tonight?
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Problem statement Problem statement: – New user problem: hard to assess user similarities without overlapping ratings  bad recommendations Proposed solution (hypothesis) – A personality based user similarity measure under cold start conditions
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. personality Personality: accounts for individual differences ( = explains the variance) – Old greeks: choleric, melancholic, phlegmatic, sanguine – The five factor model (FFM) – Big5: • Extraversion • Agreeableness • Conscientousness • Neuroticism • Openness Underlying assumption: – Users with similar personalities have similar preferences Measuring personality: – the IPIP questionnaire – For each user u a five tuple b =(b1, b2, b3, b4, b5)
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Experiment Proposed USM: Baseline USM:Simulate cold-start stage Get recommended Find similar users F measure items personality-based Rating-based USM USM
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Results F measures of all users: – At each cold start stage s we compared both USM with the t-test
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. OverviewTraditional user modelingin recommender systems Need for affective user modeling! HOW? Emotions & detection The proposed AUM framework Example 1 Example 2Dataset
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. The LDOS-PerAff-1 dataset Properties of the dataset – Content items – End users – Generic and affective metadata (for content items) – Personality metadata (for users) – Video recordings of users during consumption – Explicit ratings
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Data acquisition setup•Explanations to the user•Personality assessment with the IPIP questionnaire•Computer interaction: •Emotion induction approach •Images from the IAPS dataset •Content •Stimuli •Explicit Likert ratings •Matlab GUI •Webcam recording
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Dataset basic statistics 52 users (avg(age)=18.3 yrs, 37 females) IPIP 50 items questionnaire 70 colour images from the IAPS dataset 3640 videoclips (320x240 @ 15 fps)
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..Excerpt
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:.. [LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Future work Looking for a robust, all-encompassing user model Experimental work to prove parts of the model Validation in real-world scenarios
  • Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:.. Thank you. Questions?