Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Affective Personalization - from Psychology to Algorithms

1,211 views

Published on

The talk will cover the research carried out by the author in the domain of psychologically-driven personalized systems. In order to be truly personalized a system needs to understand the user. Current systems employ data-driven models, such as recommendations based on past ratings, clicks or purchases. However, psychologically-grounded models appear to have potential for better personalized systems. The author will cover models of emotions and personality, the unobtrusive acquisition thereof through social media crawling, video processing and machine learning and their use in personalization algorithms.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Affective Personalization - from Psychology to Algorithms

  1. 1. Affective Personalization - from Psychology to Algorithms Marko Tkalčič, Free University of Bozen-Bolzano http://markotkalcic.com Talk at the Alpen-Adria-Universität Klagenfurt 21. December 2017 Marko Tkalčič, AAU 1/53
  2. 2. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 2/53
  3. 3. Who am I? Marko Tkalčič • 2017 : habilitation as Associate Professor in Italy • 2016 - now : assistant professor at Free University of Bozen-Bolzano • 2013 - 2015: postdoc at Johannes Kepler University, Linz • 2011 - 2012: postdoc at University of Ljubljana • 2008 - 2010: PhD student at University of Ljubljana My research explores ways in which psychologically-motivated user characteristics, such as emotions and personality, can be used to improve recommender systems (personalized systems in general). It employs methods such as user studies and machine learning. Marko Tkalčič, AAU 3/53
  4. 4. Book, 2016 • Tkalčič, M., Carolis, B. De, Gemmis, M. de, Odić, A., & Košir, A. (Eds.). (2016). Emotions and Personality in Personalized Services. Springer International Publishing. https://doi.org/10.1007/978-3-319-31413-6 • Authors from • Stanford, Cambridge, Imperial College, UCL . . . • topics: • psychological models • acquisition of emotions/personality • personalization techniques • http://www.springer.com/gp/book/9783319314112 Marko Tkalčič, AAU 4/53
  5. 5. Personalized systems The research I do is about Personalization Marko Tkalčič, AAU 5/53
  6. 6. Personalized systems The research I do is about Personalization Most frequently Recommender Systems Marko Tkalčič, AAU 5/53
  7. 7. Recommender systems Marko Tkalčič, AAU 6/53
  8. 8. Recommender Systems are not Perfect Marko Tkalčič, AAU 7/53
  9. 9. Why do recommender systems make mistakes? What is Netflix recommending us? Marko Tkalčič, AAU 8/53
  10. 10. Why do recommender systems make mistakes? What is Netflix recommending us? Movies/films . . . really? Marko Tkalčič, AAU 8/53
  11. 11. Why do recommender systems make mistakes? What is Netflix recommending us? Movies/films . . . really? “I want to watch a funny movie tonight” Marko Tkalčič, AAU 8/53
  12. 12. Why do recommender systems make mistakes? What is Netflix recommending us? Movies/films . . . really? “I want to watch a funny movie tonight” Funny is all you want? Marko Tkalčič, AAU 8/53
  13. 13. But there’s more!! Marko Tkalčič, AAU 9/53
  14. 14. But there’s more!! Question: Can (rating/genre/year/director) summarize that rollercoaster? Thanks to Shlomo Berkovski for the inspiring example from the EMPIRE 2015 keynote. Image source: http://yhvh.name/?w=2646 Marko Tkalčič, AAU 9/53
  15. 15. Decision making and emotions - Damasio • physiological/evolutionary aspect • emotional processes guide (or bias) behavior, particularly decision-making • changes in both body and brain states in response to different stimuli • these physiological signals (or somatic markers) and their evoked emotion are consciously or unconsciously associated with their past outcomes and bias decision-making References Damasio, A. (1994) Descartes’ Error: Emotion, Reason, and the Human Brain Marko Tkalčič, AAU 10/53
  16. 16. Kahneman Tversky two systems • Decision making: • System 1: fast, intuitive, emotion-driven • System 2: slow, rational References Kahneman, D. (2003). A perspective on judgment and choice: mapping bounded rationality. The American Psychologist, 58(9), 697–720. Marko Tkalčič, AAU 11/53
  17. 17. Personality and preferences Personality traits (extraverted/introverted, open/conservative etc.) are linked to music genre preferences (Rentfrow et al, 2003) References Rentfrow, P. J., and Gosling, S. D. (2003). The do re mi’s of everyday life: The structure and personality correlates of music preferences. Journal of Personality and Social Psychology, 84(6), 1236–1256. Tkalčič, M., Ferwerda, B., Hauger, D., and Schedl, M. (2015). Personality Correlates for Digital Concert Program Notes. In UMAP 2015, Lecture Notes On Computer Science 9146 (Vol. 9146, pp. 364–369). Marko Tkalčič, AAU 12/53
  18. 18. Emotions are related to other things as well Why we choose to consume some kind of content? Marko Tkalčič, AAU 13/53
  19. 19. Emotions are related to other things as well Why we choose to consume some kind of content? One of the main reasons why people consume music (Lonsdale, 2011) and films (Oliver, 2008) is emotion regulation. References Lonsdale, A. J., and North, A. C. (2011). Why do we listen to music? A uses and gratifications analysis. British Journal of Psychology (London, England : 1953), 102(1), 108–34. https://doi.org/10.1348/000712610X506831 Oliver, M. B. (2008). Tender affective states as predictors of entertainment preference. Journal of Communication, 58(1), 40–61. https://doi.org/10.1111/j.1460-2466.2007.00373.x Marko Tkalčič, AAU 13/53
  20. 20. Bottomline Ratings alone are unlikely to capture the user preferences and decision-making. Marko Tkalčič, AAU 14/53
  21. 21. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 15/53
  22. 22. Personality, Mood and Emotions Marko Tkalčič, AAU 16/53
  23. 23. Personality, Mood and Emotions Marko Tkalčič, AAU 17/53
  24. 24. Personality, Mood and Emotions Marko Tkalčič, AAU 18/53
  25. 25. Personality, Mood and Emotions Marko Tkalčič, AAU 19/53
  26. 26. Personality, Mood and Emotions Marko Tkalčič, AAU 20/53
  27. 27. Personality, Mood and Emotions Marko Tkalčič, AAU 21/53
  28. 28. Personality • What is personality? • accounts for individual differences ( = explains the variance in users) in our enduring emotional, interpersonal, experiential, attitudinal, and motivational styles Marko Tkalčič, AAU 22/53
  29. 29. Models of Personality - Big5 • The five factor model (FFM) – Big5: • Extraversion • Agreeableness • Conscientousness • Neuroticism • Openness (to new experiences) The inverse of Neuroticism is sometimes referred to as Emotional Stability, References McCraMcCrae, R. R., and John, O. P. (1992). An Introduction to the Five-Factor Model and its Applications. Journal of Personality, 60(2), p175 – 215. Marko Tkalčič, AAU 23/53
  30. 30. Measuring the FFM • Extensive questionnaires (from 5 to several 100s questions) • BFI: 44 questions • TIPI : 10 questions • NEO-IPIP: 300 questions • For each user u a five tuple b = (b1, b2, b3, b4, b5) Marko Tkalčič, AAU 24/53
  31. 31. Emotion vs. Mood vs. Sentiment Let’s clear some terminology • Affect : umbrella term for describing the topics of emotion, feelings, and moods • Emotion: • brief in duration • consist of a coordinated set of responses (verbal, physiological, behavioral, and neural mechanisms) • triggered • Mood: • last longer • less intense than emotions • no trigger • Sentiment: • towards an object • positive/negative Marko Tkalčič, AAU 25/53
  32. 32. Models of Emotions • Emotions are complex human experiences • Evolutionary based • Several definitions, we take with simple models, easy to incorporate in computers: • Basic emotions • Dimensional model Marko Tkalčič, AAU 26/53
  33. 33. Basic Emotions • Discrete classes model • Different sets • Darwin: Expression of emotions in man and animal • Ekman definition (6 + neutral): • Happiness • Anger • Fear • Sadness • Disgust • Surprise Marko Tkalčič, AAU 27/53
  34. 34. Dimensional model of Emotions Three continuous dimensions • Valence/Pleasure (positive-negative) Each emotion is a point in the VAD space Self-Assessment Manikin (SAM) Marko Tkalčič, AAU 28/53
  35. 35. Dimensional model of Emotions Three continuous dimensions • Valence/Pleasure (positive-negative) • Arousal (high-low ) Each emotion is a point in the VAD space Self-Assessment Manikin (SAM) Marko Tkalčič, AAU 28/53
  36. 36. Dimensional model of Emotions Three continuous dimensions • Valence/Pleasure (positive-negative) • Arousal (high-low ) • Dominance (high-low ) Each emotion is a point in the VAD space Self-Assessment Manikin (SAM) Marko Tkalčič, AAU 28/53
  37. 37. Dimensional model of Emotions Three continuous dimensions • Valence/Pleasure (positive-negative) • Arousal (high-low ) • Dominance (high-low ) Each emotion is a point in the VAD space Self-Assessment Manikin (SAM) Marko Tkalčič, AAU 28/53
  38. 38. Dimensional model of Emotions Three continuous dimensions • Valence/Pleasure (positive-negative) • Arousal (high-low ) • Dominance (high-low ) Each emotion is a point in the VAD space Self-Assessment Manikin (SAM) References Bradley, M. M., and Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49–59. Marko Tkalčič, AAU 28/53
  39. 39. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 29/53
  40. 40. Personality Detection Our online behaviour is influenced by our personality. Marko Tkalčič, AAU 30/53
  41. 41. Personality Detection Our online behaviour is influenced by our personality. Hence, our traces in social media should reflect our personality. Marko Tkalčič, AAU 30/53
  42. 42. Personality Detection Our online behaviour is influenced by our personality. Hence, our traces in social media should reflect our personality. It is enough to acquire personality once. Marko Tkalčič, AAU 30/53
  43. 43. Kosinki - personality from FB • personality prediction from Facebook References Kosinski, M., Stillwell, D., and Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences of the United States of America, 110(15), 5802–5. https://doi.org/10.1073/pnas.1218772110 Marko Tkalčič, AAU 31/53
  44. 44. Kosinki - personality from FB Selected most predictive likes for openness Marko Tkalčič, AAU 32/53
  45. 45. Personality from Instagram • N=113 (AMT) • 22398 pictures • BFI • features • low-level image features (Hue-Value-Saturation) • filters • presence of people References Skowron, M., Ferwerda, B., Tkalčič, M., and Schedl, M. (2016). Fusing Social Media Cues : Personality Prediction from Twitter and Instagram. WWW’16 Companion, 2–3. https://doi.org/10.1145/2872518.2889368 Marko Tkalčič, AAU 33/53
  46. 46. Unobtrusive Personality Detection wo Social Media • When users don’t want to disclose their social media activity? • Relationship between personality traits and disclosure behavior • 126 subjects • Facebook simulation web app • BFI 44 items (1-5) aits and es of the stom set- Although providing isregards ons that e setting To every- know the omething ed as 0. the rela- behavior 1]) is re- he results xperience ” section. ity” (r=- e phone” Address” O C E A N 4 Current city -.24* -.20ˆ -.08 -.08 .01 5 Hometown -.25* -.18ˆ -.08 -.13 -.05 6 Places lived -.12 -.12 -.08 -.20* -.01 7 Mobile phone -.22* -.12 -.01 -.05 .10 8 Website -.22* .01 .16 .02 -.16 9 Email -.16 .09 -.23* .13 -.13 10 Address -.24* -.02 .14 -.04 -.15 11 Birth date -.18ˆ -.18ˆ -.22* -.12 .17ˆ 32 Restaurant .03 -.06 .22* -.06 .09 33 Games .10 .01 .18ˆ .02 -.13 34 Activities .05 .03 .21* .06 -.08 35 Interests .09 -.04 .17ˆ -.06 -.05 37 Foods .01 -.18 .24* .01 -.11 38 Clothing -.05 -.06 .19ˆ .01 -.09 40 Other -.05 -.19ˆ .08 -.09 .02 Note. ˆp<0.1, *p<0.05 Table 2: Correlation Matrix of the profile items disclosure against the personality traits: (O)penness, (C)onscientiousness, (E)xtraversion, (A)greeableness, (N)euroticism. Only items that show significant levels of p<0.1 are reported. References Ferwerda, B., Schedl, M., and Tkalčič, M. (2016). Personality Traits and the Relationship with ( Non- ) Disclosure Behavior on Facebook. WWW’16 Companion. https://doi.org/10.1145/2872518.2890085 Marko Tkalčič, AAU 34/53
  47. 47. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 35/53
  48. 48. Personality for mood regulation • high on openness, extraversion, and agreeableness more inclined to listen to happy music when they are feeling sad. • high on neuroticism listen to more sad songs when feeling disgusted (neurotic people choose to increase their level of worry) References Ferwerda, B., Schedl, M., and Tkalcic, M. (2015). Personality and Emotional States : Understanding Users ’ Music Listening Needs. In A. Cristea, J. Masthoff, A. Said, and N. Tintarev (Eds.), UMAP 2015 Extended Proceedings. Marko Tkalčič, AAU 36/53
  49. 49. Personality and music browsing styles • personality is correlated with music browsing styles References Ferwerda, B., Yang, E., Schedl, M., and Tkalčič, M. (2015). Personality Traits Predict Music Taxonomy Preferences. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’15 (pp. 2241–2246). https://doi.org/10.1145/2702613.2732754 Marko Tkalčič, AAU 37/53
  50. 50. Personality as user similarity • new user problem • N = 52 • images = 70 • neighborhood-based RS: Euclidian distance −1 References Tkalčič, M., Kunaver, M., Košir, A., and Tasič, J. (2011). Addressing the new user problem with a personality based user similarity measure. In F. Ricci, G. Semeraro, M. de Gemmis, P. Lops, J. Masthoff, F. Grasso, J. Ham (Eds.), Joint Proceedings of the Workshop on Decision Making and Recommendation Acceptance Issues in Recommender Systems (DEMRA 2011) and the 2nd Workshop on User Models for Motivational Systems: The affective and the rational routes to persuasion (UMMS 2011). Marko Tkalčič, AAU 38/53
  51. 51. Personality in Matrix Factorization • in (Elahi et al., 2013) and (Fernández-Tobías, 2016) • injection of personality factors in MF as additional (latent) features (a la SVD++) rui = qi (pu + a∈A(u) ya) • personality u = (2.3, 4.0, 3.6, 5.0, 1.2) maps to A(u) = {ope2, con4, ext4, agr5, neu1}. • (Fernández-Tobías, 2016) is a very comprehensive paper • iMF = (Hu et al., 2008) References Elahi, M., Braunhofer, M., Ricci, F., and Tkalčič, M. (2013). Personality-based active learning for collaborative filtering recommender systems. In M. Baldoni, C. Baroglio, G. Boella, and O. Micalizio (Eds.), AI*IA 2013: Advances in Artificial Intelligence (pp. 360–371). Fernández-Tobías, I., Braunhofer, M., Elahi, M., Ricci, F., and Cantador, I. (2016). Alleviating the new user problem in collaborative filtering by exploiting personality information. User Modeling and User-Adapted Interaction, 26(2), 1–35. https://doi.org/10.1007/s11257-016-9172-z Marko Tkalčič, AAU 39/53
  52. 52. UMUAI Special Issue • UMUAI, June 2016 • Special Issue on Personality in Personalized Systems • Issue Editors: • Marko Tkalcic, • Daniele Quercia, • Sabine Graf References Tkalčič, M., Quercia, D., and Graf, S. (2016). Preface to the special issue on personality in personalized systems. User Modeling and User-Adapted Interaction, 26(2–3), 103–107. https://doi.org/10.1007/s11257-016-9175-9 Marko Tkalčič, AAU 40/53
  53. 53. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 41/53
  54. 54. Multimodal Emotion Detection • Emotions consist of a coordinated set of responses (verbal, physiological, behavioral, and neural mechanisms) • These responses can be used to measure the emotions. • Affective Computing Marko Tkalčič, AAU 42/53
  55. 55. Overview • modalities • audio • language • visual - videos of faces (action units) • physiology • brain signals • target emotion • discrete (classification) • continuous (regression) References Schuller, B. W. (2016). Acquisition of Affect. In M. Tkalčič, B. De Carolis, M. de Gemmis, A. Odić, and A. Košir (Eds.), Emotions and Personality in Personalized Services: Models, Evaluation and Applications (pp. 57–80). Cham: Springer International Publishing. Marko Tkalčič, AAU 43/53
  56. 56. Unobtrusive Emotion Detection from Facial Videos • 2 datasets: • Posed (Kanade Cohn) • Spontaneous (LDOS-PerAff-1) • video streams of facial expressions as responses to visual stimuli • distinct classes • Gabor features • kNN classifier • Posed dataset: accuracy = 92 % • Spontaneous dataset: accuracy = 62% • Reasons for bad results: • Weak learning supervision • Non optimal video acquisition (face rotation, occlusions, changing lightning . . . ) • Non extreme facial expressions Marko Tkalčič, AAU 44/53
  57. 57. Sad Reality Marko Tkalčič, AAU 45/53
  58. 58. Sad Reality Marko Tkalčič, AAU 45/53
  59. 59. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 46/53
  60. 60. Model of usage of emotions in Recommender Systems References Tkalčič, M., Košir, A., Tasič, J., and Kunaver, M. (2011). Affective recommender systems: the role of emotions in recommender systems. In A. Felfernig, L. Chen, M. Mandl, M. Willemsen, D. Bollen, and M. Ekstrand (Eds.), Joint proceedings of the RecSys 2011 Workshop on Human Decision Making in Recommender Systems (Decisions@RecSys’11) and User-Centric Evaluation of Recommender Systems and Their Interfaces-2 (UCERSTI 2) affiliated with the 5th ACM Conference on Recommender (pp. 9–13). Marko Tkalčič, AAU 47/53
  61. 61. Affective User Modeling • Multimedia content ELICITS (induces) emotions • Underlying assumption: users differ in their preferences for emotions Marko Tkalčič, AAU 48/53
  62. 62. Affective User Modeling References Tkalčič, M., Burnik, U., and Košir, A. (2010). Using affective parameters in a content-based recommender system for images. User Modeling and User-Adapted Interaction, 20(4), 279–311. doi:10.1007/s11257-010-9079-z Tkalčič, M., Odić, A., Košir, A., and Tasič, J. (2013). Affective labeling in a content-based recommender system for images. IEEE Transactions on Multimedia, 15(2), 391–400. https://doi.org/10.1109/TMM.2012.2229970 Marko Tkalčič, AAU 49/53
  63. 63. Emotions as feedback • video-on-demand scenario • usage of hesitation as feedback • 4 recommendations, 1 selection • control group: recommend similar • hesitation group: recommend similar/diverse • quality of experience (QoE) is improved when hesitation is taken into account References Vodlan, T., Tkalčič, M., and Košir, A. (2015). The impact of hesitation, a social signal, on a user’s quality of experience in multimedia content retrieval. Multimedia Tools and Applications. doi:10.1007/s11042-014-1933-2 Marko Tkalčič, AAU 50/53
  64. 64. Table of Contents Introduction Models of Emotion and Personality Acquisition of Personality Usage of personality for personalization Acquisition of Emotions Usage of emotions for personalization Conclusion Marko Tkalčič, AAU 51/53
  65. 65. Conclusion • emotions and personality account for differences in user behavior • research is still scattered Marko Tkalčič, AAU 52/53
  66. 66. Open Issues • lack of awareness • this talk will hopefully help • lack of data • mostly small-scale data gathered through user studies • exceptions: • Movielens+personality • myPersonality • unobtrusive annotation of content • subtitles? • privacy Marko Tkalčič, AAU 53/53

×