ACM IUI 2019

1. PREDICTION OF MUSIC
PAIRWISE PREFERENCES FROM
FACIAL EXPRESSIONS
MARKOTKALČIČ1, NIMA MALEKI2, MATEVŽ PESEK3, MEHDI
ELAHI1, FRANCESCO RICCI1, MATIJA MAROLT3
1 FREE UNIVERSITY OF BOZEN-BOLZANO, ITALY
2VODAFONE ITALY
3 UNIVERSITY OF LJUBLJANA, SLOVENIA

2. BACKGROUND
1. RecSys
Preference
Elicitation
2. Pairwise
Preferences
3. Emotional
Response
THIS WORK

3. BACKGROUND -1: RECSYS
PREFERENCE ELICITATION
• Explicit
• Intrusive
• Accurate
• Ratings,Thumb Up/Down
• Implicit
• Unobtrusive
• Inaccurate
• Listening time, Play-count

4. BACKGROUND-1: RECSYS
PREFERENCE ELICITATION
• Explicit
• Intrusive
• Accurate
• Ratings,Thumb Up/Down
• Implicit
• Unobtrusive
• Inaccurate -> Less inaccurate
• Listening time, Play-count

5. BACKGROUND-2: PAIRWISE
PREFERENCES
• Single judgment
• Benchmark?
• Context-dependent
• Pairwise judgment
• Comparable items

6. BACKGROUND-2: PAIRWISE
PREFERENCES
• Single Judgment
• Benchmark?
• Context-dependent
• Pairwise Judgment
• Comparable Items

7. BACKGROUND-3: EMOTIONAL
RESPONSE
• Emotions are triggered (cf Mood)
• Music induces emotions
• Various models
• Dimensional (Valence,Arousal, Dominance)
• Basic Emotions (Fear, Joy,Anger, Disgust, Surprise, Sadness)
• Emotions can be measured from facial expressions

8. RESEARCH QUESTION
Can we infer implicitly pairwise music preferences of a user
from the facial expressions during listening to songs?
SONG 1 SONG 2
PREDICTION
MODEL

9. PROPOSED SOLUTION
• IUI 2019
• Data Acquisition (User Study)
• Prediction of Pairwise Preferences (ML)
• Future work
• Evaluation in Pairwise RecSys

10. DATA ACQUISITION
• Scenario:
• Music
• Fixed context: music for work (cognitively demanding task)
• Items:
• 200 songs
• 15s snippets
• Unknown (familiarity debiased)
• Subjects
• 75
• 29.8 yr (SD=9.5)
• 26F, 49M

11. USER STUDY

12. USER STUDY

13. USER STUDY
Affectiva SDK

14. USER STUDY
Affectiva SDK

15. ACQUIRED DATA
UserID Song 1 Song2 Score ... delta
contempt
delta
valence
...
... ... ... ... ... ... ... ...
14 34 123 -2 ... 0.34 -0.11 ...
... ... ... ... ... ... ... ...
• UserID, Song1, Song2, Score
• Raw facial expressions
• Facial expression features per song pair
• polynomial fitting 2nd order

16. PREDICTION PROBLEM
UserID Song 1 Song2 Score ... delta
contempt
delta
valence
...
... ... ... ... ... ... ... ...
14 34 123 -2 ... 0.34 -0.11 ...
... ... ... ... ... ... ... ...
• Regression problem: Score in [-2,+2]
• Classification problem: Score element of {-1,0,1}

17. PREDICTION PROBLEM
• Regression problem: Score in [-2,+2]
• Classification problem: Score element of {-1,0,1}
• Baseline: listening time
• ML: Random Forest, Gradient Boosting
• 5x random sampling 60/40

18. RESULTS
• Regression problem:
Score in [-2,+2]
• Classification problem:
Score element of {-1,0,1}

19. RESULTS
• Regression problem:
Score in [-2,+2]
• Classification problem:
Score element of {-1,0,1}

20. RESULTS
• Differences between user clusters
• RMSE

21. CONCLUSION
• New approach for implicit acquisition of preferences
• Facial expressions explain the variance in preferences
• Important features
• contempt
• valence
• joy
• sadness
• Privacy issues
• Future work: evaluation in RecSys