https://sites.google.com/view/designinghuman-centricmir/home?authuser=0

1. Supporting user
control for music
recommendations
Dr. Nava Tintarev
Assistant Professor
TU Delft/Delft Data Science/Delft Design for Values

2. Research area & Domains
Program chair: ACM IUI (Cagliari, 2020)
Senior reviewer: ACM IUI, Recsys, UMAP
Reviewer, a.o., IJCAI-ECAI, CHI, ACM TiiS
Music News
Travel Education

3. Recommender systems
3

4. What is a recommender system?
Systems that make personalized recommendations of goods, services, and people (Kautz)
• User identifies one or more objects as being of interest
• The recommender system suggests other objects that are similar (infers liking)
• Ranking and filtering algorithm
• Ranks the options, filters out lower ranking options
1
2
3
4

6. Transparency and Control

7. Explanations
Read this you must,
because…
Tintarev & Masthoff, 2012

8. Purpose Description
Transparency Explain how the system works
Effectiveness Help users make good decisions
Trust Increase users' confidence in the system
Persuasiveness Convince users to try or buy
Satisfaction Increase the ease of use or enjoyment
Scrutability Allow users to tell the system it is wrong
Efficiency Help users make decisions faster
Tintarev and
Masthoff. 2007

9. Timeline
2006| 2008 | 2010 | 2012 | 2014 | 2016 | 2018
Adaptive text explanations
Better explanations?
• Vizualizations
• Interactive interfaces
Factors that influence expl. effectiveness?
• Situational Characteristics
• Personal Characteristics

10. EXPLANATION
EFFECTIVENESS
Factors influencing effectiveness
Knijnenburg et al. 2012

11. Personal Characteristics?
• Working memory (Conati2014,
Tintarev2016, Tocker2012,
Velex2005)
• Expertise (Kobsa2001, Shah2011,
Tocker2012)
Situational Characteristics?
• Perceptions of diversity
(Ferwerda2017, Kang2016,
Willemsen2011)
• Group recommendations
(Najafian2018, Kapcak2018)

12. Personal characterstics

13. Research questions
• How do personal characteristics influence
the impact of…
• RQ1: user controls in terms of diversity and
acceptance?
• RQ2: visualizations in terms of diversity and
acceptance?
• RQ3: visualizations + user controls in terms of
diversity and acceptance?
13
Katrien Verbert
Associate Professor
Yucheng Jin
PhD researcher
Jin et al 2018: UMAP and Recsys,
UMUAI 2019

14. Study procedure
Tutorial of study
Pre-study
questionnaire:
• Demographics
• Experience
• Visual Memory
(VM) Capacity
• Musical
Sophistication
(MS)
Task
• Pick scenario
• Create a play-
list
• Explore and rate
all songs in the
final playlist
Post-study
questionnaire:
• perceived
quality
• perceived
accuracy
• perceived
diversity
• Satisfaction
• Effectiveness
• choice difficulty

15. Procedure
Platform
• Spotify API generate recommendations
• Based on up to five favourite artists.
• 14 musical attributes in order to describe musical preference
Dependent variables
• Perceived diversity: self-reported measure based on questionnaire
• Recommendation acceptance: measured by percentage of liked
songs in the play-list
15

16. Personal characteristics (Co-variates)
Musical sophistication (MS)
Measurement of the ability to engage with music
in a flexible, effective and nuanced way
(Müllensiefen et al., 2014)
Measured using the Goldsmiths Musical
Sophistication Index (Gold-MSI)
Visual memory (VM)
the ability to recall visual patterns (Tintarev and
Mastoff, 2016)
Measured by Corsi block-tapping test

17. User-centered factors
17
Exp.1 and Exp.3 used framework of
Knijnenburg et al. 2012.
Perceived accuracy: participants' perceived accuracy of the
recommended songs according to their preference.
Perceived diversity: the similarity among the recommended
songs.
•Several songs in the list of recommended songs were very
different from each other.
•The list of recommended songs covered many genres
•Most songs were from the same type
Exp. 2 used the ResQue framework
(Pu2011)
The songs recommended to me are of various kinds.
The songs recommended to me are similar to each other.

18. Hypotheses
• H1: The UI setting (user control, visualization, or both) has a
significant effect on recommendation acceptance.
• H2: The UI setting (user control, visualization, or both) has a
significant effect on perceived diversity.
• H3: Visual memory (VM) has a significant effect on
recommendation acceptance.
• H4: Visual memory (VM) has a significant effect on perceived
diversity.
• H5: Musical sophistication (MS) has a significant effect on
recommendation acceptance.
• H6: Musical sophistication (MS) has a significant effect on
perceived diversity.
18

19. Participants
19

20. User profile (PRO) Algorithm parameters (PAR) Recommendations (REC)
Experiment1:Controls

21. CFA - validity
questions
Quality
Accuracy
Effectiveness
Choice difficulty
Diversity (low AVE
value)
Trust (low AVE value)
Satisfaction
(modification
index)
The fit of our SEM
model
X(2, 98) = 257.410
p <.001
RMSEA= 0.083
CFI = 0.980
TLI = 0.968

22. “Sweetspot” of control
• Two-way interaction: does not necessarily corr. w
higher cognitive load. Adding an additional control
component to PAR increases corr w. acceptance.
PRO*PAR has lower cognitive load than PRO and PAR
• Three-way interaction: correlates w. acceptance, and
does not correlate with sig. higher cognitive load.
Increase interaction times and accuracy
• High MS correlated w higher quality, which corr. with
higher rec. acceptance
2
2

23. UI Design
23
Experiment 2: Visualizations

24. Exp. 3:
Control + Vis.
24

25. Summary
• Exp1: Acceptance (MS)
• Exp2: Diversity (MS, VS)
• Exp3:
• Acceptance (MS)
• Diversity (MS, VS)

26. Summary
• User control: Musical sophistication
interacted w the acceptance of
recommendations for user controls.
• Visualization:
• Musical sophistication also interacted w.
perceived diversity for visualizations, but
only for one of the studied
visualizations.
• Visual memory interacted with
visualizations for perceived diversity, but
only for one of the studied
visualizations.

27. Situational Characteristics

28. ContextPlay (Jin, 2019)
• RQ1: How does adding control of
contextual characteristics influence user
perceptions of the system?
• RQ2: How do different contextual
characteristics (mood, weather, time of
day, and social) influence user perceptions
Location
Activity

30. ContextPlay
(results)
• H1: The control of context
positively influence user
perception of system. (Accept)
- RQ1
• H2: The contextual
characteristics will influence
user perception of system.
(Accept) - RQ2
• E.g. Mood positively
influences perceived
quality (p<.05) and
diversity (p<.05), which
further positively
influences the perceived
effectiveness.
• Both Weather (positively)
and location (outdoor,
negatively) influence
effectiveness (p<.05).

31. The percentage of users who controlled context
In general, mood is th
in all scenarios, and w
as much as mood in th
scenario.
The most manipulated components are
the liked artists and tracks in four
scenarios.
• Mood is the most modified – all scenarios
• Weather – as much for outdoor + relaxing scenario
Controlling context

32. X
The percentage of users who controlled each user profile
age of users who controlled context
anipulated components are
ists and tracks in four
g scenarios seem to engage
to modify their profile
rios will influences user
n control. (Accept) - RQ3
• Most modified artists and tracks (all scenarios)
• Relaxing scenarios (most modifications)
• H3: Scenarios influence user requirements on control
Controlling user profile

33. Summary

34. What’s next?
• ENSURE - ExplaiNing SeqUences in Recommendations for groups with
divergent preferences in music and tourism. (2018-2022)
• XAINL - European Training Network on Explainable AI using Natural
Language
• Interactions to mitigate human biases
• FAIRView (Google Digital News Initiative) produce explainable video
summaries (2019-2020):
• to assess the quality of the video summaries in terms of how
well they represent different aspects of the original video
• to assess the potential for misinformation or bias
• Representing diverse views for polarized topics online (2019-2023, co-
funded IBM)
• Devising Metrics for Assessing Echo Chambers, Incivility, and
Intolerance (2019-2021, funded by Twitter)

35. EPSILON LAB – EXPLANATIONS
Post-docs
Oana Inel
Emily Sullivan
Mesut Kaya
(research engineer DDS)
PhDs
Shabnam Najafian
Yucheng Jin (KU Leuven)
Tim Draws
Twitter (TBA)
Multi-stakeholder recs (TBA)
nl4xai.eu (TBA)
PhD and Postdoc
openings planned!

36. Questions?
navatintarev.com
n.tintarev@tudelft.nl
@navatintarev

37. References (1)
• Conati, C., Carenini, G., Hoque, E., Steichen, B., and Toker, D. (2014). “Evaluating the
impact of user characteristics and different layouts on an interactive visualization for
decision making,” in Eurographics Conference on Visualization (EuroVis).
• Ferwerda, B., Graus, M. P., Vall, A., Tkalcic, M., & Schedl, M. (2017, April). How item
discovery enabled by diversity leads to increased recommendation list attractiveness. In
Proceedings of the Symposium on Applied Computing (pp. 1693-1696). ACM.
• Jin, Y., Htun, N.N., Tintarev, N., and Verbert, K.. "Contextplay: user control for context-
aware music recommendation". In ACM Conference on User Modeling and Adaptation
and Personalization (UMAP). 2019.
• Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics
on music recommender systems with different levels of controllability. In Proceedings of
the 12th ACM Conference on Recommender Systems (pp. 13-21). ACM.
• Jin, Y., Tintarev, N., & Verbert, K. (2018, July). Effects of individual traits on diversity-
aware music recommender user interfaces. In Proceedings of the 26th Conference on
User Modeling, Adaptation and Personalization (pp. 291-299). ACM.
• Jin, Y., Htun, N.N., Tintarev, N., and Verbert, K (2019). "Effects of personal characteristics
in control-oriented user interfaces for music recommender systems". User Modeling and
User-Adapted Interaction, 2019.
• Kang, B., Tintarev, N., Höllerer, T., & O’Donovan, J. (2016). What am I not seeing? An
interactive approach to social content discovery in microblogs. In International
Conference on Social Informatics (pp. 279-294). Springer, Cham.

38. References (2)
• Kapcak, Ö., Spagnoli, S., Robbemond, V., Vadali, S., Najafian, S. and N., Tintarev
(2018) TourExplain: A Crowdsourcing Pipeline for Generating Explanations for Groups of
Tourists. ACM RecSys Workshop on Recommenders in Tourism.
• Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012).
Explaining the user experience of recommender systems. User Modeling and User-
Adapted Interaction, 22(4-5), 441-504.
• Kobsa, A. (2001). “An empirical comparison of three commercial information
visualization systems,” in IEEE Symposium on Information Visualization (InfoVis) (San
Diego, CA).
• Müllensiefen, D., Gingras, B., Musil, J., & Stewart, L. (2014). The musicality of non-
musicians: an index for assessing musical sophistication in the general population. PloS
one, 9(2), e89642.
• Najafian, S., & Tintarev, N. (2018). Generating Consensus Explanations for Group
Recommendations: an exploratory study. In Adjunct Publication of the 26th Conference
on User Modeling, Adaptation and Personalization (pp. 245-250). ACM.
• Pu, P., Chen, L., & Hu, R. (2011). A user-centric evaluation framework for recommender
systems. In Proceedings of the fifth ACM conference on Recommender systems (pp. 157-
164). ACM.
• Shah, P., and Freedman, E. G. (2011). Bar and line graph comprehension: an interaction
of top-down and bottom-up processes. Top. Cogn. Sci. 3, 560–578.

39. References (3)
• N. Tintarev & J. Masthoff, Evaluating the effectiveness of explanations for
recommender systems: methodological issues and empirical studies on the
impact of personalization UMUAI 2012
• (Tintarev2018a) Tintarev, N., Rostami, S., & Smyth, B. (2018). Knowing the
unknown: visualising consumption blind-spots in recommender systems. In
ACM Symposium On Applied Computing (SAC).
• (Tintarev2018b) Tintarev, N., Sullivan, E., Guldin, D., Qiu, S., & Odjik, D.
(2018). Same, Same, but Different: Algorithmic Diversification of Viewpoints
in News. In Adjunct Publication of the 26th Conference on User Modeling,
Adaptation and Personalization (pp. 7-13).
• Toker, D., Conati, C., Carenini, G., and Haraty, M. (2012). “Towards adaptive
information visualization: on the influence of user characteristics,” in User
Modeling, Adaptation, and Personalization.
• Velez, M. C., Silver, D., and Tremaine, M. (2005). “Understanding visualization
through spatial ability differences,” in VIS 05. IEEE Visualization, 2005, 511–
518.
• Willemsen, M. C., Knijnenburg, B. P., Graus, M. P., Velter-Bremmers, L. C., &
Fu, K. (2011). Using latent features diversification to reduce choice difficulty
in recommendation lists. RecSys, 11, 14-20.

40. Acceptance
40
H1: The UI setting has a significant effect on recommendation acceptance
(Cannot accept)

41. Perceived diversity
41
Full Control vs Control+ComBub (M=5.43, SD=1.07, p = .005)
Full Control vs Control+SimBub (M=5.22, SD=1.20, p = .035)

42. Personal characteristics
42
Correlation results on acceptance and perceived diversity (* denotes statistical
significance at the 0.05 level, i.e., p value< 0.05)

43. Discussion
• Adding visualization to Full Control seems to increase diversity significantly.
• Control + SimBub
• Positive correlation between MS and two dependent variables (acceptance,
perceived diversity)
• Control + ComBub
• Positive correlation between personal characteristics (MS, VM) and the perceived
diversity in ComBub
• MS also has a positive effect on acceptance
•  users with high MS are good at tuning the recommender to find high
quality recommendations
43

44. 44