The effects of personal characteristics when explaining music recommendations Presented @IUI 2019

1. To explain or not
to explain

2. To explain
or not to explain
The effects of personal characteristics when
explaining music recommendations.
2

3. Martijn Millecamp* Nyi Nyi Htun* Katrien Verbert*Cristina Conati**
3
*
**
Augment

4. Why?
4

5. 5

6. 6

7. Problem?
7

8. 8
?

9. Solution
9

10. Scrutability
Tintarev, N., & Masthoff, J. (2007, April). A survey of explanations in recommender systems. In 2007 IEEE 23rd
international conference on data engineering workshop (pp. 801-810). IEEE.
10

11. PeerChooser
O'Donovan, J., Smyth, B., Gretarsson, B., Bostandjiev, S., & Höllerer, T. (2008, April). PeerChooser: visual interactive
recommendation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1085-1088).
ACM.
11

12. SmallWorlds
12
Gretarsson, B., O'Donovan, J., Bostandjiev, S., Hall, C., & Höllerer, T. (2010, June). Smallworlds: visualizing social
recommendations. In Computer Graphics Forum (Vol. 29, No. 3, pp. 833-842). Oxford, UK: Blackwell Publishing Ltd.

13. Beyond the ranked list
Tsai, C. H., & Brusilovsky, P. (2018, March). Beyond the ranked list: User-driven exploration and diversification of social
recommendation. In 23rd International Conference on Intelligent User Interfaces (pp. 239-250). ACM.
13

14. SetFusion
14
Parra, D., Brusilovsky, P., 2015. User-controllable personalization: A case study with setfusion.

15. Intersection Explorer
15
Verbert, K., Parra, D., Brusilovsky, P., & Duval, E. (2013, March). Visualizing recommendations to support
exploration, transparency and controllability. In Proceedings of the 2013 international conference on Intelligent
user interfaces (pp. 351-362). ACM.

16. TasteWeights
16
Bostandjiev, S., O'Donovan, J., & Höllerer, T. (2012, September). Tasteweights: a visual interactive hybrid recommender system

17. 17
Solution?

18. Most recent related work
"Rational explanations are only
effective on users that report being
very unfamiliar with a task" [1]
18
[1] Schaffer, J., Playa Vista, C. A., O’Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019, March). I can
do better than your AI: expertise and explanations. In Proceedings of the 24th International Conference
on Intelligent User Interfaces (pp. 240-251). ACM.
[2] Dodge, J., Liao, Q. V., Zhang, Y., Bellamy, R. K., & Dugan, C. (2019). Explaining Models: An Empirical
Study of How Explanations Impact Fairness Judgment. arXiv preprint arXiv:1901.07694.
"We highlight that there is no one-size-fits-all
solution for effective explanations-it depends on
the kind of fairness issues and user profiles" [2]

19. User-centered evaluation framework
19
Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012). Explaining the user experience of
recommender systems. User Modeling and User-Adapted Interaction, 22(4-5), 441-504.

20. ValueChart
20
Conati, C., Carenini, G., Hoque, E., Steichen, B., & Toker, D. (2014, June). Evaluating the impact of user
characteristics and different layouts on an interactive visualization for decision making. In Computer Graphics
Forum (Vol. 33, No. 3, pp. 371-380).

21. ScaViz
21
Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics on music recommender
systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender
Systems (pp. 13-21). ACM.

22. Argumentation-Based Explanations
22
Naveed, S., Donkers, T., & Ziegler, J. (2018, July). Argumentation-Based Explanations in Recommender Systems:
Conceptual Framework and Empirical Results. In Adjunct Publication of the 26th Conference on User Modeling,
Adaptation and Personalization (pp. 293-298). ACM.

23. Combination?
23

24. Research Questions
24

25. 1. How do personal characteristics impact user perception of
the system when recommendations are explained?
2. How do personal characteristics impact user interaction
with the system when recommendations are explained?
25

26. Interface
26

27. 27
https://bit.ly/2u2cz3r

28. Personal Characteristics
28

29. Personal Characteristics
◦ Locus of control
▫ "The extent to which people believe the have power over events in their lives"
Fourier
◦ Need for cognition
▫ "Tendency for an individual to engage in, and enjoy effortful cognitive activities"
Cacioppo
◦ Visualisation Literacy
▫ "The ability to interpret and to make meaning from information presented in the
form of images and graphs" Boy
◦ Visual Working memory
▫ "The part of our cognitive system that is responsible for short-term holding of
information for further processing" Corsi
◦ Musical Experience
▫ "The construct that can refer to musical skills, expertise, achievements, and
related behaviours across a range of facets" Gold-MSI
29

30. Experimental design
30

31. Participants
◦ N = 71)
◦ Amazon Mechanical Turk
◦ $2
31
PC Mean SD
Loc 12.099 4.58
NFC 62.845 14.96
VisLit -0.073 0.977
VWM 5.93 1.477
MS 67.673 20.051

32. Procedure
◦ Pilot study
◦ N = 20
◦ Playlist of 15 songs
◦ Recommend 10 songs
32

33. Procedure
33

34. Measurements perception
◦ Recommender effectiveness
▫ The songs recommended to me match my interest
▫ The recommender helped to find good songs for my playlist
◦ Good understanding
▫ I understood why the songs were recommended to me.
▫ The information provided for the recommended songs is sufficient to make a
decision
▫ The songs recommended to me had similar attributes to my preference
◦ Trust
▫ I trust the system to suggest good songs
◦ Novelty
▫ The recommender system helped me discover new songs
◦ Use intention
▫ I will use the system again
◦ Satisfaction
▫ Overall, I am satisfied with the recommended system
◦ Confidence
▫ I am confident about the playlist I have created
34

35. Measurements interaction
◦ Nb_slider
▫ The number of times the participant used a slider
◦ precision
▫ Songs liked / songs recommended
◦ Nb_play
▫ Total number of songs played
◦ Nb_why
▫ Number of explanations clicked open
◦ Duration_why
▫ The amount of time the explanation is open
35

36. 36
So what?
To explain or
not to explain?

37. Impact on user perception
37
X2 (1) =8.73, p=0.003
Cai, C. J., Jongejan, J., & Holbrook, J. (2019, March). The effects of example-based explanations in a
machine learning interface. In Proceedings of the 24th International Conference on Intelligent User
Interfaces (pp. 258-262). ACM.

38. Impact on user interaction
38
t = -2.975, p=0.00592

39. Impact on user interaction
39
t = 2.052, p=0.04408

40. Impact on user interaction
40
t = -2.795, p=0.00592

41. Impact on user perception
41
t = -2.755, p=0.00757

42. Thematic analysis
42
Explanations
Musical attributes
Novelty
Quality of
recommendations

43. Thematic analysis
43
Explanations
Low NFC High NFC

44. Thematic analysis
44
Musical attributes
Low NFC High NFC

45. Thematic analysis
45
Novelty
Low MS High MS

46. Thematic analysis
46
Quality of recommendations

47. 47

48. Guidelines
48

49. Guidelines
49

50. 50Augment
Martijn Millecamp
https://bit.ly/2u2cz3r
@AugmentHCI
@MMillecamp94
augment@cs.kuleuven.be
martijn.millecamp@kuleuven.be
Questions?