Mixed-initiative recommender systems: towards a next generation of recommender systems through user involvement
•
2 likes•345 views
This document summarizes Katrien Verbert's research experience and interests. It outlines her positions at KU Leuven from 2003 to present, where she has focused on recommender systems, visualization, and learning analytics. Her work aims to make recommendations more understandable and give users more control over the recommendation process. Key projects include TalkExplorer, which visualized recommendations from multiple perspectives, and IntersectionExplorer, which used a Venn diagram to show item relevance across user tags and recommender agents. User studies on these systems found that allowing exploration of item intersections increased effectiveness and user satisfaction with recommendations. The document also provides an overview of Verbert's research topics from 2012 to 2018, which span learning analytics, media consumption, research information systems
Recommended
More Related Content
Similar to Mixed-initiative recommender systems: towards a next generation of recommender systems through user involvement
Similar to Mixed-initiative recommender systems: towards a next generation of recommender systems through user involvement (20)
More from Katrien Verbert
More from Katrien Verbert (15)
Recently uploaded
Recently uploaded (20)
Mixed-initiative recommender systems: towards a next generation of recommender systems through user involvement
- 1. Mixed-initiative Recommender Systems Katrien Verbert Augment/HCI - KU Leuven @katrien_v
- 3. Human-Computer Interaction group PhD. researcher Oct. 2003 – Feb. 2008 Post-doc Feb. 2008 – Dec. 2012 Assistant Professor Jan. 2013 – Sept 2014 Assistant Professor Oct. 2014 – Sept. 2018 learning analytics – recommender systems – visualisation Associate Professor Oct. 2018 – …
- 4. Augment/HCI team Robin De Croon Postdoc researcher Katrien Verbert Associate Professor Francisco Gutiérrez PhD researcher Tom Broos PhD researcher Martijn Millecamp PhD researcher Sven Charleer Postdoc researcher Nyi Nyi Htun Postdoc researcher Houda Lamqaddam PhD researcher Yucheng Jin PhD researcher Oscar Alvarado PhD researcher http://augment.cs.kuleuven.be/ Diego Rojo Carcia PhD researcher
- 5. Peter Brusliovsky Nava Tintarev Cristina ConatiDenis Parra Collaborations
- 7. 7
- 8. Verbert, Katrien; Manouselis, Nikos; Ochoa, Xavier;Wolpers, Martin; Drachsler, Hendrik; Bosnic, Ivana; Duval, Erik. Context- aware recommender systems for learning: a survey and future challenges, IEEE Trans. on Learning Technologies, 18 p. (2012) 8
- 9. 9
- 10. Combining recommendation and visualization Core objectives: • make recommendations understandable for users • enable users to steer the recommendation process
- 11. Flexible interaction with RecSys Research visit ¤ Host: Carnegie Mellon University & University of Pittsburg ¤ Collaboration: John Stamper, Peter Brusilovsky, Denis Parra ¤ Period: April 2012 – June 2012 (3 months) Second post-doctoral fellowship FWO ¤ host university: KU Leuven, Belgium ¤ supervisor: Erik Duval ¤ period: Oct 2012 – Sept 2015 11
- 12. Overview research topics 12 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 Learning Analytics - Media Consumption – Research Information Systems - Healthcare
- 13. Motivation 1 3 ¤ multiple relevance prospects in personalized social tagging systems ¤ community relevance prospects ¤ social relevance prospect ¤ content relevance prospect ¤ existing personalized social systems ¤ do not allow to explore and combine multiple relevance prospects ¤ only one prospect can be explored at a given time
- 14. Also recommendations è personalized relevance prospect 14
- 15. Shortcomings recommender systems 15 ¤ cold-start issues ¤ difficult to explain the rationale behind recommendations ¤ user control often limited
- 16. 16 He, C., Parra, D., & Verbert, K. (2016). Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications, 56, 9-27.
- 17. John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual interactive recommendation. CHI '08 Related work: PeerChooser
- 18. Smallworlds 18 Gretarsson, B., O'Donovan, J., Bostandjiev, S., Hall, C. and Höllerer, T. SmallWorlds: Visualizing Social Recommendations. Comput. Graph. Forum, 29, 3 (2010), 833-842.
- 20. Contributions ¤ new approach to support exploration, transparency and controllability ¤ recommender systems are shown as agents ¤ in parallel to real users and tags ¤ users can interrelate entities to find items ¤ evaluation study that assesses ¤ effectiveness ¤ probability of item selection 20 Verbert, K., Parra, D., Brusilovsky, P., & Duval, E. (2013). Visualizing recommendations to support exploration, transparency and controllability. In Proceedings of the IUI 2013 international conference on Intelligent user interfaces (pp. 351-362). ACM.
- 21. Conference Navigator The studies were conducted using Conference Navigator 3 http://halley.exp.sis.pitt.edu/cn3/
- 22. TalkExplorer - I Entities Tags, Recommender Agents, Users 22
- 24. Our Assumptions ¤ Items which are relevant in more that one aspect could be more valuable to the users ¤ Displaying multiple aspects of relevance visually is important for the users in the process of item’s exploration 24
- 25. User study 1 ¤ Setup ¤ supervised user study ¤ 21 participants at UMAP 2012 and ACM Hypertext 2012 conferences ¤ Procedure ¤ Tasks ¤ interact with users and their bookmarks ¤ interact with agents ¤ interact with tags ¤ Post-questionnaire 25
- 26. Evaluation ¤ Data collection ¤ recordings of voice and screen using camtasia studio ¤ system logs ¤ Measurements ¤ effectiveness: # bookmarked items / #explorations ¤ yield: : # bookmarked items / sum of items in selection set 26
- 27. Summary results Sign. effect. Sign. yield multiple versus one entity 0.003 <0.001 user versus (user + entity) 0.593 <0.001 agent versus (agent + entity) 0.341 <0.001
- 28. User study 2 ¤ Setup ¤ Unsupervised user study ¤ Conducted at LAK 2012 and ECTEL 2013 (18 users) ¤ Subjects familiar with visualizations, but not much with RecSys ¤ Procedure ¤ Users were left free to explore the interface. ¤ Interactions were logged ¤ Post-questionnaire 28 Verbert, K., Parra, D., & Brusilovsky, P. (2016). Agents vs. users: Visual recommendation of research talks with multiple dimension of relevance. ACM Transactions on Interactive Intelligent Systems (TIIS), 6(2), 11.
- 29. Summary results 29 Sign. effect. multiple versus one entity <0.001 user versus (user + entity) 0.3682 agent versus (agent + entity) 0.4426
- 30. Results of Studies 1 & 2 ¤ Effectiveness increases with intersections of more entities ¤ Effectiveness wasn’t affected in the field study (study 2) ¤ … but exploration distribution was affected 30 Average effectiveness Total number of explorations
- 31. Drawback: visualizing intersections Venn diagram: more natural way to visualize intersections 31 Clustermap Venn diagram Verbert, K., Parra, D., Brusilovksy, P. (2014). The effect of different set-based visualizations on user exploration of recommendations. In : IntRS@RecSys, 2014 (pp. 37-44).
- 32. IntersectionExplorer (IEx) 32 Cardoso, B., Sedrakyan, G., Gutiérrez, F., Parra, D., Brusilovsky, P., & Verbert, K. (2018). IntersectionExplorer, a multi-perspective approach for exploring recommendations. International Journal of Human-Computer Studies.
- 34. Research questions ¤ RQ1: Under which condition may a scalable visualisation increase user acceptance of recommended items? ¤ RQ2: Does a scalable set visualisation increase perceived effectiveness of recommendations. ¤ RQ3: Does a scalable set visualisation increase user trust in recommendations? ¤ RQ4: Does a scalable set visualisation improve user satisfaction with a recommender system? 34
- 35. Study 1 ¤ Within-subjects study with 20 users ¤ Mean age: 32.9 years; SD: 6.32; female: 3 ¤ Baseline: exploration of recommendations in CN3 ¤ Second condition: exploration of recommendations in IE ¤ Data from two conferences: ¤ EC-TEL 2014 (172 items) ¤ EC-TEL 2015 (112 items) 35
- 36. Results 36
- 37. Subjective feedback Questionnaire results with statistical significance. Differences between the aspects “Fun” and “Choice satisfaction” were not significant after the Bonferroni-Holm correction. 37
- 38. Study 2: Digital Humanities 38 ¤ 39 users, less technically oriented ¤ Mean age: 38 years; SD: 10; female: 11 ¤ Data from DH conference: +1000 participants
- 39. Results 39
- 40. Study 3: IUI 2017 40 ¤ 42 users, technically oriented ¤ Mean age: 32.4 years; SD: 10; female: 17 ¤ Data from IUI conference: 111 accepted papers
- 41. Results 41
- 43. Study 1 vs Study 2 vs Study 3 ¤ Overall ”augmented agents” were used in all three studies ¤ Participants of study 2 (Digital Humanities) ¤ more interested in content perspective ¤ Rated several dimensions lower (use intention, fun, information sufficiency, control) 43
- 44. Overview research topics 44 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 Learning analytics - Media Consumption – Research Information Systems - Healthcare
- 45. Research questions ¤ How do personal characteristics influence the impact of ¤ RQ1: user controls in terms of diversity and acceptance? ¤ RQ2: visualizations in terms of diversity and acceptance? ¤ RQ3: visualizations + user controls in terms of diversity and acceptance? 45
- 46. 46
- 47. Study procedure ¤ Tutorial of study ¤ Pre-study questionnaire: ¤ Demographics ¤ Experience ¤ Visual Memory (VM) Capacity ¤ Musical Sophistication (MS) ¤ Task: Based on a chosen scenario for creating a play-list, explore songs and rate all songs in the final playlist ¤ Post-study questionnaire: ¤ perceived quality ¤ perceived accuracy ¤ perceived diversity ¤ Satisfaction ¤ Effectiveness ¤ choice difficulty
- 48. Experimental platform The Spotify API allows to generate recommendations ¤ based on up to five favourite artists. ¤ 14 musical attributes in order to describe musical preference 48
- 49. Dependent variables ¤ Perceived diversity: self-reported measure based on questionnaire ¤ Recommendation acceptance: measured by percentage of liked songs in the play-list 49
- 50. Personal characteristics ¤ Musical sophistication (MS): ¤ measurement of the ability to engage with music in a flexible, effective and nuanced way (Müllensiefen et al., 2014) ¤ Measured using the Goldsmiths Musical Sophistication Index (Gold-MSI) ¤ Visual memory (VM): ¤ the ability to recall visual patterns (Tintarev and Mastoff, 2016) ¤ Measured by Corsi block-tapping test 50
- 51. User-centered factors ¤ Exp.1 and Exp.3 used framework of Knijnenburg et al. 2012 ¤ Perceived quality: participants' perceived quality of the recommended songs. ¤ Perceived accuracy: participants' perceived accuracy of the recommended songs according to their preference. ¤ Perceived diversity: the similarity among the recommended songs. ¤ Satisfaction: participants' satisfaction about their chosen recommendations. ¤ Choice difficulty: difficulty of choosing a recommended song. ¤ Effectiveness: usefulness of recommendations generated from systems. ¤ Exp. 2 used the ResQue framework 51
- 52. Hypotheses ¤ H1: The UI setting (user control, visualization, or both) has a significant effect on recommendation acceptance. ¤ H2: The UI setting (user control, visualization, or both) has a significant effect on perceived diversity. ¤ H3: Visual memory (VM) has a significant effect on recommendation acceptance. ¤ H4: Visual memory (VM) has a significant effect on perceived diversity. ¤ H5: Musical sophistication (MS) has a significant effect on recommendation acceptance. ¤ H6: Musical sophistication (MS) has a significant effect on perceived diversity. 52
- 53. Participants 53
- 54. Experiment 1: Effects of User control Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics on music recommender systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 13-21). ACM.
- 55. User profile (PRO) Algorithm parameters (PAR) Recommendations (REC)
- 56. Evaluation method ¤ Between-subjects ¤ Independent variable: settings of user control ¤ 2x2x2 factorial design ¤ Dependent variables: ¤ Acceptance (ratings) ¤ Cognitive load (NASA-TLX), MS, VM ¤ Framework Knijnenburg et al.
- 57. CFA - validity questions Quality Accuracy Effectiveness Choice difficulty Diversity Trust Satisfaction The fit of our SEM model X(2, 98) = 257.410 p <.001 RMSEA= 0.083 CFI = 0.980 TLI = 0.968
- 58. CFA - validity questions Quality Accuracy Effectiveness Choice difficulty Diversity (low AVE value) Trust (low AVE value) Satisfaction (modification index) The fit of our SEM model X(2, 98) = 257.410 p <.001 RMSEA= 0.083 CFI = 0.980 TLI = 0.968
- 59. Conclusion ¤ Main effects: from REC to PRO to PAR → higher cognitive load ¤ Two-way interaction: does not necessarily result in higher cognitive load. Adding an additional control component to PAR increases the acceptance. PRO*PAR has less cognitive load than PRO and PAR ¤ Three-way interaction: it increases acceptance, and does not lead to higher cognitive load. Increase interaction times and accuracy ¤ High MS leads to higher quality, and thereby result in higher acceptance 59
- 60. Experiment 2: effects of visualizations Yucheng Jin, Nava Tintarev, and Katrien Verbert. 2018. Effects of Individual Traits on Diversity-Aware Music Recommender User Interfaces. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization (UMAP '18). ACM, New York, NY, USA, 291-299. DOI: https://doi.org/10.1145/3209219.3209225
- 62. UI Design - Baseline SimBub 62
- 63. Research Questions 63 ¤ How do personal characteristics influence the impact of ¤ RQ1: user controls in terms of diversity and acceptance? ¤ RQ2: visualizations in terms of diversity and acceptance? ¤ RQ3: visualizations + user controls in terms of diversity and acceptance?
- 64. Study design ¤ Within-subjects ¤ Independent variable: ¤ Type of bubble charts ¤ Dependent variables: ¤ Perceived diversity ¤ Overall usability ¤ Support to identify blind-spots 64
- 65. Measurements ¤ VM: Corsi block-tapping test ¤ MS: Goldsmiths Musical Sophistication Index ¤ Support to identify blind spots: #explored genres / #available genres ¤ Perceived diversity: ResQue (item-item diversity, categorial diversity, novelty, and serendipity) ¤ Usability: ResQue (usefulness, user’s attitude, behavioral intention) 65
- 66. Acceptance 66 H1: The UI setting (visualization) has a significant effect on recommendation acceptance (Cannot accept)
- 67. Perceived diversity 67 H2: The UI setting (visualization) has a significant effect on perceived diversity (Cannot accept) Mann-Whitney test (U = 2614.00, p = .08)
- 68. Personal characteristics 68 ComBub supports the participants with high VM to gain higher perceived diversity than SimBub. (H4) ComBub supports the participants with high MS to gain higher perceived diversity than SimBub. (H6)
- 69. Discussion ¤ In general, visualizing the audio features of music has a limited impact on perceived diversity ¤ Positive correlation between personal characteristics (MS, VM) and the perceived diversity in ComBub 69
- 70. Experiment 3: Effects of Combining User Control and Visualizations 70
- 71. User interface ¤ Full Control + ComBub ¤ Full Control + SimBub 71
- 74. Research Questions 74 ¤ How do personal characteristics influence the impact of ¤ RQ1: user controls in terms of diversity and acceptance? ¤ RQ2: visualizations in terms of diversity and acceptance? ¤ RQ3: visualizations + user controls in terms of diversity and acceptance?
- 75. Acceptance 75 H1: The UI setting has a significant effect on recommendation acceptance (Cannot accept)
- 76. Perceived diversity 76 Full Control vs Control+ComBub (M=5.43, SD=1.07, p = .005) Full Control vs Control+SimBub (M=5.22, SD=1.20, p = .035)
- 77. Personal characteristics 77 Correlation results on acceptance and perceived diversity (* denotes statistical significance at the 0.05 level, i.e., p value< 0.05)
- 78. Discussion ¤ Adding visualization to Full Control seems to increase diversity significantly. ¤ Control + SimBub ¤ Positive correlation between MS and two dependent variables (acceptance, perceived diversity) ¤ Control + ComBub ¤ Positive correlation between personal characteristics (MS, VM) and the perceived diversity in ComBub ¤ MS also has a positive effect on acceptance ¤ => users with high MS are good at tuning the recommender to find high quality recommendations 78
- 79. Overview research topics 79 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 Learning Analytics - Media Consumption – Research Information Systems - Healthcare
- 80. Additional personal characteristics ¤ Need for cognition, visual literacy, … ¤ In collaboration with Cristina Conati 80
- 81. Recommender systems for health 81
- 82. Augmented reality 82 Gutiérrez, Francisco, Htun, Nyi Nyi, Charleer, Sven, De Croon, Robin, Verbert, Katrien (2019) Designing Augmented Reality Applications for Personal Health Decision-Making. Proceedings of HICSS-52.
- 83. Tangible Algorithms ¤ Study with Netflix users ¤ Semiotic inspection ¤ Design workshop ¤ Interviews ¤ Abstract representations ¤ Archetype representations 83 Alvarado, O., Geerts, D. and Verbert, K. Towards Tangible Algorithms: Exploring Algorithmic Experience with Users’ Profiling Representations. Submitted to CHI 2019.
- 84. Job explorer 84
- 85. RECOMMENDER ALGORITHMS MACHINE LEARNING INTERACTIVE DASHBOARDS SMART ALERTS RICH CARE PLANS OPEN IoT ARCHITECTURE