Agents vs Users: Visual Recommendation of Research Talks with Multiple Dimension of Relevance


Published on

Published in ACM TiiS: Verbert, K., Parra, D., & Brusilovsky, P. (2016). Agents Vs. Users: Visual Recommendation of Research Talks with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(2), 11.
Presented at IUI 2017

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Our work has been motivated by the presence of multiple relevance prospects in modern social tagging systems. An important feature pioneered by social tagging systems and later used in other kinds of social systems is the ability to explore different community relevance prospects by examining items bookmarked by a specific user or items associated by various users with a specific tag. Items bookmarked by a specific user offer a social relevance prospect: if this user is known and trustable or appears to be like-minded (bookmarked a number of items known as interesting) a collection of his or her bookmarks is perceived as an interesting and relevant set that is worth to explore for more useful items. Similarly, items marked by a specific tag offer a content relevance prospect. Items related to a tag of interest or a tag that was used to mark many known interesting items are also perceived as potentially relevant and worth to explore. In a social tagging system extended with a personalized recommender engine
  • Related work: peerchooser John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual interactive recommendation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08).
  • - In field study, users explore mainly clusters connected to one entity
    Users don’t even get to explore interactions between 4 items
    enabling users to explore interrelationships between prospects increases probability of finding a relevant item
    interrelating tags with other entities increase their effectiveness significantly
  • Agents vs Users: Visual Recommendation of Research Talks with Multiple Dimension of Relevance

    1. 1. Agents vs Users: Visual Recommendation of Research Talks with Multiple Dimension of Relevance Katrien Verbert – KU Leuven – @katrien_v Denis Parra – PUC Chile - @denisparra Peter Brusilovsky – University of Pittsburgh - @peterpaws Presented at IUI 2017
    2. 2. Motivation • multiple relevance prospects in personalized social tagging systems o community relevance prospects o social relevance prospect o content relevance prospect • existing personalized social systems o do not allow to explore and combine multiple relevance prospects o only one prospect can be explored at a given time 2
    3. 3. Also recommendations  personalized relevance prospect 3
    4. 4. Shortcomings recommender systems • cold-start issues • difficult to explain the rationale behind recommendations • user control has positive effect o several ways to control elements o which are most effective for the user experience? 4
    5. 5. 5 He, C., Parra, D., & Verbert, K. (2016). Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities. Expert Systems with Applications, 56, 9-27.
    6. 6. John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual interactive recommendation. CHI '08 Related work: PeerChooser 6
    7. 7. Related work: Smallworlds Gretarsson, B., O'Donovan, J., Bostandjiev, S., Hall, C. and Höllerer, T. SmallWorlds: Visualizing Social Recommendations. Comput. Graph. Forum, 29, 3 (2010), 833-842. 7
    8. 8. Related work: TasteWeights Bostandjiev,S.,O'Donovan,J.andHöllerer,T.TasteWeights:avisualinteractivehybrid recommendersystem.InProceedingsofthesixthACMconferenceonRecommender systems(RecSys'12).ACM,NewYork,NY,USA(2012),35-42. 8
    9. 9. Contributions • new approach to support exploration, transparency and controllability o recommender systems are shown as agents o in parallel to real users and tags o users can interrelate entities to find items • evaluation study that assesses o effectiveness o probability of item selection 9
    10. 10. Conference Navigator The studies were conducted using Conference Navigator 3
    11. 11. TalkExplorer - I 11 Entities Tags, Recommender Agents, Users
    12. 12. TalkExplorer - II Recommender Recommender Cluster with intersect ion of entities Cluster (of talks) associated to only one entity Canvas Area: Intersections of Different Entities User
    13. 13. TalkExplorer - III 13 Items Talks explored by the user
    14. 14. Our Assumptions • Items which are relevant in more that one aspect could be more valuable to the users • Displaying multiple aspects of relevance visually is important for the users in the process of item’s exploration 14
    15. 15. User study 1 • Setup o supervised user study o 21 participants at UMAP 2012 and ACM Hypertext 2012 conferences • Procedure o Tasks • interact with users and their bookmarks • interact with agents • interact with tags o Post-questionnaire 15
    16. 16. Evaluation • Data collection o recordings of voice and screen using camtasia studio o system logs • Measurements o effectiveness: # bookmarked items / #explorations o yield: : # bookmarked items / sum of items in selection set 16
    17. 17. Effectiveness results Summary explored actions and their effectiveness Effectiveness = # bookmarked items / #explorations 17
    18. 18. Summary results   Sign. effect. Sign. yield multiple versus one entity 0.003 <0.001 user versus (user + entity) 0.593 <0.001 agent versus (agent + entity) 0.341 <0.001
    19. 19. Post-questionnaire 19
    20. 20. Post-questionnaire 20
    21. 21. User study 2 • Setup o Unsupervised user study o Conducted at LAK 2012 and ECTEL 2013 (18 users) o Subjects familiar with visualizations, but not much with RecSys • Procedure o Users were left free to explore the interface. o Interactions were logged o Post-questionnaire 21
    22. 22. Summary results 22   Sign. effect. multiple versus one entity <0.001 user versus (user + entity) 0.3682 agent versus (agent + entity) 0.4426
    23. 23. Summary results   Sign. effectiveness ranked list versus agents 0.7008 ranked list versus (agent + entity) <0.001 ranked list versus multiple entities 0.0012 23
    24. 24. Results of Studies 1 & 2 • Effectiveness increases with intersections of more entities • Effectiveness wasn’t affected in the field study (study 2) • … but exploration distribution was affected Average effectiveness Total number of explorations 24
    25. 25. Drawback: visualizing intersections Clustermap Venn diagram Venn diagram: more natural way to visualize intersections 25 Verbert, K., Parra, D., Brusilovksy, P. (2014). The effect of different set-based visualizations on user exploration of recommendations. In : IntRS@RecSys, 2014 (pp. 37-44).
    26. 26. IntersectionExplorer (IEx) 26 Verbert, K., Seipp, K., He, C., Parra, D., Wongchokprasitti, C., Brusilovsky, P. (2016). Scalable exploration of relevance prospects to support decision making. In : IIntRS@RecSys. 2016 (pp. 28-35). CEUR-WS.
    27. 27. IntersectionExplorer (IEx) 27
    28. 28. Feedback very welcome! Please participate in a short user study: 148 28
    29. 29. Thank you! 29