The document discusses the challenges and prospects of personalized social tagging systems in providing multiple relevance dimensions for users. It presents a new approach that visualizes recommendations as agents alongside user interactions, enhancing exploration and control. Evaluation studies indicate that considering multiple entities significantly improves the effectiveness of recommendation systems.

1. Agents vs Users: Visual
Recommendation of Research Talks
with Multiple Dimension of Relevance
Katrien Verbert – KU Leuven – @katrien_v
Denis Parra – PUC Chile - @denisparra
Peter Brusilovsky – University of Pittsburgh - @peterpaws
Presented at IUI 2017

2. Motivation
• multiple relevance prospects in
personalized social tagging
systems
o community relevance prospects
o social relevance prospect
o content relevance prospect
• existing personalized social
systems
o do not allow to explore and combine
multiple relevance prospects
o only one prospect can be explored at a given
time
2

3. Also recommendations  personalized
relevance prospect
3

4. Shortcomings recommender systems
• cold-start issues
• difficult to explain the rationale behind recommendations
• user control has positive effect
o several ways to control elements
o which are most effective for the user experience?
4

5. 5
He, C., Parra, D., & Verbert, K. (2016). Interactive recommender systems: A survey of the state
of the art and future research challenges and opportunities. Expert Systems with Applications,
56, 9-27.

6. John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual
interactive recommendation. CHI '08
Related work: PeerChooser
6

7. Related work:
Smallworlds
Gretarsson, B., O'Donovan, J., Bostandjiev, S.,
Hall, C. and Höllerer, T. SmallWorlds: Visualizing
Social Recommendations. Comput. Graph. Forum,
29, 3 (2010), 833-842. 7

8. Related work: TasteWeights
Bostandjiev,S.,O'Donovan,J.andHöllerer,T.TasteWeights:avisualinteractivehybrid
recommendersystem.InProceedingsofthesixthACMconferenceonRecommender
systems(RecSys'12).ACM,NewYork,NY,USA(2012),35-42.
8

9. Contributions
• new approach to support exploration, transparency and
controllability
o recommender systems are shown as agents
o in parallel to real users and tags
o users can interrelate entities to find items
• evaluation study that assesses
o effectiveness
o probability of item selection
9

10. Conference Navigator
The studies were conducted using Conference Navigator 3
http://halley.exp.sis.pitt.edu/cn3/10

11. TalkExplorer - I
11
Entities
Tags, Recommender Agents,
Users

12. TalkExplorer - II
Recommender
Recommender
Cluster
with
intersect
ion of
entities
Cluster (of
talks)
associated
to only one
entity
Canvas Area: Intersections of Different Entities
User

13. TalkExplorer - III
13
Items
Talks explored by the
user

14. Our Assumptions
• Items which are relevant in more that one aspect could be
more valuable to the users
• Displaying multiple aspects of relevance visually is
important for the users in the process of item’s exploration
14

15. User study 1
• Setup
o supervised user study
o 21 participants at UMAP 2012 and ACM Hypertext 2012 conferences
• Procedure
o Tasks
• interact with users and their bookmarks
• interact with agents
• interact with tags
o Post-questionnaire
15

16. Evaluation
• Data collection
o recordings of voice and screen using camtasia studio
o system logs
• Measurements
o effectiveness: # bookmarked items / #explorations
o yield: : # bookmarked items / sum of items in selection set
16

17. Effectiveness
results
Summary explored actions
and their effectiveness
Effectiveness = # bookmarked
items / #explorations
17

18. Summary results
Sign. effect. Sign.
yield
multiple versus one entity 0.003 <0.001
user versus (user + entity) 0.593 <0.001
agent versus (agent + entity) 0.341 <0.001

19. Post-questionnaire
19

20. Post-questionnaire
20

21. User study 2
• Setup
o Unsupervised user study
o Conducted at LAK 2012 and ECTEL 2013 (18 users)
o Subjects familiar with visualizations, but not much with RecSys
• Procedure
o Users were left free to explore the interface.
o Interactions were logged
o Post-questionnaire
21

22. Summary results
22
Sign. effect.
multiple versus one entity <0.001
user versus (user + entity) 0.3682
agent versus (agent + entity) 0.4426

23. Summary results
Sign. effectiveness
ranked list versus agents 0.7008
ranked list versus (agent + entity) <0.001
ranked list versus multiple entities 0.0012
23

24. Results of Studies 1 & 2
• Effectiveness increases
with intersections of more
entities
• Effectiveness wasn’t
affected in the field study
(study 2)
• … but exploration
distribution was affected
Average effectiveness
Total number of explorations
24

25. Drawback: visualizing intersections
Clustermap Venn diagram
Venn diagram: more natural way to visualize intersections
25
Verbert, K., Parra, D., Brusilovksy, P. (2014). The effect of different set-based visualizations on user
exploration of recommendations. In : IntRS@RecSys, 2014 (pp. 37-44).

26. IntersectionExplorer (IEx)
26
Verbert, K., Seipp, K., He, C., Parra, D., Wongchokprasitti, C., Brusilovsky, P. (2016). Scalable exploration of relevance
prospects to support decision making. In : IIntRS@RecSys. 2016 (pp. 28-35). CEUR-WS.

27. IntersectionExplorer (IEx)
27

28. Feedback very welcome!
Please participate in a short user study:
http://halley.exp.sis.pitt.edu/cn3/iestudy3.php?conferenceID=
148
28

29. Thank you!
Katrien.verbert@cs.kuleuven.be
peterb@pitt.edu
denisparra@gmail.com
29