This document discusses interfaces for providing transparent and user-controlled recommendations. It outlines problems with single ranked lists and proposes solutions like explaining recommendations, visualizing relevance processes, and allowing users to explore and control personalization. Specific solutions discussed include visualization tools, open learner/user models, and interfaces that combine relevance from multiple sources and allow controlling factors. Studies found that explorable and controllable recommendations better support understanding relevance and finding relevant items.

1. Interfaces for User-Controlled
and Transparent
Recommendations
Peter Brusilovsky
with Jae-Wook Ahn, Denis Parra, Katrien
Verbert, Chun-Hua Tsai, Jordan Barria-
Pineda

2. Outline
• Fighting the ranked list: problems and solutions
• Transparency and control: two sides of the coin
• Solutions for single-source ranking
– Visualize! Explore! Control!
• Combining relevance sources in Conference Navigator
– TalkExplorer
– Intersection Explorer
– SetFusion
– RelevanceTuner
• Transparency and control for educational recommendation
2

3. 3

4. • Why an item is at a specific position?
– Items might be relevant for to the user profile (or
query) for different reasons
• Single-source: different parts/aspects of the profile
• Hybrid: different sources of information or approaches
• It might not be the right position!
– A recommendation approach is tuned to an
overall/generic situation, but users could consult
recommendation for different needs
– Some profile aspects, sources, approaches are less
relevant in the current context, but some are more
4
While Single Ranked List is A Problem?

5. What are Possible Solutions?
• Explain (words)
– Why a specific item is considered relevant?
– Why it is placed in a specific position?
• Visualize (beyond ranking list)
– Make the ranking/relevance process transparent
• Explore (change visualization)
– Change visualization parameters to play with the results,
better understand the process, isolate most relevant results
• Control (change how personalization work)
– Change user profile
– Change parameters (how personalization is produced)
5

6. Two Sides of the Same Coin
Explain Visualize
ExploreControl
6
Transparency
Interactivity
No full transparency
without interactivity
Control is challenging
without transparency

7. VISUALIZE!
Present recommendations visually helping users to understand
how relevance mechanism work
7

8. KWIC: Simple relevance visualization
8

9. InfoCrystal (Spoerri 1993)
Spoerri,Anselm.1993."InfoCrystal:avisualtoolforinformationretrieval
andmanagement."InCIKM'93:thesecondinternationalconferenceon
Informationandknowledgemanagement,11-20.Washington,D.C.,United
States:ACM.

10. VIBE (Olsen, 1993)
10
Olsen, K. A., R. R. Korfhage, K. M. Sochats, M. B. Spring, and J. G. Williams. 1993. 'Visualisation of a document
collection: The VIBE system', Information Processing and Management, 29.

11. More "InfoCrystals"
11

12. TileBars (Hearst, 1995)
12
Hearst,MartiA.1995."TileBars:Visualizationoftermdistributioninformation
infulltextinformationaccess."InCHI'9559-66.Denver:ACM.

13. John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: visual
interactive recommendation. CHI '08
PeerChooser (O’Donovan, 2008)
13

14. Explaining CBR (Tsai, 2019)
Recommending people to meet at the conference using cosine similarity of
users’ publications.
Tsai, Chun-Hua, and Peter Brusilovsky. 2019. "Evaluating Visual Explanations for Similarity-Based Recommendations:
User Perception and Performance." In the 27th ACM Conference on User Modeling, Adaptation and Personalization,
UMAP 2019, 22-30. Larnaca, Cyprus: ACM.

15. Explaining Social Rec (Tsai, 2019)
Recommending people to meet using social connections: degree of
network distance, based on a shared co-authorship network.
Tsai, Chun-Hua, and Peter Brusilovsky. 2019. "Designing Explanation Interfaces for Transparency and Beyond " In
Workshop on Intelligent User Interfaces for Algorithmic Transparency in Emerging Technologies at the 24th ACM
Conference on Intelligent User Interfaces, IUI 2019. Los Angeles, USA.

16. Experiments with Exploration
• Adaptive Vibe (2006-2015)
– With Jae-Wook Ahn
• Relevance Explorer (2013-2016)
– With Katrien Verbert and Denis Parra
• Intersection Explorer (2017-2019)
– With Katrien Verbert, Karsten Seipp, Chen He,
Denis Parra, Bruno Cardoso, Gayane Sedrakyan,
Francisco Gutiérrez
16

17. EXPLORE!
Make the ranking process explorable. Allow users to play with
presentation parameters to understand aspects of relevance and
find best items in the given context
17

18. KS-VIBE (2006)
18
Ahn,J.-w.,Farzan,R.,andBrusilovsky,P.(2006)Atwo-leveladaptive
visualizationforinformationaccesstoopen-corpuseducationalresources.
In:ProcofWorkshopontheSocialNavigationandCommunity-Based
AdaptationTechnologiesatthe4thInternationalConferenceonAdaptive
HypermediaandAdaptiveWeb-BasedSystems,Dublin,Ireland,June20th,
2006,497-505.

19. QuizVIBE (2006)
Ahn, J.-w., Brusilovsky, P., and Sosnovsky, S. (2006) QuizVIBE: Accessing Educational Objects with Adaptive
Relevance-Based Visualization. In: Proc. of World Conference on E-Learning, E-Learn 2006, Honolulu, HI,
USA, October 13-17, 2006, AACE, pp. 2707-2714.

20. CONTROL!
Allow the user to control multiple aspects of the recommendation
process to better adapt personalization for the current context as
well as better explore recommendation results
20

21. What Can Be Controlled?
21
Profile Generation Presentation
Student Model
User Model
Single Source
Fusion
EXPLORE!

22. Open Learner Model (ELM-ART)
22
Weber, G. and Brusilovsky, P. (2001) ELM-ART: An adaptive versatile system for Web-based instruction. International Journal of Artificial
Intelligence in Education 12 (4), 351-384.

23. Open User Model (YourNews)
Ahn, J.-w., Brusilovsky, P., Grady, J., He, D., and Syn, S. Y. (2007) Open user profiles for adaptive news
systems: help or harm? In: 16th international conference on World Wide Web, WWW '07, Banff, Canada, May 8-12,
2007, ACM, pp. 11-20

24. Concept-Level Open User Model
(SciNet)
24
Glowacka, Dorota, Tuukka Ruotsalo, Ksenia Konuyshkova, Kumaripaba Athukorala, Samuel Kaski, and
Giulio Jacucci. 2013. "Directing Exploratory Search: Reinforcement Learning from User Interactions with
Keywords." In international conference on Intelligent user interfaces, IUI '2013, 117-27. Santa Monica, USA:
ACM Press.

25. O'Donovan, John, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. "PeerChooser: visual
interactive recommendation." In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing
systems, 1085-88. Florence, Italy: ACM.
PeerChooser (Controllable CF)
25

26. TaskSieve: Controlable Personalized Search
Ahn, Jae-wook, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li. 2008. "Personalized Web Exploration with
Task Models." In the 17th international conference on World Wide Web, WWW '08, 1-10. Beijing, China: ACM.

27. TaskSieve Controllable Ranking
• Post-filtering
• Combine query relevance and task relevance
– Alpha * Task_Model_Score + (1-alpha) * Search Score
– Alpha : user control (0.0, 0.5, or 1.0)
• Results
– Better than regular adaptive search
– Better then non adaptive baseline even in cases when
profile was excluded
– Users were really good in deciding when to engage the
profile and how
27

28. Adaptive VIBE: Exploring and
Controlling Adaptive Search
29https://www.youtube.com/watch?v=Yt1fMEFlLVA&index=2&list=PLyCV9FE42dl7JG_i7m_kvwuYRpfwwJ4iY
Ahn,Jaewook,andPeterBrusilovsky.2013.'Adaptivevisualizationfor
exploratoryinformationretrieval',InformationProcessingandManagement,
49:1139–64.

30. VIBE based query-profile fusion
User Profile Terms
Query Terms
Documents
Mixing user profile and query terms as VIBE POI

31. • User profile is added on the same playfield
as user query
• Topology is adaptive
• Mediate between profile (green POI) and
query (red POI) terms
• Browse documents free with control on
profile and query terms
Adaptive topology in VIBE

32. Adaptive VIBE with Concepts
36

33. Some Study Results
• A sequence of user studies
– Search vs. VIBE vs. VIBE+NE
• Search -> VIBE -> VIBE+NE offers:
– Better visual separation of relevant documents (system)
– Supports better opening relevant documents (user)
• VIBE+NE supports more meanigful interaction
– No degradation found even with active visual UM
manipulation
– While over performance retained or increased
Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In:
Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA, March 29-
April 1, 2015, ACM, pp. 202-212

34. TasteWeights: Profile and Mechanism
Control
38
Knijnenburg, Bart P., Svetlin Bostandjiev, John O'Donovan, and Alfred Kobsa. 2012. "Inspectability and Control in Social
Recommenders." In 6th ACM Conference on Recommender System, 43-50. Dublin, Ireland.

35. Multiple Recommendation Sources
in Conference Navigator
• Conference Navigator
• Explore!
– TalkExplorer
– Intersection Explorer
– ScatterViz
• Control!
– SetFusion
– RelevanceTuner
39

36. Talk Relevance in Conference Navigator
• Classic content-based relevance prospects (search)
– Items that has a specific keyword
• Social relevance prospects (browsing)
– Items bookmarked by a specific user
• Tag relevance prospects (browsing)
– Items tagged by a specific tag
• Personal relevance prospects (recommendation)
– Several different recommender engines
– Each engine offer one relevance prospect
40
Brusilovsky, P., Oh, J. S., López, C., Parra, D., and Jeng, W. (2017) Linking information and people in a social
system for academic conferences. New Review of Hypermedia and Multimedia.

37. Social Prospect: Details of a talk in CN3
41

38. Social prospect: User schedules
42

39. Tag relevance: Tag page in CN3
43

40. Multiple Recommender Engines
44

41. Relevance Explorer
• Context: multiple dimensions of relevance
– social - users, content - tags, recommender engines
• Using set relevance visualization
– One dimension of relevance = one set
• Agent metaphor to mix user- tag- and
engine-based relevance
– Users, tags, and recommender systems are shown as
agents collecting relevant talks
– Multiple-relevance match -> stronger evidence
46

42. TalkExplorer
• Recommendation engines are shown as agents in parallel to users and tags
• Uses Aduna clustermap library: http://www.aduna-software.com/
47

43. Interrelations agents and users
51

44. Evaluation
• Setup
– supervised user study
– 21 participants at UMAP 2012 and ACM Hypertext 2012 conferences
• Results
– The more aspects of relevance are fused, the more effective it is for
getting to relevant items. Especially effective are fusions across
relevance dimensions
– The more relevance prospects are merged, the better is the yield, the
easier is to find good items
– Dimensions of relevance are not equal
– ADUNA approach is challenging for beyond fusion of 3 aspects 52
Verbert, K., Parra-Santander, D., and Brusilovsky, P. (2016) Agents Vs. Users: Visual Recommendation of Research Talks
with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems 6 (2), Article No. 11

45. Intersection Explorer
• Based on ideas of
Talk Explorer
• New approach for
scalable multi-set
visualization
53
Cardoso, Bruno, Gayane Sedrakyan, Francisco Gutiérrez, Denis Parra, Peter Brusilovsky, and Katrien Verbert. 2019.
'IntersectionExplorer, a multi-perspective approach for exploring recommendations', International Journal of Human-Computer
Studies, 121: 73-92.

46. Intersection Explorer at IUI2017
54
http://halley.exp.sis.pitt.edu/cn3/iestudy3.php?conferenceID=148

47. ScatterViz: Diversity-Focused
Exploration of Hybrid Recommendations
Tsai, Chun-Hua, and Peter Brusilovsky. 2018. "Beyond the Ranked List: User-Driven Exploration and Diversification of Social
Recommendation." In 23rd International Conference on Intelligent User Interfaces, 239--50. Tokyo, Japan: ACM.

48. SetFusion
• Using set relevance visualization in
the familiar Venn diagram form
– One recommendation source = one
set
• Allow controlled ranking
fusion
• Combine ranking with
annotation showing
source(s) of recommendation 57

50. Brief Results of Two Studies
• SetFusion provides strong engaging effect
– Number of engaged users, bookmarked talks,
explored talks doubled
– The effect is larger in UMAP “natural” settings
• SetFusion allows more efficient work
– Increases yield of bookmarks in relation to
overhead actions
• But only 3 dimensions of relevance!
• How to control for more than 3 dimensions?
– See our RelevanceTuner design coming next!
64

51. RelevanceTuner: Control+Visualization
in a Hybrid Social Recommender
Tsai, Chun-Hua and Peter Brusilovsky (2018) Beyond the Ranked List: User-Driven Exploration and Diversification of Social
Recommendation. In 23rd International Conference on Intelligent User Interfaces, 239--50. Tokyo, Japan: ACM.

52. Explaining Hybrid Recommendations

53. Explaining “Publication
Similarity”
cosine similarity of users’ publication text.

54. Transparent Educational Recommender?
68
Lack of
Transparency !

55. ATEC Workshop
2 0 1 9
Los Angeles
69
Explanations in Mastery Grids
Barria-Pineda,Jordan,andPeterBrusilovsky.2019."Explaining
EducationalRecommendationsThroughaConcept-levelKnowledge
Visualization."InProceedingsofthe24thInternationalConferenceon
IntelligentUserInterfaces:Companion,103--04.NewYork,NY,USA:
ACM.

56. Remedial Recommendations
Textual explanations
# of “struggled” concepts
# of “proficient concepts”
(Knowledge Est. > .66)
70
Barria-Pineda, Jordan, Kamil Akhuseyinoglu, and Peter Brusilovsky. 2019. "Explaining Need-based Educational
Recommendations Using Interactive Open Learner Models." In International Workshop on Transparent Personalization
Methods based on Heterogeneous Personal Data, ExHUM at the 27th ACM Conference On User Modelling, Adaptation
And Personalization, UMAP '19. Larnaca, Cyprus.

57. Remedial Recommendations
Remedial Visual
explanations
Related concepts highlighted
Knowledge estimates as bar-chart
Recent success rate as bar-color
Warning sign on “struggled”
concepts
71

58. Readings
• Ahn, Jae-wook, Peter Brusilovsky, Jonathan Grady, Daqing He, and Sue Yeon Syn (2007) Open user profiles
for adaptive news systems: help or harm? In the 16th international conference on World Wide Web, WWW '07, 11-20.
• Ahn, Jae-wook, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li.( 2008.) Personalized Web
Exploration with Task Models."In the 17th international conference on World Wide Web, WWW '08, 1-10. Beijing, China:.
• Ahn, J. and Brusilovsky, P. (2013) Adaptive visualization for exploratory information retrieval. Information Processing
and Management 49 (5), 1139–1164.
• Ahn, J., Brusilovsky, P., and Han, S. (2015) Personalized Search: Reconsidering the Value of Open User Models. In:
Proceedings of Proceedings of the 20th International Conference on Intelligent User Interfaces, Atlanta, Georgia, USA,
March 29-April 1, 2015, ACM, pp. 202-212
• Verbert, K., Parra-Santander, D., and Brusilovsky, P. (2016) Agents Vs. Users: Visual Recommendation of
Research Talks with Multiple Dimension of Relevance. ACM Transactions on Interactive Intelligent Systems 6 (2), Article
No. 11
• Parra, D. and Brusilovsky, P. (2015) User-controllable personalization: A case study with SetFusion. International
Journal of Human-Computer Studies 78, 43–67.
• Cardoso, Bruno, Gayane Sedrakyan, Francisco Gutiérrez, Denis Parra, Peter Brusilovsky, and Katrien
Verbert (2019). IntersectionExplorer, a multi-perspective approach for exploring recommendations, International
Journal of Human-Computer Studies, 121: 73-92.
• Verbert, K., Parra-Santander, D., Brusilovsky, P., Cardoso, B., and Wongchokprasitti, C. (2017) Supporting
Conference Attendees with Visual Decision Making Interfaces. In: Companion of the 22nd International Conference on
Intelligent User Interfaces (IUI '17), Limassol, Cyprus, ACM.
• Tsai, Chun-Hua and Peter Brusilovsky (2018) Beyond the Ranked List: User-Driven Exploration and Diversification
of Social Recommendation. In 23rd International Conference on Intelligent User Interfaces, 239--50. Tokyo, Japan: ACM.
72