Tsai, Chun-Hua, and Peter Brusilovsky. 2019. "Evaluating Visual Explanations for Similarity-Based Recommendations: User Perception and Performance." In the 27th ACM Conference on User Modeling, Adaptation and Personalization, UMAP 2019, 22-30. Larnaca, Cyprus: ACM.

1. Evaluating Visual Explanations for
Similarity-Based Recommendations:
User Perception and Performance
Chun-Hua Tsai and Peter Brusilovsky
University of Pittsburgh
Tsai, Chun-Hua, and Peter Brusilovsky. 2019. "Evaluating Visual Explanations for Similarity-Based Recommendations:
User Perception and Performance." In the 27th ACM Conference on User Modeling, Adaptation and Personalization,
UMAP 2019, pp. 22-30. Larnaca, Cyprus

2. Hybrid Social Recommender Systems
• Recommending people to meet at a
research confence
• Multiple information sources
• Traditional approach: “Optimal”
static fusion of ranking from
different sources
• Changing information needs?
• ☞ User-controlled fusion
User-Controlled Hybrid Fusion: Relevance Tuner

3. Presenting of Explanations

4. Explaining “Publication Similarity”
cosine similarity of users’ publication text.

5. Stage-based Participatory Process
Similarity-based
Recommendation
Developing
Explanation
Interfaces
Evaluating
Explanation
Interface
Three models:
• Text similarity
• Topic similarity
• Item similarity
Study 1: Comparing
Explanation Interfaces
• Card-Sorting Analysis
• Seven Explanatory Goals
Study 2: Evaluating
Explanation Interfaces
• Behavior
• User Perception
• User Performance

6. Similarity-based Recommendation Model
What can be explained?
Data
• Publication text (title &
abstract)
Method
• Publication text
• Cosine similarity
What to explain?
• Terms
• Term frequency
Data
• Publication text
(title & abstract)
Method
• Topic Modeling
• LDA
What to explain?
• Cluster topics
• Topical words
Data
• Bookmaking & following
data
Method
• Number of shared
bookmarks/links
What to explain?
• Papers
• Authors

7. • Conference Navigator System: Controlled Lab Study
• A total of 15 (6 female) participants (N=15)
• First, or second-year graduate students (major in information sciences) at the
University of Pittsburgh with age ranged from 20 to 30 (M=25.73, SE=2.89).
• All participants had no previous experience of using the CN system.
• Card Sorting tasks of 19 explanation factors and interface prototyping
• E.g., transparency, trust, persuasiveness, satisfaction, etc.
• Each participant received USD$20 compensation and signed an
informed consent form.
Developing Explanation Interfaces
Study 1: Comparing Explanation Interfaces

8. Developing Explanation Interfaces
1. Explaining Publication Similarity
Data
• Publication text
(title & abstract)
Method
• Publication text
• Cosine similarity
What to explain?
• Terms
• Term frequency

9. Developing Explanation Interfaces
2. Explaining Topic Similarity
Data
• Publication text
(title & abstract)
Method
• Topic Modeling
• LDA
What to explain?
• Cluster topics
• Topical words

10. Developing Explanation Interfaces
3. Explaining CN3 Interest Similarity
Data
• Bookmaking &
following data
Method
• Number of shared
bookmarks/links
What to explain?
• Papers
• Authors

11. • We implemented the top-rated designs and ‘enhance’ it by the
second-rated interfaces.
• Sim1+: E1-2 Two-way Bar Chart (Sim1) + E1-4 Venn Word Cloud
• Sim2+: E2-4 Topical Radar (Sim2) + E2-3 FLAME (Word Clouds)
• Sim3+: E3-4: Venn Tags (sim3) + an extra list
• We aimed to answer the following research questions (RQs):
• How does the visual interface reach the explanation goals?
• How does user perception vary with the enhanced interface?
• How does the explanation interface affect the user performance
(inspectability) across recommendations?
Assessing Explanation Interfaces
Enhancing top-rated with second-rated interfaces

12. • Conference Navigator System: Controlled Lab Study
• Six explanation interfaces (three baseline and three enhanced interfaces)
• Within-Subjects Study Design
• A total of 18 (11 female) participants (N=18)
• The subjects were required to act on the basis of visualization, not just
assess it subjectively: rank the recommendations by relevance solely based
on the visual explanations
• How well an explanation interface supports the user performance of comparing the
relevance across recommendations?
• How difficult the sorting is according to objective (clicks, time) and subjective data
• Each participant received USD$20 compensation and signed an informed
consent form.
Assessing Explanation Interfaces
Study 2: Evaluating Explanation Interfaces

13. • Enhanced interface required
significantly higher number of clicks.
• Enhanced interface received
significantly higher ratings in
• Multiple explanatory goals:
Transparency, Scrutability, Trust and,
Effectiveness.
• (NASA-TLX) Perceiving performance.
Assessing Explanation Interfaces
Sim1. Venn Word Cloud to explain text similarity
Sim1+

14. • Enhanced interface required
significantly higher number of clicks
• No significant difference in all
explanatory goals, but enhanced
interface received higher rating
• No significant difference in NASA-
TLX Survey Analysis
Assessing Explanation Interfaces
Sim2. Topical Radar to explain topic similarity
Sim2+

15. • No significant difference in number
of clicks and the time spent
• Slightly higher, but no significant
difference in all explanatory goals.
• No significant difference in NASA-
TLX Survey Analysis
Assessing Explanation Interfaces
Sim3. Venn Tags to explain item similarity
Sim2+

16. • We used the correct rate to
define the sorting difficulty
among the explanation
interfaces
• Compare gold-standard sorting
with user sorting using
Levenshtein distance
• Adding visual component can
be helpful, useless, or
misleading in assisting the
subjects to complete the
sorting task.
Assessing Explanation Interfaces
User Performance: Sorting Difficulty

17. Conclusion
• We presented two user studies of explanation interfaces for three
similarity-based recommendation models.
• Study 1: Developing Explanation Interfaces
• We selected top-rated interfaces to explain the recommendation model.
• Study 2: Assessing Explanation Interfaces
• For each model, we compared the top- rated design (baseline) with a combination of
top and second-rated interfaces (enhanced).
• The proposed explanation interfaces did reach the explanation goals.
• Adding a visual component (enhanced explanation interface) might contribute to a
higher user perception score in the explanation goals.
• However, that adding another visual component may result in increasing the
cognitive overload and even creating a mental conflict.

18. Summary
Expert
Mental
Model
User
Mental
Model
Target User
Model
Iterative
Prototyping
Evaluation
Key components:
• Publication
• Topic
• Interest
Build user
mental model
by 7
explanatory
goals
Target Mental Model
• Transparency
• Satisfaction
• Persuasiveness
• Etc.
A total of 15
explanation
interfaces for five
recommendation
features
Assessing
Visual Explanations

19. Thanks. J