1) Recommendations should be viewed as a conversation with the user, where asking questions of the user and adapting based on feedback is important. 2) Good recommendations ask good questions of the user initially and over time, make mistakes but provide explanations, and adapt to the user's feedback. 3) The goal is to consider asking the user for information rather than just guessing, provide explanations for recommendations, and allow users to correct mistakes through feedback to improve recommendations over time.

1. Recommendations as a
Conversation with the User
Daniel Tunkelang
Principal Data Scientist at LinkedIn
Recruiting Solutions 1

2. Introductions
2

3. Let’s talk about how we talk with machines…
3

4. Clifford Nass’s secret:
1) Find a conclusion by a social science researcher.
2) Change
“People do X when interacting with other people.”
to
“People do X when interacting with a computer.”
3) Profit!
4

5. Let’s work on our relationship.
5

6. Core Message
Recommendations are a conversation with the user.
1) Consider asking vs. guessing.
2) Ask good questions.
3) It's ok to make mistakes…
if you have a good explanation
and adapt to feedback.
6

7. Our goal:
http://www.wilsoninfo.com/computerclipart.shtm
l
7

8. Overview
1) Theory
2) Examples
3) Action Items
8

9. 1) Theory
9

10. Pragmatics: the Study of Conversation
Paul Grice
10

11. Grice’s Maxims of Conversation
Maxim 1: Quality
Maxim 2: Quantity
Maxim 3: Relation
Maxim 4: Manner
H. P. Grice, "Logic and conversation” [1975]
11

12. Maxim 1: Quality
12

13. Quality: Above All, the Truth
Xiao, Bo and Benbasat, Izak. 2011. "Product-Related Deception in E-Commerce: A
Theoretical Perspective," MIS Quarterly, (35: 1) pp.169-195.
13

14. Don’t Lie
1) Don’t use “recommended” when you really mean
“sponsored” or “excess inventory”.
2) Optimize for the user’s utility.
3) Apply a standard of evidence (quality, quantity) that
you believe in.
14

15. Maxim 1: Quantity
15

16. Right Amount of Information
1) Exchange small units of information.
2) If recommendations supplement other content,
consider overall cognitive load.
3) Provide short, meaningful explanations.
16

17. Maxim 3: Relation
17

18. Relevant to the User
1) Offer value to the user.
2) Respect task context.
3) Don’t be obnoxious.
18

19. Maxim 4: Manner
19

20. Relevant to the User
1) Eschew obfuscation.
2) Avoid ambiguity.
3) Be brief.
4) Be orderly.
20

21. Another Perspective
Gary Marchionini
21

22. Human-Computer Information Retrieval
Empower people to explore large-scale information
but demand that
people also take responsibility for this control
by expending cognitive and physical energy.
Marchionini, G., “Toward Human-Computer Information Retrieval” [2006]
22

23. Principles of HCIR
1) Do more than deliver relevant information:
facilitate sensemaking.
2) Increase user responsibility and control:
require and reward effort.
3) Adapt to increasingly knowledgeable users over time.
4) Be engaging and fun to use!
23

24. Facilitate Sensemaking
24

25. Require and Reward Effort
http://www.posterenvy.com/catalog/ask_why.jpg
25

26. Adapt to User Knowledge
26

27. Be Engaging!
http://bluenile.com/
27

28. Applying the theory to…
1) Personalized Recommendations
2) Social Recommendations
3) Item Recommendations
28

29. Personalized Recommendations
1) Be transparent about model so users gain insight.
2) Allow users to modify models to correct mistakes.
3) Solicit just enough information to provide value.
29

30. Social Recommendations
1) Identify the right set of similar users.
2) Allow users to manipulate the social lens.
3) Accommodate users who break your model.
30

31. Item Recommendations
1) Explain recommendations to users.
2) Watch out for non-sequiturs (e.g., diapers -> beer).
3) Play well with user-controlled filtering and sorting.
31

32. 2) Examples
32

33. 33

34. Initial User Experience
34

35. “It just takes 2 minutes…”
35

36. Asking Before Guessing
36

37. Let’s try some answers:
37

38. Uh oh…
38

39. Expressing my gustibus…
39

40. New Star Trek = Yes; New Star Wars = No
40

41. Testing my patience…
41

42. Bring on the quality!
42

43. And continue the conversation.
43

44. Learning from Netflix
1) Ask the user for help up front. But not too much help.
2) Pay attention to what the user tells you!
3) Give users value early and often.
75% of Netflix views result from recommendations
44

45. 45

46. Initial User Experience
46

47. Seed with an artist…
47

48. Or track or genre.
48

49. Goo Goo G'joob!
49

50. Ease user into recommendation space…
50

51. And go wild!
51

52. Shared Product: Personalized Stream
52

53. Positive and Negative Feedback
53

54. Learning from Pandora
1) Get meaningful input from user in one step.
2) Explain recommendations to users.
3) Solicit feedback and act on it immediately.
54

55. 55

56. My home page…
56

57. Explanations and Humility
57

58. Explain What and Why
58

59. Recommendations as a Starting Point
59

60. Learning from Amazon
1) Show the factors that drive your conclusions.
2) Distinguish different kinds of recommendations.
3) Combine recommendations with user control.
Amazon: 35% of sales result from recommendations
60

61. 3) Action Items
61

62. Increase explainability.
Explanations can be even more important than the
recommendations themselves.
Herlocker et al., “Explaining collaborative filtering recommendations” [2000]
Sinha and Swearingen, “The role of transparency in recommender systems”
[2002]
Tintarev and Masthoff, “Effective explanations of recommendations: User-
centered design” [2007]
(via Òscar Celma’s book, Music Recommendation and Discovery: The Long
Tail, Long Fail, and Long Play in the Digital Music Space)
62

63. Some models more explainable than others.
1) Consider decision trees and rule-based systems.
2) Avoid using latent, unlabeled features.
3) If the model is opaque, use examples as surrogates.
63

64. Make a good first impression.
Your user’s first experience is critical.
Use popularity as a default if it makes sense.
Solicit one valuable piece of information as quickly and
painlessly as possible.
“Do you like the taste of beer?”
http://blog.okcupid.com/index.php/the-best-questions-for-first-dates/
64

65. Design feedback into your system.
You can make mistakes, if users can easily fix them.
Challenging if models use offline computation.
Respond instantly; generalize as quickly as possible.
Agarwal and Chen, “Machine Learning for Large Scale Recommender Systems”
[ICML 2011 Tutorial]
65

66. Integrate recommendations with search.
Recommend next steps, not just items.
In a task context, recommendations are just another
source of information scent.
Be careful in integrating offline recommendations with
online features like search and navigation.
Pirolli, Information Foraging Theory: Adaptive Interaction with Information [2007]
66

67. Summary
Recommendations are a conversation with the user.
1) Consider asking vs. guessing.
2) Ask good questions.
3) It's ok to make mistakes…
if you have a good explanation
and adapt to feedback.
67

68. Thank You!
Questions?
Contact:
dtunkelang@linkedin.com
We’re Hiring!
http://engineering.linkedin.com/
68