Recommendation algorithms and their variations such as ranking are the most common way for machine learning to find its way into a product where it is not the main focus. In this talk we’ll dig into the subtleties of making recommendation algorithms a seamless and integral part of your UX (goal: it should completely fade into the background. The user should not be aware she’s interacting with any kind of machine learning, it should just feel right, perhaps smart or even a tad like cheating); how to solve the cold start problem (and having little training data in general); and how to effectively collect feedback data. I’ll be drawing from my experiences building Metabase, an open source analytics/BI tool, where we extensively use recommendations and ranking to keep users in a state of flow when exploring data; to help with discoverability; and as a way to gently teach analysis and visualization best practices; all on the way towards building an AI data scientist.

1. simon@metabase.com
@sbelak
The subtle art of
recommendation
(systems)

3. x

4. Metabase ❤
github.com/metabase/metabase
• Open source BI/analytics tool
• Runs on-premise, data-agnostic
• 15k+ companies use us daily
• Focus on UX and friendliness
• Building a “data scientist in a box”

5. Metabase ❤
github.com/metabase/metabase
• Open source BI/analytics tool
• Runs on-premise, data-agnostic
• 15k+ companies use us daily
• Focus on UX and friendliness
• Building a “data scientist in a box”

6. Flow

7. Affordances of our tools
shape how we approach
problems

8. From artificial intelligence
to augmenting human
intelligence

9. The many uses of
recommender systems

10. (more engagement or
sales)

11. Aid discoverability

12. Teach

13. Reduce friction

14. (Re)use context

16. Direct action
(“news feed”)

18. The 2 types of
recommendations:
1-on-1, most cases

19. Building a
recommender system

20. Recommend based on:
• Similar users
• Similar items
• Hybrid

21. Recommendation at a
glance
1.Project user and/or item into a feature space
2.Search/prune candidates
3.Rank
4.Pick best N candidates

22. Adding structure:
from similar to related

23. Multiple axis of fit

24. Relevancy changes with
time and external stimuli

25. Preserving diversity
with quotas

26. Multi-faceted
recommendations
1.Project user and/or item into a feature space
2.For each axis of relevancy:
1.Search/prune candidates
2.Rank
3.Pick best N candidates
3.Combine results

27. 1.Project user and/or item into a feature space
2.For each axis of relevancy:
1.Search/prune candidates
2.Rank
3.Pick best N candidates
3.Combine results
or: find centroids of clusters}
Multi-faceted
recommendations

28. The serendipity
option

29. Solving the cold-start
problem

30. Start with pre-set heuristics,
then tweak mixture based
on user feedback

31. Collecting user-
feedback (learning)

32. Every (non-)interaction
is a vote

33. Beware:
interaction does not
mean intention
Not all interactions are created equal: being ignored
is (probably) better than to disappoint

34. Beware:
novelty effect

35. Beware:
ordering matters

36. Indirect metrics: have the
overall patterns of
interaction changed?
# of actions taken, # of sessions, time in app, DAU, …

37. Time decay
(Remember: relevancy changes with time)

38. Non-linear responses

39. Test by omission

40. Explicit feedback
“Was this useful?”

41. Building a magical
experience
“This should almost feel like we’re cheating"

42. It shouldn’t feel like a
recommendation

43. Build tools for, and
seamlessly intertwine
with manual curation

44. The problem with
consistency

45. Takeouts
• Affordances of our tools shape how we approach
problems
• Recommender systems are a tool with many uses
• Make it seamless
• Embrace ambiguity and context (there’s often not one
single “correct” answer)
• Magical can be simple (if done right)

46. Questions
P.S. We’re hiring!