These are the slides of a talk about some of our research at Spotify, as part of the celebration kickoff of Chalmers AI Research Centre in Gothenburg. I always like to make a story in my talk, and this time I wanted to reflect on the "push" (think recommender system) and "pull" (think search) paradigms. I am using this quote from Nicholas Belkin and Bruce Croft from their Communications of the ACM article published in 1992 to frame my story: "We conclude that information retrieval and information filtering are indeed two sides of the same coin. They work together to help people get the information needed to perform their tasks."

1. Recommending and
Searching
Research @ Spotify
Mounia Lalmas
Chalmers University of Technology, 4-5 March 2019

2. Making AI works at Spotify
Qualitativeresearch
Businessmetrics
Algorithm(s)
Training & Datasets
Metric(s)
Evaluation offline and online
Interaction & feedbacks data
Features
(item)
Features
(user)
Features
(context)

3. What we do at Spotify

4. Spotify’s mission is to
unlock the potential of
human creativity — by
giving a million creative
artists the opportunity
to live off their art and
billions of fans the
opportunity to enjoy
and be inspired by it.

5. http://everynoise.com/

6. Our team mission:
Match fans and artists in a personal and relevant way.
ARTISTS FANS

8. songs
playlists
podcasts
...
catalog
search
browse
talk
users
What does it mean to match fans and artists
in a personal and relevant way?Artists
Fans

9. “We conclude that information retrieval and
information filtering are indeed two sides of
the same coin. They work together to help
people get the information needed to
perform their tasks.”
Information filtering and information retrieval: Two sides of the same coin? NJ Belkin & WB Croft,
Communications of the ACM, 1992.

10. “We can conclude that recommender
systems and search are also two sides of
the same coin at Spotify. They work
together to help fans get the music they will
enjoy listening”.
PULL
PARADIGM
PUSH
PARADIGM
is this the case?

11. Home … the push paradigm

12. Home
Home is the default screen of the mobile app
for all Spotify users worldwide.
It surfaces the best of what Spotify has to
offer, for every situation, personalized
playlists, new releases, old favorites, and
undiscovered gems.
Help users find something they are going to
enjoy listening to, quickly.

13. Streaming UserBaRT
Explore, Exploit, Explain: Personalizing Explainable Recommendations with Bandits. J McInerney, B Lacker, S Hansen, K Higley, H.Bouchard, A Gruson
& R Mehrotra, RecSys 2018.
BaRT: Machine learning algorithm for
Spotify Home

14. BaRT (Bandits for Recommendations as Treatments)
How to rank playlists (cards) in each shelf first, and then how to rank the shelves?

15. https://hackernoon.com/reinforcement-learning-part-2-152fb510cc54
Explore vs Exploit
Flip a coin with given probability of tail
If head, pick best card in M according to predicted reward r → EXPLOIT
If tail, pick card from M at random → EXPLORE
BaRT: Multi-armed bandit
algorithm for Spotify Home

16. Success is captured by the reward function
Reward
Binarised Streaming Time Success is when user
streams the playlist
for at least 30s.
BaRT UserStreaming

17. Is success the same for all playlists?
Consumption time of a sleep playlist is longer than average playlist consumption time.
Jazz listeners consume Jazz and other playlists for longer period than average users.

18. one reward function
for all users and all
playlists
success independent of
user and playlist
one reward function
per user x playlist
success depends on user and
playlist
too granular, sparse, noisy,
costly to generate & maintain
one reward function
per group of users x
playlists
success depends on group of
users listening to group of
playlists
Personalizing the reward function for BaRT

19. Co-clustering
Co-clustering
Dhillon, Mallela & Modha, "Information-theoretic co-clustering”, KDD 2003.
Caveat no theoretical foundation for
selecting the number of co-clusters apriori
group = cluster
group of user x playlist = co-cluster

20. Co-clustering for Spotify Home
Users
Playlists
User
groups
Playlist groups
Any (interaction) signal can be used to generate the co-clusters.

21. Reward function per co-cluster using
distribution of streaming time
continuousadditivemean

22. Counterfactual
methodology, which works
like offline A/B
Thresholding methods:
mean, additive, continuous
& random
Baseline (one threshold),
playlists only, users only,
both
One week of random
sample of 800K+ users,
900K+ playlists, 8M
user-playlist interactions
Expected stream rate
Experiments
Deriving User- and Content-specific Rewards for Contextual Bandits. P Dragone, R Mehrotra
& M Lalmas, WWW 2019.

23. Conclusions
Accounting for user experience and playlist consumption matters.
Co-clustering users and playlists surfaces patterns of user
experience x playlist consumption.
Using one interaction signal and a simple thresholding method can
already provide effective personalised success metrics.

24. Metric 1
Metric 2
Metric 3
Multiple objective functionsRecommendation
in a 2-sided
Marketplace

25. ● Policy I: Optimizing Relevance
● Policy II: Optimizing Fairness
● Policy III: Probabilistic Policy
● Policy IV: Trade-off Relevance &
Fairness
● Policy V: Guaranteed Relevance
● Policy VI: Adaptive Policy I
● Policy VI: Adaptive Policy II
“Fairness” Relevance
Optimising for fairness
and satisfaction at the
same time
Towards a Fair Marketplace: Counterfactual Evaluation of the
trade-off between Relevance, Fairness & Satisfaction in
Recommendation Systems. R Mehrotra, J McInerney, H Bouchard,
M Lalmas & F Diaz, CIKM 2018.
Recommendation
in a 2-sided
Marketplace

26. ML Lab
An offline evaluation framework to
launch, evaluate and archive machine
learning studies, ensuring
reproducibility and allowing sharing
across teams.
Offline Evaluation to Make Decisions About Playlist
Recommendation Algorithms. A Gruson, P Chandar, C Charbuillet,
J McInerney, S Hansen, D Tardieu & B Carterette, WSDM 2019.
Offline
evaluation for
Home

27. Search … pull & push paradigms

28. Searching for music

29. Overview of the user journey in search
TYPE/TALK
User
communicates
with us
CONSIDER
User evaluates
what we show
them
DECIDE
User ends the
search session
INTENT
What the user
wants to do
MINDSET
How the user
thinks about
results

30. FOCUSED
One specific thing in mind
OPEN
A seed of an idea in mind
EXPLORATORY
A path to explore
● Find it or not
● Quickest/easiest
path to results is
important
● From nothing good
enough, good enough
to better than good
enough
● Willing to try things out
● But still want to fulfil
their intent
● Difficult for users to
assess how it went
● May be able to answer
in relative terms
● Users expect to be
active when in an
exploratory mindset
● Effort is expected
How the user
thinks about
results
Just Give Me What I Want: How People Use and Evaluate Music
Search. C Hosey, L Vujović, B St. Thomas, J Garcia-Gathright &
J Thom, CHI 2019.

31. Focused
mindset
Search Mindsets: Understanding Focused and Non-Focused
Information Seeking in Music Search. A Li, J Thom, P Ravichandran,
C Hosey, B St. Thomas & J Garcia-Gathright, WWW 2019.
65% of searches were focused.
When users search with a Focused
Mindset
Put MORE effort in search.
Scroll down and click on lower rank results.
Click MORE on album/track/artist and
LESS on playlist.
MORE likely to save/add but LESS likely
to stream directly.
Understanding mindset helps us understand
search satisfaction.
When users know what they want
to find.
The pull paradigm and how it
translates to the music context.
Findings from large-scale in-app
survey + behavioral analysis.

32. Search by
voice
A type of push paradigm and
how it translates to the music
context.
Findings from qualitative
research.
Users ask for Spotify to play music, without saying
what they would like to hear (open mindset)

33. Search as
recommendation
Delivering for the open mindset.
Conversational search.
Non-specific querying is a way for a user
to effortlessly start a listening session
via voice.
Non-specific querying is a way to remove
the burden of choice when a user is open
to lean-back listening.
User education matters as users will not
engage in a use-case they do not know
about.
Trust and control are central to a positive
experience. Users need to trust the
system enough to try it out.

34. Conclusions
Focused mindset is a typical and common case of pull paradigm.
Understanding the focus mindset can inform measures of search
satisfaction.
Open mindset is important for discovery and lean-back
experiences.
Conversational search (Voice) allows for pull & push paradigms if
done right.

35. Some final words

36. Making AI works at Spotify
Qualitativeresearch
Businessmetrics
Algorithm(s)
Training & Datasets
Metric(s)
Evaluation offline and online
Interaction & feedbacks data
Features
(item)
Features
(user)
Features
(context)

37. Making AI works at Spotify … in this talk
Qualitativeresearch
Businessmetrics
Algorithm(s)
Training & Datasets
Metric(s)
Evaluation offline and online
Interaction & feedbacks data
Features
(item)
Features
(user)
Features
(context)
BaRT
ML-Lab
Rewardfunction
forBaRT
Focused mindset in search
2-side marketplace
Conversational search (Voice)

38. Thank you!