Vito Ostuni - The Voice: New Challenges in a Zero UI World The adoption of voice-enabled devices has seen an explosive growth in the last few years and music consumption is among the most popular use cases. Music personalization and recommendation plays a major role at Pandora in providing a daily delightful listening experience for millions of users. In turn, providing the same perfectly tailored listening experience through these novel voice interfaces brings new interesting challenges and exciting opportunities. In this talk we will describe how we apply personalization and recommendation techniques in three common voice scenarios which can be defined in terms of request types: known-item, thematic, and broad open-ended. We will describe how we use deep learning slot filling techniques and query classification to interpret the user intent and identify the main concepts in the query. We will also present the differences and challenges regarding evaluation of voice powered recommendation systems. Since pure voice interfaces do not contain visual UI elements, relevance labels need to be inferred through implicit actions such as play time, query reformulations or other types of session level information. Another difference is that while the typical recommendation task corresponds to recommending a ranked list of items, a voice play request translates into a single item play action. Thus, some considerations about closed feedback loops need to be made. In summary, improving the quality of voice interactions in music services is a relatively new challenge and many exciting opportunities for breakthroughs still remain. There are many new aspects of recommendation system interfaces to address to bring a delightful and effortless experience for voice users. We will share a few open challenges to solve for the future.

1. ‘The Voice’: New Challenges
in a Zero UI world
Vito Ostuni
Staff Scientist, Search & Voice
MLconf SF 2019

2. 2
Pandora
○ Music and Podcasts
○ Exclusive Content
○ Free on-demand
○ Personalized Radio
Stations
○ Curated and Personalized
Playlists

3. 3
“Hey Pandora“
Voice Mode

4. 4
“Play Party Music“
“Now playing your
Party Soundtrack“
“Hey Pandora“
Voice Mode

5. 5
Search and Recommendation
Search
Recommendation

6. 6
Search and Recommendation
Search
Recommendation
“Play Drake“ Known-item queries

7. 7
Search and Recommendation
Search
Recommendation
“Play Drake“
“Pull up some workout music“
Known-item queries
Thematic queries

8. 8
Search and Recommendation
Search
Recommendation
“Play Drake“
“Pull up some workout music“
“Just play something awesome“
Known-item queries
Thematic queries
Open-ended queries

9. 9
Known-Item Queries
● Clear user intent
○ “Play Drake”
● Challenges
○ ASR errors
○ Partial/Imperfect user queries
○ Ambiguities
“Play In My
Feelings“
“Play the song that goes
I got horses on my back“
“Play latest
Ed Sheeran
single“
Known-item Queries

10. 10
Search Challenges
Señorita - Shawn Mendes & Camila Cabello
ASR Errors

11. 11
Señorita - Shawn Mendes & Camila Cabello
“Play send your Rita”
“Play sang Eureka”
“Play send your data”
Search Challenges
ASR Errors

12. 12
Search Challenges
“blue da boo dee da boo die“
Lyrics

13. 13
Search Challenges
Imperfect ASR and homophonous music entities
“Play John Maus”
ASR
“play john mouse”

14. 14
Search Challenges
Imperfect ASR and homophonous music entities
John Maus
John Mouse“Play John Maus”
ASR
Voice
Search
“play john mouse”

15. 15
Search Challenges
Imperfect ASR and homophonous music entities
John Maus
John Mouse“Play John Maus”
ASR
Voice
Search
“play john mouse”

16. 16
Personalized Entity Search
Robust
Retrieval
○ Phonetic and fuzzy matching
○ Query reformulation
○ Aliases
■ “Dead mouse” → “Deadmau5”

17. 17
Personalized Entity Search
Robust
Retrieval
○ Phonetic and fuzzy matching
○ Query reformulation
○ Aliases
■ “Dead mouse” → “Deadmau5”
Personalized
Ranking
○ User–item historical interactions (exploit)
○ User–item affinity (generalize)

18. 18
UX Considerations
● Design UX to elicit user feedback
● Collect feedback for low confidence results
“Play John Maus”
ASR Voice
Search
“play john mouse” confirm(“john maus”)
“Do you want me to
play your John Maus
radio?”
“Yes”
yes

19. 19
Thematic Queries
● Semi-ambiguous intent
○ “Play me road trip music”
“relaxing drive” playlist “road trip” movie soundtrack
“Play some music
to party“
“I am looking for
something happy“
“I want some music
to help me sleep“

20. 20
Thematic Recommendations
Query
Tagging
○ Intent prediction
○ Slots filling

21. 21
Thematic Recommendations
Retrieval
Query
Tagging
○ Intent prediction
○ Slots filling
○ Content tagging
○ Shared taxonomy
○ Tag based search

22. 22
Thematic Recommendations
Retrieval
Personalized
Ranking
○ User–Item affinity
○ Query–Result relevance
Query
Tagging
○ Intent prediction
○ Slots filling
○ Content tagging
○ Shared taxonomy
○ Tag based search

23. 23
Thematic Recommendations
Retrieval
Personalized
Ranking
○ User–Item affinity
○ Query–Result relevance
Query
Tagging
State
Tracker
○ Intent prediction
○ Slots filling
○ Content tagging
○ Shared taxonomy
○ Tag based search

24. 24
Natural Language Understanding
NLU
Intent detection
Slot filling
Voice2text
Intent: play music
Slots: character: romantic
context: dinner
“play some romantic dinner music”

25. 25
NLU Architecture
play classics by aretha franklinInput query
CRF
Classifier
Bidirectional
LSTM
[Hakkani-Tur, INTERSPEECH 2016]
[Reimers & Gurevych, EMNPL 2017]
O B-obj O B-art I-art
Word features
word embeddings, CNN char features

26. 26
Broad Queries
● Open-ended intent
○ “Play something awesome”
● Pure recommendation problem
○ No query
“Play something new“
“Play music I like“
“Play music“

27. 27
Bandit Solution
Station
Recs
User Playlists
New Music
Recs
….
Multi-Armed
Bandit
Station
Recs
play time

28. 28
Other Considerations
● Different evaluation tasks
○ Entity search
○ Thematic recommendations
○ General recommendations
● Implicit feedback
○ Play time (no click data)
○ Query retrials and session analysis
● Selection bias
○ Only one recommended item
○ Exploration policies

29. 29
A Great Team!

30. Any
questions?
Sorry, I can’t
find what
you are
looking for,
please try
again.
“Questions?“