Artificial Intelligence is transforming almost every kind of product as innovative techniques receive deserved attention. But careful leadership from Product Managers is crucial in turning that innovation into something that’s not only valuable but that also respects your own values. This talk provides frameworks to identify where AI can impact our products in the ways we want and to maximize that impact throughout the product life cycle.
Leverage Zilliz Serverless - Up to 50X Saving for Your Vector Storage Cost
Leveraging AI the Right Way (for Product Managers)
1. Leveraging Artificial Intelligence
the Right Way
David Murgatroyd ( @dmurga)
#Unbox2018
October 12, 2018
(please don’t sue me for copyright violation, Flight of the Conchords!)
34. @dmurga
(Disclosure: I’m the user.)
~3K monthly listeners
Popularity rank for this user: 92nd
~3M monthly listeners
Popularity rank for this user: 1,117th
Which result is better?
38. @dmurga
Situations
Query is exact match for artist name
Query is prefix of artist name with 1 letter wrong
Artist is in user’s top-100 by popularity
Artist has >1M monthly listeners
How could a system make this decision?
39. @dmurga
Situations
% of time this was true of
clicked artist
Query is exact match for artist name 30%
Query is prefix of artist name with 1 letter wrong 10%
Artist is in user’s top-100 by popularity 20%
Artist has >1M monthly listeners 35%
How could a system make this decision?
40. @dmurga
Query is exact match for artist
name
30%
Artist is in user’s top-100 by
popularity
20%
Query is prefix of artist name
with 1 letter wrong
10%
Artist has >1M monthly listeners 35%
Which result is better?
41. @dmurga
Query is exact match for artist
name
30%
Artist is in user’s top-100 by
popularity
20%
Query is prefix of artist name
with 1 letter wrong
10%
Artist has >1M monthly listeners 35%
30% * 20% 10% * 35%
Which result is better?
42. @dmurga
Query is exact match for artist
name
30%
Artist is in user’s top-100 by
popularity
20%
Query is prefix of artist name
with 1 letter wrong
10%
Artist has >1M monthly listeners 35%
30% * 20% 10% * 35%
Which result is better?
43. @dmurga
Query is exact match for artist
name
30%
Artist is in user’s top-100 by
popularity
20%
Query is prefix of artist name
with 1 letter wrong
10%
Artist has >1M monthly listeners 35%
Machine Learning is just fancy counting.
Which result is better?
52. @dmurga
1. What are your high-level business metrics?
4 Steps & 3 Skills To Tweak Your Product with ML
53. @dmurga
What are your high-level business metrics?
Increase consumption
(minutes/week)
54. @dmurga
1. What are your high-level business metrics?
2. What fuzzy decisions impact those metrics?
4 Steps & 3 Skills To Tweak Your Product with ML
55. @dmurga
What fuzzy decisions impact those metrics?
It’d be great if we could
predict what
users will click.
Decision: ordering of
search results
57. @dmurga
Skill #1 - Hypothesis Generation
Fuzzy decision hypotheses machine learning enables:
● “We could move our metrics if we could predict …”
○ Which way is this stock price likely to move?
○ Is it going to rain in the next hour?
● “We could move our metrics if we could perceive …”
○ Who is in this picture?
○ What did the user just say?
○ What places are mentioned in this document?
● “We could move our metrics if we could personalize …”
○ Which new legislation is most likely to affect this company?
○ What new songs are most likely to be liked by this user?
58. @dmurga
1. What are your high-level business metrics?
2. What fuzzy decisions impact those metrics?
4 Steps & 3 Skills To Tweak Your Product with ML
59. @dmurga
1. What are your high-level business metrics?
2. What fuzzy decisions impact those metrics?
3. What are the quality metrics for those fuzzy decisions?
4 Steps & 3 Skills To Tweak Your Product with ML
60. @dmurga
What are the quality metrics for those fuzzy decisions?
How often did the user click
something and listen to it?
63. @dmurga
1. What are your high-level business metrics?
2. What fuzzy decisions impact those metrics?
3. What are the quality metrics for those fuzzy decisions?
4 Steps & 3 Skills To Tweak Your Product with ML
64. @dmurga
1. What are your high-level business metrics?
2. What fuzzy decisions impact those metrics?
3. What are the quality metrics for those fuzzy decisions?
4. Is there data to measure those quality metrics?
4 Steps & 3 Skills To Tweak Your Product with ML
66. @dmurga
Is there data to measure those quality metrics?
Billions of searches
with clicks!
67. @dmurga
Skill #3 - Data
User query Time Result clicked?
dmurga pfr 09/23/20183:15PM PFR true
dmurga pfr 09/23/20183:15PM Paramore false
dmurga pfr 09/23/20183:15PM Prince false
Where to get it?
● Logs
68. @dmurga
Skill #3 - Data
User query Time Result clicked?
dmurga pfr 09/23/20183:15PM PFR true
dmurga pfr 09/23/20183:15PM Paramore 0
dmurga pfr 09/23/20183:15PM Prince 0
Persona query Result Relevance?
(1-5)
Folk Dad pfr PFR
Folk Dad pfr Paramore
Folk Dad pfr Prince
Where to get it?
● Logs
● Annotation
69. @dmurga
Skill #3 - Data
User query Time Result clicked?
dmurga pfr 09/23/20183:15PM PFR true
dmurga pfr 09/23/20183:15PM Paramore 0
dmurga pfr 09/23/20183:15PM Prince 0
Persona query Result Relevance?
(1-5)
Folk Dad pfr PFR 3
Folk Dad pfr Paramore 2
Folk Dad pfr Prince 2
Where to get it?
● Logs
● Annotation
70. @dmurga
Skill #3 - Data
User query Time Result clicked?
dmurga pfr 09/23/20183:15PM PFR 1
dmurga pfr 09/23/20183:15PM Paramore 0
dmurga pfr 09/23/20183:15PM Prince 0
Persona query Result Relevance?
(1-5)
Folk Dad pfr PFR 3
Folk Dad pfr Paramore 2
Folk Dad pfr Prince 2
Where to get it?
● Logs
● Annotation
● 3rd Parties
71. @dmurga
1. What are your high-level business metrics?
2. What fuzzy decisions impact those metrics?
3. What are the quality metrics for those fuzzy decisions?
4. Is there data to measure those quality metrics?
4 Steps & 3 Skills To Tweak Your Product with ML
86. @dmurga
ML’s output is
often wrong,
unintutive,
off-brand, and
varying by user.
ML can elicit
unexpected behavior
or unintended
incentives in users.
Skill #5 - Design & ML
88. @dmurga
Skill #6 - Unfairness & ML
Unfair
Characteristics
Unfair
Focus
(See pubs by my colleagues Henriette Cramer, Jean Garcia-Gathright, and Sravana Reddy for more!)
89. @dmurga
Skill #6 - Unfairness & ML
(See pubs by my colleagues Henriette Cramer, Jean Garcia-Gathright, and Sravana Reddy for more!)
91. @dmurga
Metric All Dark
F
Dark
M
Light
F
Light
M
Error % 6.3 20.8 6.0 1.7 0.0
Skill #6 - Unfairness & ML
(See pubs by my colleagues Henriette Cramer, Jean Garcia-Gathright, and Sravana Reddy for more!)
Unfair
Characteristics
Unfair
Focus
106. @dmurga
Skill #10 - Science & ML
Don’t believe either end of
the hype spectrum.
Industry giants with large resources
and product portfolios do a lot of
basic research.
WE’RE GONERS
110. @dmurga
1. Hypothesis Generation
2. Metrics
3. Data
4. Product Discovery
5. Design
6. Fairness
7. Prioritization
8. Team Structure
9. Performance Tradeoffs
10. Research
11. Strategy
11 Product Management Skills that ML Changes
%
%
111. @dmurga
1. Hypothesis Generation This Talk from Andrew Ng
2. Metrics Tutorial from my colleague
3. Data Wikipedia Article on Data Splits
4. Product Discovery Article on Kinds of ML Tasks
5. Design Book: Designing Agentive Tech
6. Fairness Paper from my colleagues
7. Prioritization A talk of mine that goes deeper
8. Team Structure A different talk of mine that goes deeper
9. Performance Tradeoffs Article: Striking a balance
10. Research TWIML Podcast
11. Strategy Book: The Inevitable
11 Product Management Skills that ML Changes - Resources
%
%
112. Thanks! Questions?
David Murgatroyd (@dmurga)
Suggestions:
Say more about how ML affects the product life cycle?
What roles do teams building ML products need?
How do you set OKRs for ML efforts?
What is this Deep Learning thing I keep hearing about?
Didn’t y’all just win MassTLC’s ML in Action Award? Hiring in Boston,
NYC, London,
and Stockholm!