This document describes courses offered by Product School to help product managers gain skills in areas like product management, coding, data analytics, digital marketing, UX design, and product leadership. It also provides an overview of a talk on applying machine learning given by a Lyft senior product manager. The talk explains what machine learning is, the different types of machine learning problems, and how product managers can identify opportunities, define problems, and guide machine learning solutions and teams. Examples are provided around replacing cash bail and automating food delivery order disputes.

1. www.productschool.com
How to Apply Machine Learning by
Lyft Senior Product Manager

2. Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3. C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job

4. C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers

5. C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts

6. C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers

7. C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end experience

8. C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Leadership

9. C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills

10. Jimmy Young
T O N I G H T ’ S S P E A K E R

11. hello!
I am Jimmy Young
You can reach me at jimmy@jimyou.com
11

12. 1.
About me
How did I get here?
12

13. 2.
About This Talk
What are we covering?
13

14. Why?
1. Almost all mid-stage companies, and some early stage can utilize
Machine learning to great effect.
2. Buzz term, a lot of great content, not much focused to PMs
Useful when even <10% better means $$$$$
14

15. Who this is/isn’t for
✘ People who want to create their own ML system themselves
1. You are/want to be a product manager.
2. You will work with research/data scientists and engineers.
My goal is to get you excited to learn more.
15

16. 3.
Introduction
What is ML?
16

17. What is ML?
“algorithms and statistical models that computer systems use to
effectively perform a specific task without using explicit instructions,
relying on patterns and inference instead”
-Wikipedia
17

18. Classification
Give a label to something.
“Is this a hot dog or not a hot dog?”
“Is this user fraudulent?”
“Will it rain tomorrow?”
18

19. 19

20. Forecasting/estimation
Forecast a number in a sequence
Predict a given quantity
“What will sales be next month?”
“How many refunds will be sent?”
20

21. 21

22. Optimization
Generate something given
constraints
“What is the shortest route?” (GPS)
“What is the route with the lowest
cost” (UPS)
“Which passengers should I match
together?” (Lyft)
22

23. 23

24. Ranking/Recommendations
Sort a list of items by some metric
“Which three items is this user most
likely to buy?” (Recommendations)
“Sort all web pages by most
relevant” (Search engine)
24

25. 25

26. 4.
Anatomy of ML
26

27. 27
Model
E.g. Types of models:
● Neural Nets
● Decision Tree
● Gradient boosting
● Linear programming
● Naive Bayes
● K-means
Inputs
Aka
“Features”
Output
E.g.
● Score
● Prediction
● Classification
● Result

28. 5.
What do PMs do?
28
Here’s where you add value

29. Product Managers do 3 things
Identify
Opportunity
Define
the problem
and target
Guide
The solution
& the team
29

30. 1. Identify
Identify the opportunity where ML can have business impact.
● Areas where a small improvement has large impact
○ (generally steps/features closest to conversion/price/cost)
● Problems that are easily solveable with simple ML
● Areas that have a work-around or temporary solution
30

31. “People worry that computers will
get too smart and take over the
world, but the real problem is
that they're too stupid and
they've already taken over the
world. - Pedro Domingos
31

32. Clue #1: Look for stupid but essential
If you see a part of business process/logic or
code that has a large list of if/else statements
or rule-based logic (heuristics).
32

33. Problem:
Suppose you’re a PM at a forum
Should I delete this comment due to spam?
e.g.
● If the user is new AND
● Comment contains A URL AND
● URL is not top 100 url list AND
● another user already marked it as spam
33

34. Clue #2: Probably not a problem that is
completely new
Generally a poor idea for v1 to be ML-based
Launch a heuristic first.
E.g. Youtube -> “Other videos by this artist”
Twitch -> “recently watched channels”
34

35. 2. Define
Define the problem & optimization criteria for your team.
Product managers have the most context of the product strategy, and
the broader business. Help define the problem for scientists and
engineers.
35

36. Clue: Set the right target
E.g. Youtube recommendations:
● Optimize for?
○ What you’re most likely to click next?
■ Skew towards better cover art?
○ What you’re most likely to click and watch for 5 mins?
■ Might not watch for a long time?
○ What gives the longest minutes-watched?
36

37. 3. Guide
Brainstorm features (inputs) and measure the outputs.
E.g. Youtube recommendations:
● My subscribed channels
● Other channels people watched
● What i’ve watched recently
● Language
● Referrer source (i.e. reddit? organic?)
● topic
37

38. 38
Model
E.g. Types of models:
● Neural Nets
● Decision Tree
● Gradient boosting
● Linear programming
● Naive Bayes
● K-means
Inputs
Aka
“Features”
Output
E.g.
● Score
● Prediction
● Classification
● Result

39. PM should worry about the inputs and outputs
only. Assume everything in the middle is a
blackbox.
39

40. 1. “Accuracy” itself is a poor metric
I have a piece of software that can predict the
whether it will rain in SF with 82% accuracy
40

41. 82% accurate piece of software
Will it rain?
Answer: No, it won’t rain.
Use a simple rule (like this) as a baseline for your model.
41

42. 2. Model evaluation metrics
Classification: Precision & Recall, ROC
● True positive, true negative, false positive, false negative
Forecasting: MAE & MSE
42

43. What do PMs do - concretely
1. Outline the opportunity
2. Define the optimization/success metric
3. Brainstorm features (inputs)
4. Define how the output is used Bonus: Bad technical results need great product work-arounds
5. Define some good baselines
43

44. 6.
Examples
44

45. Replacing cash bail
Courts in california have
defendents pay cash bail. The
money is returned if the defendent.
If you don’t post bail, you wait in jail.
45

46. Product Managers do 3 things
Identify
Opportunity
Define
the problem
and target
Guide
The solution
& the team
46

47. 1. Identify
Identify the opportunity where ML can have impact.
47

48. 2. Define
Define the problem & optimization criteria for your team.
48

49. 3. Guide
Brainstorm features (inputs) to the models/algorithm
49

50. Replacing cash bail
Courts in california have
defendents pay cash bail. The
money is returned if the defendent.
If you don’t post bail, you wait in jail.
50

51. Automating disputes
Imagine you’re ordering food
delivery.
In a significant amount of cases,
users are complaining items are
missing.
Today: Support agents discretion
Goal: Reduce support agent cost
51

52. 1. Identify
Identify the opportunity where ML can have impact.
52

53. 2. Define
Define the problem & optimization criteria for your team.
53

54. 3. Guide
Brainstorm features (inputs) to the models/algorithm
54

55. 7.
Closing Notes
55

56. Technical difficulty
56
Easy Hard
High amounts of data
(bytes) & high
accuracy bar
Low amounts of data
(bytes), and low
accuracy bar

57. PM’s help reduce the accuracy bar
57

58. Generally Hard - cutting edge research
58

59. thanks!
Any questions?
You can find me at
jimmy@jimyou.com
59

60. www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design, Product Leadership courses and
Corporate Training