With more products being launched today that utilize artificial intelligence (AI), product managers and decision makers need to understand how developing AI-driven products differs from developing traditional software. Michael Feng, a founder and product manager who has been building AI-driven products for the past 5 years, shares lessons learned and an framework for AI product management, illustrated by real-world examples. …

1. BUILDING AI-DRIVEN
PRODUCTS
MICHAEL FENG

2. ABOUT ME
2013: Table extraction from PDF files

3. ABOUT ME
2014-2015: Document analytics

4. ABOUT ME
2015-2017: Smart documents

5. BUILDING AI-DRIVEN PRODUCTS
AGENDA
▸ Someone needs to wear the PM hat
▸ Traditional PM process isn’t a good fit
▸ 4 habits of effective AI product managers

6. SOMEONE NEEDS TO
WEAR THE PM HAT

7. SOMEONE NEEDS TO WEAR THE PM HAT
ACADEMIA VS REAL WORLD
Characteristics
▸ Explicit problem
statements
▸ Well-structured
datasets
▸ Defined success
criteria

8. SOMEONE NEEDS TO WEAR THE PM HAT
ACADEMIA VS REAL WORLD
Goals
▸ Publish research paper
▸ Improve on established
benchmark vs past
methods ImageNet Classification with Deep
Convolutional Neural Networks (2012)

9. SOMEONE NEEDS TO WEAR THE PM HAT
ACADEMIA VS REAL WORLD
“I’d like you to explore our
server logs and use machine
learning to provide insights
to our customers.”
“That’d be grrreat.”

10. SOMEONE NEEDS TO WEAR THE PM HAT
AI PRODUCT MANAGERS OWN THE QUESTIONS
‣ What problem are we solving?
‣ Why is that problem valuable?
‣ Do we have relevant data that may solve this problem?
‣ How do we measure success?
‣ Are we building a system that learns?

11. I DEAL WITH THE GODDAMN CUSTOMERS
SO THAT THE ENGINEERS DON'T HAVE TO.
I HAVE PEOPLE SKILLS!
Product Manager
SOMEONE NEEDS TO WEAR THE PM HAT

12. YOU’RE WEARING
THE PM HAT
IF THERE’S NO PM
Founder
Data scientist
Manager

13. TRADITIONAL PM PROCESS
ISN’T A GOOD FIT

14. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
PROBLEM —> SOLUTION
1. Gather user stories
2. Define requirements
3. Design mockups
4. Perform acceptance
testing

15. EXAMPLE: NETFLIX FOR DOCUMENTS

16. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
PROBLEM: USERS DON’T USE THE PRODUCT!

17. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
SOLUTION: NEW USER TOUR

18. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
USER STORY
“As a new user, I want to know
what the top 3 features are so that
I can start using them”
MOCKUPS
Did all requirements pass?ACCEPTANCE TESTING
REQUIREMENTS
‣ User see a pop-up with a carousel of top 3 features,
and the background is opaque
‣ User can exit anytime by clicking on background
‣ Works on web, iOS, Android

19. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
PROBLEM: USERS STILL DON’T USE THE PRODUCT!

20. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
SOLUTION: IMPROVE RELEVANCE OF
CONTENT RECOMMENDATIONS

21. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
USER STORY
MOCKUPS
“As a user, I want to see content
shared by people I know”
Add a rule to the algorithm
ACCEPTANCE TESTING
REQUIREMENTS
Show documents shared by people in
same office
Were users are more likely to click
on results shown?
USER STORY #1

22. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
USER STORY
MOCKUPS
“As a user, I want to see content
related to my work”
Add another rule to the algorithm
ACCEPTANCE TESTING
REQUIREMENTS
Show documents shared by people in
same functional department
Were users are more likely to click
on results shown?
USER STORY #2

23. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
USER STORY
MOCKUPS
“As a user, I want to see content
about company social events”
Add yet another rule to the algorithm
ACCEPTANCE TESTING
REQUIREMENTS
Show documents shared by people on
the social committee
Were users are more likely to click
on results shown?
USER STORY #3

24. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
if (A)
return w1 * f1 + w2 * f2 + … + wN * fN
else if (B)…
A BUNCH OF HEURISTICS

25. TRADITIONAL PM PROCESS ISN’T A GOOD FIT
A set of heuristics isn’t AI because it’s not
a system that learns (i.e improve with
more data)

26. 4 HABITS OF EFFECTIVE
AI PRODUCT MANAGERS

27. UNDERSTAND
THE MODEL
1

28. UNDERSTAND THE MODEL
BASIC EXAMPLE: COLLABORATIVE FILTERING
▸ Users “rate”
documents
▸ Sparse ratings matrix
▸ Latent user and
document factors
▸ Predict unknown
ratings

29. UNDERSTAND THE MODEL
Classes
‣ 2 = Strong interest
‣ 1 = Lukewarm interest
‣ 0 = No interest
Blank = Not rated
Doc 1 Doc 2 Doc 3
User 1 2 1
User 2 1 2
User 3 1 0 ?
RATINGS MATRIX

30. UNDERSTAND THE MODEL
User vector
1.5
2.0
2.0
-1.0
Document vector
0.5
-1.5
1.0
-2.0
1.75
LATENT FACTORS
Predicted
rating

31. UNDERSTAND THE MODEL
Doc 1 Doc 2 Doc 3
User 1 2 1
User 2 1 2
User 3 1 0
FITTING THE MODEL
Training
Doc 1 Doc 2 Doc 3
User 1 1.95 0.53 0.91
User 2 1.08 1.35 2.62
User 3 1.33 0.03 1.75
Prediction
Adjust latent factors to minimize
prediction error vs training data

32. UNDERSTAND THE MODEL
MAKING THE PREDICTION
Show the user documents:
‣ with the highest predicted
ratings
‣ not owned by him/her

33. GET LABELED
DATA
2

34. Labeled
Unlabeled data is the new dirt

35. GET LABELED DATA
Labeled data has explicit labels of
exactly what you’re trying to predict

36. GET LABELED DATA
Not labeled data
Server logs
Satellite images
Files in S3
LABELED VS UNLABELED DATA
Labeled data
+ which requests came
from malicious IPs
(hack detector)
+ markers of pools
(pool detector)
+ marked confidential
or not
(confidentiality classifier)

37. GET LABELED DATA
HOW TO GET LABELED DATA
Outsourced labeling
Brute force
HOW LONG DOES IT
TAKE TO GET 10K
SAMPLES?
User feedback
Transfer learning

38. CONSTRUCT A
FEEDBACK LOOP
3

39. CONSTRUCT A FEEDBACK LOOP
Display predictions
Collect user
feedback
Add to training set
Re-fit model

40. CONSTRUCT A FEEDBACK LOOP
For each document shown, assign rating:
‣ 1 if user opens the document once
‣ 2 if the user opens the document again
‣ 0 if the document has been shown 3+
times and not opened
SAME DIMENSIONS AS YOUR TRAINING DATA

41. CONSTRUCT A FEEDBACK LOOP
Doc 1 Doc 2 Doc 3
User 1 2 1
User 2 1 2
User 3 1 0 1
SAME DIMENSIONS AS YOUR TRAINING DATA

42. CONSTRUCT A FEEDBACK LOOP
Having the feedback loop on day 1
means you’re not wasting valuable
new labeled data

43. MEASURE RELATIVE
PERFORMANCE
4

44. MEASURE RELATIVE PERFORMANCE
THE FIRST VERSION WILL PROBABLY SUCK

45. MEASURE RELATIVE PERFORMANCE
Relative performance tells you how
much more data you need to get to
“good enough”

46. MEASURE RELATIVE PERFORMANCE
Accuracy
Amount of
data
“good enough”
How much data
you need to get
to “good enough”
slope
=
relative
perform
ance

47. PUTTING IT ALL TOGETHER

48. PUTTING IT ALL TOGETHER
LABELED DATA
MODEL
Predictions
FEEDBACKLOOP
RELATIVE
PERFORMANCE
1. Understand the
model
2. Get labeled data
3. Construct a
feedback loop
4. Measure relative
performance

49. Thanks!
Michael Feng
mifeng@gmail.com