This document discusses product management for AI/ML, highlighting the importance of understanding AI concepts, strategies, and types of learning. It emphasizes the role of human purpose in AI design, the iterative nature of building AI systems, and the need for trust and feedback from users. Resources for further learning and the significance of using design thinking and agile methodologies in AI development are also provided.

1. Product Management
for AI/ML
The Product Mentor, Season 7
Resources available:
https://goo.gl/TfUxac

2. Chris Butler
Director of Prod Strat @
Philosophie NYC
The Best Product Person 2016
17 years of product and BD
Microsoft, Waze, Horizon
Ventures, KAYAK, and started
my own company (failed)
chrisbutler@philosophie.is
@chrizbot

4. Product management for AI/ML
● What do I need to know about these things?
● How do they impact product’s role
○ Purpose and strategy
○ Learning
○ Building
○ Prioritizing
○ Measuring
○ Technical

5. What is...artificial intelligence?
...artificial general intelligence?
...narrow artificial intelligence?
...a neural network?
...machine learning?
...deep learning?
…?

7. Where to start?

8. Learns from
from (good) data

9. Attempts to reduce an error
against desired outcomes

11. Why are AI programs different?
● Content: models, not programs
● Process: training, not debugging
● Release: retraining, not patching
● Uncertainty: of objective
● Uncertainty: of action and recommendation
● Uncertainty: propagates through model

12. Training and inference

13. Types of problems it can solve (possibly)
● Ranking - Google search results
● Recommendation - Netflix movie recommendations
● Regression (or prediction) - Zillow predicting house prices
● Classification - Image is a cat or dog
● Clustering - Tumblr social network analysis to find groups
of topics

14. ● Supervised
● Unsupervised
● Supervised
● Unsupervised
● Reinforcement
● Semi-supervised
● One shot
● Few shot
Types of learning
● Transfer
● Active
● Imitation
● Q
● Transduction
● ...

15. Resources to start learning
Books
● Programming Collective Intelligence by Toby Segaran
● The Master Algorithm by Pedro Domingos
● Introduction to Machine Learning by Nils Nilsson
● Data Mining by Ian Whitten
● Data Science for Business by Foster Provost
● Neural Networks and Deep Learning by Michael
Nielsen
● Make Your Own Neural Network by Tariq Rashid
Courses
● Introduction to Machine Learning by Andrew Ng
(highly recommended)
● Machine Learning Engineer by Udacity
● Machine learning is Fun! by Adam Geitgey
● How to use Tensorflow for Classification by Siraj
Raval
● Learning AI if you suck at Maths by Daniel Jeffries
Must-Reads:
● WTF is Artificial Intelligence by Sam DeBrule
● Machine learning for Product Managers by Ken
Norton
● AI, Deep Learning, and Machine Learning: A Primer
by Frank Chen
● Artificial Intelligence is the new electricity by Andrew
Ng
● The current state of Machine Learning by Shivon Zilis
● How Google is remaking itself a ‘Machine Learning
First’ company by Steven Levy
● An executives guide to machine learning by Dorian
Pyle (Mckinsey)
● Experience Design in the Machine Learning Era by
Fabien Girardin
● A human’s guide to Machine learning by Sam DeBrule
(subscribe to his newsletter)
● What every manager should know about Machine
Learning by Mike Yeomans
https://hackernoon.com/machine-learning-and-product-managers-930b691b1b37

16. Fast.ai MOOC is very hands on

17. Unless you are in research, the real
focus should be on what differentiates
your product and gives it meaning. Not
finding a better way to detect the
difference between cats and dogs in
ImageNet images.
https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d

18. Purpose and strategy

24. Without human purpose, a computer is
just a rock that we tricked into thinking.
https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d

27. Design Thinking, Lean, and Agile?

28. Design Thinking
(AKA Human-Centered Design)

30. Human at the center being assisted and
augmented by AI/ML

31. Lean

33. Start small with simple models that build
confidence in what you are doing

34. Agile

36. Continuously improve the data, the
model, training, etc.

37. Design Thinking, Lean, Agile, and AI
are about emergent practice

38. Design Thinking, Lean, Agile, and AI
are about learning and adapting

39. How to learn about purpose

40. When using AI, you want to know:
● Are we helping solve a problem?
● Do they trust the information?
● Do they feel comfortable giving feedback to the system?

41. Rich Picture

42. Empathy Mapping (for the machine)

43. Confusion matrix

44. Decision boundaries

45. Patterns for building with purpose

46. Self driving cars classifications

47. Does everything Learning Watching
Approving Confident Recommending
Veto’ing Proven Taking action
Human System State Machine Action

48. Intelligent CTA

49. “Calculating” and explainability

50. Be up front about possible errors
and sometimes wrong

51. Feedback

52. Wizard of Oz to learn

53. Bootstrapping

54. How to prioritize with purpose

55. Prioritization

56. Outcome Mapping

57. How to measure with purpose

58. Research questions
● Think back to the last time you did this, how did you come to
that decision?
● Do you trust these suggestions for what to do next?
● How do you think the system decided [action]?
● Was there enough information for you to [take action]?
● How much do you trust the system to make the right decision in
the future? It is more or less than before?

59. Gathering feedback from people

60. Gathering feedback from people

61. Back to the confusion matrix!

62. Technical concerns

63. Simulation and QA

64. Scaling

65. Technical debt
***

66. In closing

67. In closing
● We give machines their purpose - focus on problems you
are solving, not new toys
● Building these systems are a journey - iterate and learn
● We deal with nondeterministic systems all day in our
teams, industries, and markets - AI is no different

68. Don’t get stuck with a rock that doesn’t
help you meet your purpose

69. Thank you
For more information:
https://goo.gl/TfUxac

72. Appendix: removed slides

73. Background for internal review
● Audience: product people (new and experienced)
● When: 9/17/17
● Alternate use: planning on using parts for Design Thinking for AI workshops
● Feedback needed:
○ Good enough overview of AI? Design Thinking/Lean?
○ Does it feel like a good journey/order?
○ Anything unnecessary? Missing?
○ Did you learn something?

74. Definition
“A computer program is said to
learn from experience E with
respect to some class of tasks T
and performance measure P if its
performance at tasks in T, as
measured by P, improves with
experience E.”
-Tom Mitchell, 1997

78. Design Thinking and Lean,
not either/or

79. Diverge and converge

80. The “spiral”

81. Stanford d.school Design Thinking process
Learn
Build
Measure
Iterate
Lean

84. Empathy Mapping (for the machine)

85. Iterate

86. Perfect is the enemy of good…
...and not possible with AI