Main takeaways: - Key stages in the Data Science process - Unique challenges ML products present - Opportunities for Product Managers to make a big impact

1. www.productschool.com
Delivering Machine Learning
Solutions by fmr Sears Dir of PM

2. Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3. C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job

4. C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers

5. C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts

6. C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers

7. C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end experience

8. C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Leadership

9. C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills

10. Kuleen Mehta
T O N I G H T ’ S S P E A K E R

11. Building Successful ML
Products
Kuleen Mehta

12. Overview
● ML Opportunity & Challenges
● ML Development Process
● Key Contributions of PM
○ Case Study

13. What is Machine Learning ?
Machine Learning is the field of study that gives Computers the
ability to learn without being explicitly programmed.
- Arthur Samuelson (circa. 1959) Artificial Intelligence
Machine Learning
Deep Learning
Machine
Learning
Classical
Programming
Answers
Rules
Data
Data
Answers
Rules

14. The Fourth Industrial Revolution
$ 35.8 Best. spend on AI in 2019 (IDC)
8 out of 10AI projects are stalled and 96% report running into data
issues (Dimensional Research Survey)

15. Key Considerations for applying ML
User Needs
UX solutions
Heuristics
MLaaS options
User Experience
Expected Accuracy
Impact of Errors
Understandability
Data Availability
Sourcing (labelled) data
Representative &
Unbiased
Privacy Policy
Iterate and
Improve
Close the Loop
Fail Forward
ML when: Lots of data, Dynamic patterns, Personalized Experience at scale…

16. The ML Product Process
CRISP-DM workflow courtesy: Kenneth Jensen

17. Saving unnecessary ER trips - Hypothetical
Health Care System Goals
● Fast resolution of (real) Emergencies
○ Better patient outcomes
○ Higher patient satisfaction (wait times..)
● Save Money
○ Lower staffing Capacity
○ Reduce Overtime pay
○ Improved staff morale and retention
Patient Goals
● Fast Resolution
○ No long ER waits
○ Get DIY remedy if possible
● Save Money
○ ER $ > Doctor’s office $ > OTC Medicine $

18. Business Understanding
• Understand the problem
• Define the Solution
• Define Success criteria (MTMM)
• Define the User Experience
• MVP vs. v1
• Map to Model Efficacy goals (PR, AUC...)
• Define Constraints – e.g.
Interpretability, Understandability
Define
Target
Customer
Underserved
Needs
Value
Proposition
Feature Set
UX
Test with the
customers
Courtesy: Dan Olson
Product-Market Fit

19. Define the Solution - FBM Chatbot
ScheduleTriage Consult an Expert ($)
Natural Language dialog with AI bot
Natural Language Search
through Knowledge base
Select from “common”
symptoms and remedies
Live audio/video chat
Offline diagnosis through
pictures / video
Text chat with an expert
Find a Doctor / Specialist
Find nearest ER / Request Ambulance
Schedule Appointment

20. Data Acquisition
•Internal
•External
•None - Cold Start (Build UX for collection, Unsupervised techniques)
Sources
•Recruit Users (Crowdsource)
•Internal Content Experts
•Paid(Mechanical turk, CrowdFlower etc).
Labels
•Standards: Enterprise vs. Consumer, By Industry (HIPAA)
•Encryption, Audit trails, GDPR Req.
•Privacy conscious ML techniques - FL, DP
Privacy and
Governance
•Instrument your UI (1st and 3rd party Analytics)
•Ask the user (In-situ, surveys , NPS scores)Feedback Loop

21. Data Understanding
Collaborate

22. Data Preparation, Modeling & Evaluation
● Be Curious, not prescriptive
● Create Checkpoints, measure progress
● Communicate and Manage Stakeholders
● Know when to pivot
Evaluation
Precision-Recall /
AUC Targets
Modeling
Model Architecture
Hyper Param
tuning,Reg.
Data Preparation
Feature selection,
Cleanup, Feat.
Engineering
Communicate

23. Deployment
• ML Pipelines are expensive to build and maintain - set stakeholder expectations
• Sensitivity Analysis on Models
• Design for the long term, build for your MVP
• Prioritize your Feedback loop Deliver

24. Test with Customers & Iterate
ScheduleTriage Consult an Expert ($)
NL Dialog Agent
NL Search through Knowledge
base
Select from “common”
symptoms-remedies
Live audio / video chat
Offline diagnosis through
pictures / video
Text chat with an expert
Find nearest ER / Request Ambulance
20% Resolve
Personalize
56% Abandon
Find a Doctor / Specialist
Schedule Appointment 65% Compltn.
90% TP rate

25. The ML Process
CRISP-DM workflow courtesy: Kenneth Jensen
Define
Acquire
Collaborate
Communicate
Iterate
Deliver
PM Contributions
Test & Iterate

26. Questions ?
Linkedin: https://linkedin.com/kuleenmehta
Medium: https://medium.com/@kuleenm

27. Deep vs. Shallow Learning
Courtesy: Andrew Ng

28. www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design, Product Leadership courses and
Corporate Training