- Model building is only 10% of an ML Product, the rest is good Product Management. Focus on building good tracking and testing. Models will get better.
- ML Product Roadmaps look very different. It may take 2-3 years to get true gain (test vs actual accuracy). Keep managing stakeholder misinformation. Educating is part of the job.
- ML is more than a feature in your app. Relook at the whole canvas. You'll need expertise beyond data; Re-think tech design, UX and most importantly Business strategy.
8. Tools you will need
Common Sense
(abstraction of reality)
+
Intuition
(experience)
9. What is Machine Learning good for
Essentially pattern recognition.
Especially good when patterns are too complex or too fuzzy.
Things beyond rules-sets and basic statistics.
10. What is Machine Learning good for
Classifying
Grouping
Predicting
Responding
18. What is Machine Learning good for
Classifying Grouping
Predicting Responding
2, 4, 6, __ X
O
X
O X
O
19. Rate patients on cognitive impairment and dementia
and as a measure of spatial dysfunction and neglect.
20. Needles in a haystack of supply, categorized to 1 of 13,000
possible product categories. 19 of which are other needles.
21. The best time-range, sequence, care-type and provider
combinations to help a person recovery from their specific surgery.
22. Read stacks of published literature to compute a meta-analysis of
the benefits of switching to a new surgical procedure.
23. Personalize health actions to each user based on their health
conditions, benefits and interests so the most interesting and
useful actions come first.
24. Tools you will need
Common Sense
(abstraction of reality)
+
Intuition
(experience)
30. Product
ML
Ensembler
(x models)
Rule-based
Engine
UXData
Reports
Plan for multiple models, releases, version, ensembling etc.
Intuition +1
Roadmap
Rules engine
Model Mgmt
Complexity for Rules vs. Model Mgmt
system depends on what your
problem space has more of
Team
● PM
● Architect
● Engineers
● ML Architect
Problem Space
Problem worth solving Solution worth a value
31. A kid learns to read Fonts through exposure
Data > Algos. Getting data right is a life saver.
Common Sense +1
32. Product
Rule-based
Engine
UXData
Reports
Most Data is about the common cases
Intuition +1
Roadmap
Rules engine
Model Mgmt Data Features Models
ML
Ensembler
(x models)
Team
● PM
● Architect
● Engineers
● ML Architect
● Data Scientists
Problem Space
Problem worth solving Solution worth a value
33. ML UX vs. Traditional UX
ML driven interfaces have huge opportunities of UX Improvement. Spend the time here.
Common Sense +1
34. Product
Rule-based
Engine
ML UXData
Reports
This is a multiplier to all your metrics. Get that low-hanging fruit!
Intuition +1
Roadmap
Rules engine
Model Mgmt Data Features Models
ML
Ensembler
(x models)
Team
● PM
● Architect
● Engineers
● ML Architect
● Data Scientists
● UX Designers
UX A/B + Actuals
Problem Space
Problem worth solving Solution worth a value
35. Lab vs. Real Life
What works in the lab will likely drop in real-life. Real life is a lot more chaotic and your data and features can’t map it all.
Intuition +1
36. Product
A/B Testing + Actual User Feedback
Rule-based
Engine
ML UXData
Reports
This may be costly and complicated. But this the compass for your journey
Intuition +1
Roadmap
Rules engine
Model Mgmt Data Features Models
ML
Ensembler
(x models)
Team
● PM
● Architect
● Engineers
● ML Architect
● Data Scientists
● UX Designers
UX A/B + Actuals
Problem Space
Problem worth solving Solution worth a value
37. Product
A/B Testing + Actual User Feedback
Rule-based
Engine
ML UXData
Reports
Start collecting data for future features / models. It takes a year.
Intuition +1
Roadmap
Rules engine
Model Mgmt Data Features Models
ML
Ensembler
(x models)
Team
● PM
● Architect
● Engineers
● ML Architect
● Data Scientists
● UX Designers
UX A/B + Actuals Future cases
Problem Space
Problem worth solving Solution worth a value
38. ML Gains
37x
10x
0 1 2 3
Eng work
Gains/Value
True gains will take a couple years. Year 1 is infrastructure and faking it. Year 2 is focused on Models. Year 3 is shooting for maturity.
Intuition +1
39. Product
ML UX
Reports
Data
Data
A/B Testing + Actual User Feedback
Rule-based
Engine
ML UXData
Reports
Roadmap
Rules engine
Model Mgmt Data Features Models
ML
Ensembler
(x models)
Team
● PM
● Architect
● Engineers
● ML Architect
● Data Scientists
● UX Designers
UX A/B + Actuals Future cases
Focus on building Year 1. Focus on Models Year 2. Maturity Year 3.
Intuition +1
Problem Space
Problem worth solving Solution worth a value
40. Business Side
● ML is often over-prescribed as solution
● ML solutions are often expected unrealistic results
● Expectations are often vaporware driven
● Capabilities are often over marketed
● Goals are often playing catch-up with public spin
● There is a lot of Naysayers
● With ML, data drives possibilities just as much as aspiration
○ From Product driven -> Data driven
● Attribution is hard on efforts. Models may help immensely, may not.
41. Education is part of the job.
Product
ML UX
Reports
Data
Data
A/B Testing + Actual User Feedback
Problem worth solving Solution worth a value
Rule-based
Engine
ML UXData
Reports
Roadmap
Rules engine
Model Mgmt Data Features Models
ML
Ensembler
(x models)
Team
● PM
● Architect
● Engineers
● ML Architect
● Data Scientists
● UX Designers
UX A/B + Actuals Future cases
Create a tons of shareable content. Education is a necessary part of your job.
Intuition +1
Problem Space
42. Redo Canvas when designing ML or bringin ML in Products
Quality of Action vs. Action. Get paid to perform, not do.
Intuition +1
43. Common Sense Lessons
● Real world is a distribution with long tails
● There are no absolutes in guesses (ML)
● Data > Algos. Getting data right is a life saver
● ML driven interfaces = huge UX opportunities
● A lab is an ideal subset of real-life
● Data is the fuel for solutioning
● True gains will take a couple years
● ML is exciting and confusing
● ML is predictive
Intuition Lessons
● Always have a rule based engine alongside ML models
● Plan for multiple models, releases, version, ensembling etc
● Most Data is about the common cases, hence the models are too
● This is a multiplier to all your metrics. Get that low-hanging fruit!
● Real-life = easy 10% drop. Manage expectations and goals
● Start collecting data for future features / models. It takes a year
● Focus on building Year 1. Focus on Models Year 2. Maturity Year 3
● Education is part of the job. Don’t let excitement set goals
● Get paid for performance not actions
If you’re interested to connect with other Product Managers, aspiring PMs, or those within tech, join our Slack community of over 40,000 professionals. It’s a great place to network and to find interesting content. We host a weekly AMA through our Slack channel on Tuesdays from 11:15am - 12pm PST. We have also recently launched the Job Portal where you can find the latest Product Management opportunities! As members of the Product School community, we'd like to provide you with these resources at your disposal.
Product School’s Product Management Certificate Path comprises of 3 part-time courses for professionals with strong technical or business background who want to further explore Product Management at software-based companies.
During Product Management Training you will first learn Product Management fundamentals to understand the software product lifecycle and what it takes to successfully transition into a product management role.
You’ll then be trained to retrieve data, understand its value and make impactful decisions with SQL, data visualization and Tableau. Learn to understand your users to deliver exceptional UX/UI design and develop a robust digital marketing plan. During the Full Stack Product Management Training, you will deep dive into the technical knowledge to enhance your ability to work with agile teams.
Finally, Product Leadership Training will elevate your product knowledge to become an effective Product Leader. You'll do an in-depth analysis on how to implement best PM practices on a strategic level to significantly impact your company’s portfolio and revenue. Learn the soft skills to manage product teams and manage stakeholders to deliver performing products.
As well as individual courses we provide corporate training across the world! If you’d like to upskill your product team this is the best option for you. We have trained employees from multiple companies such as Deloitte, Salesforce, JP Morgan, Bank of America amongst many other companies across all industries.
Tonight's talk is “ [TITLE] ” with [NAME]. Welcome, [NAME].
Feel free to speak with me and I can point you in the right direction (explain where to apply). Or you can visit www.productschool.com
Have a good night!