Basic tips for Product Managers on readiness and preparing for building AI products within a Agile Startup culture. Defining problems, understanding solutions and future considerations.

1. BUILDINGAGILE&AI
STARTUPS
BASIC TIPS FOR PRODUCT MANAGERS
By John Fagan

2. R & Pandas & pythons & trees
Notebooks
Tensor flow
Models
AL / ML / Deep learning
Supervised learning
Unsupervised learning
Semi unsupervised
Transferred learning
Reinforcement learning
Back propagation
Forward propagation
Over fitting / under-fitting
Confusion Matrix
WENOWHAVE
DOCTORSINTHEHOUSE
INTIMIDATING NEW WORLD FOR PRODUCT MANAGERS
Inference
Classification, Regression, Correlation
FPs
Outlier / Normalize
Statistical Significance
Variance
Feature Engineering
Neural Networks
LSTMS
Quantile
Random Forest (may forests!)
Confidence interval
F1 Score
Precision / Recall
Mean Squared Err
@johnbfagan

3. WHATDOTHESEWIZARDSDO?@johnbfagan

4. https://towardsdatascience.com/demystifying-the-data-science-job-families-c74f2294b1bd
THEREARENOWIZARDS
@johnbfagan

5. https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007
FIRST.ISYOURDATASTACKREADY?
@johnbfagan

6. • Understand basic components and work flows
• Get familiar with basic terms, methods and
constructs e.g. supervised, unsupervised,
classification, algorithm and models, model
evaluation, variance, bias, overfitting, precision,
recall
• Listen to Podcasts
• Loads on Youtube
• Do a short course
ACQUIREDATASCIENCELITERACY
PREPAREBEFORETHE
DOCTORSARRIVE
@johnbfagan

7. Use the Retrospective to make small and continuous
improvements.
Data hygiene & integrity
Data models
Transaction ids
Data flows
Simple data archiving mechanic
Gradually tune up your Definition of Done
Consider hiring a data science consultant on short term
contract
PREPAREBEFORETHE
DOCTORSARRIVE
GET YOUR DATA READY.
@johnbfagan

8. https://medium.com/@hugh_data_science/the-pyramid-of-data-needs-and-why-it-matters-for-your-career-b0f695c13f11
https://courses.lumenlearning.com/wmintrobusiness/chapter/reading-stages-of-the-product-life-cycle/
PYRAMID OF DATA NEEDS
THINK.EVOLUTIONNOTREVOLUTION
ALIGNED TO PRODUCT DEVELOPMENT & STARTUP LIFECYCLE
@johnbfagan

9. https://medium.com/jalapeno-app/waterfall-agile-spectrum-e0bff7efa2e1
PLAN.FORCLASHOFCULTURES
🤔CANYOUHAVEA3POINTUSERSTORY ANDTHROWINA“DOTHEDATASCIENCE”SUBTASK?
@johnbfagan

10. https://www.oreilly.com/radar/a-manifesto-for-agile-data-science/
GREAT MODEL FOR UNLOCKING VALUE IN A AGILE FRAMEWORK, STORIES AT EACH LEVEL
THINK.ABOUTTHEDATAVALUEPYRAMID
@johnbfagan

11. READY?DEFINEYOURPROBLEM

12. Allows you to align expectations and outcomes.
You should spend time collaborating with your
team (engineers, data science, testers and
management) on defining the problem,
assumptions & expected outcomes. More so
than you would with a classic problem which is
solved by CRUD.
Luckily Machine Learning Mastery have a great
template, which I have adapted to agile stories.
https://machinelearningmastery.com/how-to-define-your-machine-learning-problem/
MUST.DEFINETHE
PROBLEM
PUT A LOT OF LOVE INTO THIS
@johnbfagan

13. MACHINELEARNING
MASTERY Tests the boundaries
and re-tests the
problem statement and
assumptions
Breaks the solution
down into layman level
@johnbfagan

14. IN ORDER TO become a social medial influencer
AS A regular twitter user
I NEED twitter to predict if my draft tweet content will get retweets
GIVEN there is a history of tweets from @illizian
AND some have retweets
AND some do not
WHEN @illizian composes a new tweet
THEN classify the tweet if its going to get retweets or not
AND ensure the classification model has an accuracy score as a percentage.
AND the accuracy score is the number of tweets predicted correctly out of all tweets
AND The specific words he used in the tweet matter to the model.
AND The specific user that retweets does not matter to the model.
AND The number of retweets may matter to the model.
AND Older tweets are less predictive than more recent tweets.
TRANSLATETOUSERSTORIESWITHBDD
@johnbfagan

15. F1 score - measure of a test's accuracy. It considers both
the precision p and the recall r
False Positives - a test result which wrongly indicates that a
particular condition or attribute is present.
False Negatives - a test result which wrongly indicates that
a particular condition or attribute is absent.
Tradeoffs - impact mapping milestone a great way to
describe tradeoffs of quality versus, time and cost and
define you Go vs No-Go Metrics
MUST. CAREABOUT
SUCCESSMETRICS
WHAT DOES SUCCESS LOOK LIKE?
https://www.productschool.com/blog/product-management-2/great-machine-learning-product-management-google/
https://www.impactmapping.org/
@johnbfagan

16. SOLVINGYOURPREDICTIVE
CHALLENGE

17. We all have a great solution, the best solution, but machine
learning is just one solution along with many others.
First create a super dumb baseline model (!AI), e.g.
• 100% certainty each tweet will be RT’d!
• Use average % of last 100 tweets that got RT’d
• If any words (excluding stop words) previously got
RT’d, then 100% certain tweet will get RT’d!
You might be surprised that your super simple solution is fit
for purpose
STARTWITHTHE
DUMBESTSOLUTION
@johnbfagan

18. https://www.youtube.com/watch?v=TK-2189UcKk
MUST. CAREABOUT
TESTDATA.
BUILD YOUR TEST DATA & AGREE KPIS
@johnbfagan

19. https://www.datarobot.com/wiki/cross-validation/
MUST.UNDERSTANDMODEL
EVALUATION
Validation data, later used for training too
Validation techniques
• Hold out - cross validation
• K fold - cross-validation
Beware of bias, overfitting & underfitting
CROSS VALIDATION
@johnbfagan

20. https://www.youtube.com/watch?v=EuBBz3bI-aA
VARIANCE AND BIAS
MUST.ENSUREYOUARETRAINING
FORTHERIGHTOUTCOMES.
@johnbfagan

21. https://machinelearningmastery.com/a-tour-of-machine-learning-algorithms/
FORGET.ALGORITHMS
YOU HIRE DATA SCIENTISTS TO WORRY ABOUT THIS.
@johnbfagan

22. https://www.oreilly.com/library/view/evaluating-machine-learning/9781492048756/ch01.html
MAKE SURE YOU LET YOUR TEAM INVEST IN THIS
MUST.WORRYABOUT INFRASTRUCTURE&DEPLOYMENT
@johnbfagan

23. GOINGFROMSTARTUPTOSCALEUP

24. https://research.aimultiple.com/data-science-tools/
BUILD.DATASCIENCESTACKS
@johnbfagan

25. Software is usually static, but data is
always changing.
Monitor algorithms performance for drift
Adapt by understanding, re-fitting,
updating, weighting, learning the
changes.
MONITOR.DRIFT
BEHAVIOURS ALWAYS CHANGE.
https://www.semanticscholar.org/paper/Concept-drift-adaptation-for-learning-with-data-Liu/5b105e357936f989cfb46ddd055ea44a2b0aed04
https://machinelearningmastery.com/gentle-introduction-concept-drift-machine-learning/
@johnbfagan

26. https://www.grazitti.com/blog/data-lake-vs-data-warehouse-which-one-should-you-go-for/
GROW.DATALAKES
@johnbfagan

27. HTTPS://WWW.GOOGLE.COM/URL?SA=I&URL=HTTPS%3A%2F%2FWWW.DIALOGUES.ORG%2F08%2F02%2F2018%2FSOFTWARE-ENGINEER-VS-DATA-SCIENTIST-INTERVIEW-FEATURING-JOMA-TECH-
%2F1533228768&PSIG=AOVVAW18WWIM9CIR1QCAHVJYJHVH&UST=1590168327220000&SOURCE=IMAGES&CD=VFE&VED=0CA0QJHXQFWOTCLITHOW8XEKCFQAAAAADAAAAABAI
ENABLE.COLLABORATION
@johnbfagan

28. THANKS