David Murgatroyd, profile picture
Uploaded byDavid Murgatroyd
PDF, PPTX264 views

Managing Your Machine Learning Portfolio

The document discusses the management of machine learning portfolios, highlighting challenges such as misplaced modeling, crude tools, and under-supported teams. It outlines a framework for the machine learning product lifecycle, emphasizing the progression from global to local models and the importance of strategic investments in shared tooling. Additionally, it addresses necessary skills for team members and various structures for effective team collaboration.

Embed presentation

Download as PDF, PPTX
1 / 111
2 / 111
3 / 111
4 / 111
5 / 111
6 / 111
7 / 111
8 / 111
9 / 111
10 / 111
11 / 111
12 / 111
13 / 111
14 / 111
15 / 111
16 / 111
17 / 111
18 / 111
19 / 111
20 / 111
21 / 111
22 / 111
23 / 111
24 / 111
25 / 111
26 / 111
27 / 111
28 / 111
29 / 111
30 / 111
31 / 111
32 / 111
33 / 111
34 / 111
35 / 111
36 / 111
37 / 111
38 / 111
39 / 111
40 / 111
41 / 111
42 / 111
43 / 111
44 / 111
45 / 111
46 / 111
47 / 111
48 / 111
49 / 111
50 / 111
51 / 111
52 / 111
53 / 111
54 / 111
55 / 111
56 / 111
57 / 111
58 / 111
59 / 111
60 / 111
61 / 111
62 / 111
63 / 111
64 / 111
65 / 111
66 / 111
67 / 111
68 / 111
69 / 111
70 / 111
71 / 111
72 / 111
73 / 111
74 / 111
75 / 111
76 / 111
77 / 111
78 / 111
79 / 111
80 / 111
81 / 111
82 / 111
83 / 111
84 / 111
85 / 111
86 / 111
87 / 111
88 / 111
89 / 111
90 / 111
91 / 111
92 / 111
93 / 111
94 / 111
95 / 111
96 / 111
97 / 111
98 / 111
99 / 111
100 / 111
101 / 111
102 / 111
103 / 111
104 / 111
105 / 111
106 / 111
107 / 111
108 / 111
109 / 111
110 / 111
111 / 111
Managing Your Machine Learning Portfolio David Murgatroyd ( @dmurga) MassTLC ML Dev Day September 4, 2018 (please don’t sue me for copyright violation, Touchstone)
Your customer
Your customer Your ML portfolio
Your product Your customer
Credit: Marta Mandile, DensityDesign Research Lab. CC BY-SA 4.0 Your product Your customer
Your product Your customers
Your product Your customer
Your product Your customer
Your product Your customer
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
A Framework for the ML Product Lifecycle
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Sure it’s possible? Sureit’svaluable?
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Unknown to World Known to World Sure it’s possible? Known to Our Organization Sureit’svaluable?
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product M.V. ML Product
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product No New ML Proto ML Product M.V. ML Product
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
@dmurga Research Delivery Applied Alternative ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product Proto ML Product Academic Results M.V. ML Product Industrial Lab Results Discovery Fundamental
@dmurga Research Discovery Fundamental Delivery Applied Preferred ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
Challenges for ML Portfolios 1. Misplaced modelling.
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Global: More data which changes relatively slowly. Local: Less data which changes relatively quickly.
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Local Models Global Models
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Global Models Local Models
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond More Global More Local ML Product Lifecycle ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
Suggestions for ML Portfolios 1. Progress from global to local models.
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools.
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Then here Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML Proto ML Product
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond
Suggestions for ML Portfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling.
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project Two kinds of support: 1. Individual’s Skills 2. Team Structure
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML ML EngineerSkills Needed
@dmurga Roles
@dmurga Roles
@dmurga Roles ‣ Applied ML Eng ‣ ML Tool Eng ‣ Core Researcher
@dmurga Roles ‣ Applied ML Eng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Core Researcher
@dmurga Roles ‣ Applied ML Eng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
@dmurga Roles # Needed‣ Applied ML Eng ‣ Product ‣ Domain ‣ Experiments ‣ Systems ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Core ML Researcher Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated MLv.non-MLwork Team Structure
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad ‣ More collaboration among MLers ‣ Harder to be product-aligned ‣ Possible deliverables: Models Systems Development effort MLv.non-MLwork Team Structure
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad ‣ Less collaboration among MLers, so need to be senior ‣ Easier to be product-aligned ‣ MLers likely to do non-ML They need to be flexible PM needs to be skilled to cross-prioritize MLv.non-MLwork
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream ‣ Allows more focus when ML progress is more valuable ‣ Prioritization is higher-level: across streams, not tasks ‣ Lots of variety in structure and sharing MLv.non-MLwork
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad ‣ Even more focus for “step change” ‣ Need structure for alignment between siblings (eg., for online experimentation) ‣ Rare state since product discovery may be more valuable MLv.non-MLwork
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
Suggestions for ML Portfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling. 3. Equip individuals and their teams.
@dmurga Research Discovery Fundamental Delivery Applied Preferred ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
@dmurga
Thanks! Questions? David Murgatroyd (@dmurga) Suggestions: What kinds of ways can global and local models combine? What relation does org size have to tool source? How do you train Product Mgrs, Designers, etc., for ML product development? How does the ethics of ML Product Development fit? I hear you’re up for MassTLC’s ML in Action Award? What kind of ML roles are you hiring for in Boston? Hiring in Boston, NYC, London, and Stockholm!

More Related Content

PDF
How to train your product owner
byDavid Murgatroyd
 
PDF
DataScientist Job : Between Myths and Reality.pdf
byJedha Bootcamp
 
PDF
The CMLC 1.0.pdf
byFilipRomeling1
 
PDF
Module_1_Slide_01.pdf
byFazleeKan
 
PPTX
Mission-Driven Machine Learning
byDavid Murgatroyd
 
PDF
Iste 3 out of 5 tech trends that bend 2-2014 final
byJason Ohler
 
PDF
Leveraging AI the Right Way (for Product Managers)
byDavid Murgatroyd
 
PDF
Emerging Best Practises for Machine Learning Engineering- Lex Toumbourou (By ...
byThoughtworks
 
How to train your product owner
byDavid Murgatroyd
 
DataScientist Job : Between Myths and Reality.pdf
byJedha Bootcamp
 
The CMLC 1.0.pdf
byFilipRomeling1
 
Module_1_Slide_01.pdf
byFazleeKan
 
Mission-Driven Machine Learning
byDavid Murgatroyd
 
Iste 3 out of 5 tech trends that bend 2-2014 final
byJason Ohler
 
Leveraging AI the Right Way (for Product Managers)
byDavid Murgatroyd
 
Emerging Best Practises for Machine Learning Engineering- Lex Toumbourou (By ...
byThoughtworks
 

Similar to Managing Your Machine Learning Portfolio

PDF
Machine Learning Teams - Full Stack Deep Learning
bySergey Karayev
 
PDF
Best Practices for Building Successful LLM Applications
byBhavulGauri1
 
PDF
A beginner's guide to machine learning for SEOs - WTSFest 2022
byLazarinaStoyanova
 
PDF
CI/CD for Machine Learning
byC4Media
 
PPTX
UNIT 1 Early Trends AND TECHNOLOGY in ML.pptx
bylakshithashree002
 
PPTX
The Future of AI (September 2019)
byJulien SIMON
 
PDF
Strategic AI Integration in Engineering Teams
byUXDXConf
 
PDF
Fundamental MLOps
bySaripudin Gon
 
PDF
DN18 | Technical Debt in Machine Learning | Jaroslaw Szymczak | OLX
byDataconomy Media
 
PDF
Technical debt in ML | Jaroslaw Szymczak | DN18
byDataconomyGmbH
 
PDF
Elena Grewal, Data Science Manager, Airbnb at MLconf SF 2016
byMLconf
 
PDF
Planning for Success in MDD
bySteven Kelly
 
PDF
Starting your AI/ML project right (May 2020)
byJulien SIMON
 
PDF
Rsqrd AI: From R&D to ROI of AI
bySanjana Chowdhury
 
PDF
DevOps Days Rockies MLOps
byMatthew Reynolds
 
PPTX
Why many data science projects fail
byOmnia Safaan
 
PDF
Managing machine learning
byDavid Murgatroyd
 
PDF
Rsqrd AI: ML Tooling at an AI-first Startup
bySanjana Chowdhury
 
PPTX
Challenges and strategies in bringing AI models to production
byDavid Qixiang Chen
 
ODP
Human + Machine Learning
byMike Wolfson
 
Machine Learning Teams - Full Stack Deep Learning
bySergey Karayev
 
Best Practices for Building Successful LLM Applications
byBhavulGauri1
 
A beginner's guide to machine learning for SEOs - WTSFest 2022
byLazarinaStoyanova
 
CI/CD for Machine Learning
byC4Media
 
UNIT 1 Early Trends AND TECHNOLOGY in ML.pptx
bylakshithashree002
 
The Future of AI (September 2019)
byJulien SIMON
 
Strategic AI Integration in Engineering Teams
byUXDXConf
 
Fundamental MLOps
bySaripudin Gon
 
DN18 | Technical Debt in Machine Learning | Jaroslaw Szymczak | OLX
byDataconomy Media
 
Technical debt in ML | Jaroslaw Szymczak | DN18
byDataconomyGmbH
 
Elena Grewal, Data Science Manager, Airbnb at MLconf SF 2016
byMLconf
 
Planning for Success in MDD
bySteven Kelly
 
Starting your AI/ML project right (May 2020)
byJulien SIMON
 
Rsqrd AI: From R&D to ROI of AI
bySanjana Chowdhury
 
DevOps Days Rockies MLOps
byMatthew Reynolds
 
Why many data science projects fail
byOmnia Safaan
 
Managing machine learning
byDavid Murgatroyd
 
Rsqrd AI: ML Tooling at an AI-first Startup
bySanjana Chowdhury
 
Challenges and strategies in bringing AI models to production
byDavid Qixiang Chen
 
Human + Machine Learning
byMike Wolfson
 

More from David Murgatroyd

PDF
Agile Deep Learning
byDavid Murgatroyd
 
PPTX
NLP in the Real World
byDavid Murgatroyd
 
PPTX
HltCon overview
byDavid Murgatroyd
 
PPTX
Choosing a Job for the Right Reasons
byDavid Murgatroyd
 
PDF
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
byDavid Murgatroyd
 
PPTX
Simple fuzzy name matching in solr
byDavid Murgatroyd
 
PDF
Linguistic Considerations of Identity Resolution (2008)
byDavid Murgatroyd
 
PDF
Technology & Faith: from Coding to Culture
byDavid Murgatroyd
 
PPTX
System combination for HLT
byDavid Murgatroyd
 
PPTX
From Research to Reality: Advances in HLT 2013
byDavid Murgatroyd
 
Agile Deep Learning
byDavid Murgatroyd
 
NLP in the Real World
byDavid Murgatroyd
 
HltCon overview
byDavid Murgatroyd
 
Choosing a Job for the Right Reasons
byDavid Murgatroyd
 
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
byDavid Murgatroyd
 
Simple fuzzy name matching in solr
byDavid Murgatroyd
 
Linguistic Considerations of Identity Resolution (2008)
byDavid Murgatroyd
 
Technology & Faith: from Coding to Culture
byDavid Murgatroyd
 
System combination for HLT
byDavid Murgatroyd
 
From Research to Reality: Advances in HLT 2013
byDavid Murgatroyd
 

Recently uploaded

PPTX
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
PDF
Agentic AI Roadmap 2026: Mastering Autonomous AI Workflows and Systems
byAeafat Ahmed Mubin
 
PDF
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
PDF
Introduction to Linux and Basic Commands
byiDev Semarang
 
PDF
Talaria: A Sound-Driven Music Search Engine for Discovery Beyond Metadata
byShuen-Huei Guan
 
PDF
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
PDF
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
PPTX
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
PPTX
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
PDF
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
PDF
Computer Science Conferences 2026 | Research by SAI Conference
byHealth Conference 2026
 
PDF
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
PDF
Smart Cloud Solutions-Smart,Secure & Scalable
byfloydjsmith02
 
PDF
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
PPTX
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
PPTX
Single Cell Protein from Methane or Methanol - Process development research c...
byJohn Downs
 
DOCX
Best Web to Learn About Buying Verified Skrill Accounts (Los Angeles).docx
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
PDF
Lightweight, Travel-Ready Freedom for Adults and Seniors
byTopmate
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfortable, Stable Mobility for Every...
byTopmate
 
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
Agentic AI Roadmap 2026: Mastering Autonomous AI Workflows and Systems
byAeafat Ahmed Mubin
 
Skills to Pass the UiPath Agentic Automation Associate (UiAAA) Certification
byUiPathCommunity
 
Introduction to Linux and Basic Commands
byiDev Semarang
 
Talaria: A Sound-Driven Music Search Engine for Discovery Beyond Metadata
byShuen-Huei Guan
 
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
Presentation of Cybersecurity and data protection
bytarikaityahya2
 
Understanding-Search-Algorithms_09-12-25.pptx
byshaikmahammedtauheed
 
API Gateway Architecture - Technical Report 2026
byPowersoft2026
 
What ONIX can do: Leveraging metadata to support the discoverability of First...
byBookNet Canada
 
Computer Science Conferences 2026 | Research by SAI Conference
byHealth Conference 2026
 
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
Smart Cloud Solutions-Smart,Secure & Scalable
byfloydjsmith02
 
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
AI Meets DBA Transforming Database Administration with Intelligence
byGeoPITS Global Pvt Ltd
 
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
Single Cell Protein from Methane or Methanol - Process development research c...
byJohn Downs
 
Best Web to Learn About Buying Verified Skrill Accounts (Los Angeles).docx
byhttps://topsellerit.com/product/buy-verified-binance-accounts/
 
Lightweight, Travel-Ready Freedom for Adults and Seniors
byTopmate
 
TopMate ES11 3-Wheel Electric Scooter: Comfortable, Stable Mobility for Every...
byTopmate
 

Managing Your Machine Learning Portfolio

  • 1.
    Managing Your Machine LearningPortfolio David Murgatroyd ( @dmurga) MassTLC ML Dev Day September 4, 2018 (please don’t sue me for copyright violation, Touchstone)
  • 2.
  • 3.
  • 4.
  • 5.
    Credit: Marta Mandile,DensityDesign Research Lab. CC BY-SA 4.0 Your product Your customer
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 11.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 12.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 13.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 14.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 15.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 16.
    Challenges for MLPortfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
  • 17.
    A Framework forthe ML Product Lifecycle
  • 18.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work Development Sure it’s possible? Sureit’svaluable?
  • 19.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work Development Unknown to World Known to World Sure it’s possible? Known to Our Organization Sureit’svaluable?
  • 20.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 21.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 22.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 23.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 24.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 25.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 26.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 27.
    @dmurga Research Discovery Fundamental Delivery Applied Kindsof R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 28.
    @dmurga Research Discovery Fundamental Delivery Applied MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 29.
    @dmurga Research Discovery Fundamental Delivery Applied MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML
  • 30.
    @dmurga Research Discovery Fundamental Delivery Applied MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product
  • 31.
    @dmurga Research Discovery Fundamental Delivery Applied MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product M.V. ML Product
  • 32.
    @dmurga Research Discovery Fundamental Delivery Applied MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product No New ML Proto ML Product M.V. ML Product
  • 33.
    @dmurga Research Discovery Fundamental Delivery Applied MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
  • 34.
    @dmurga Research Delivery Applied Alternative MLProduct Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product Proto ML Product Academic Results M.V. ML Product Industrial Lab Results Discovery Fundamental
  • 35.
    @dmurga Research Discovery Fundamental Delivery Applied PreferredML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
  • 36.
    Challenges for MLPortfolios 1. Misplaced modelling.
  • 37.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 38.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 39.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 40.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 41.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 42.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 43.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project
  • 44.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project
  • 45.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project
  • 46.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project
  • 47.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project Global: More data which changes relatively slowly. Local: Less data which changes relatively quickly.
  • 48.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project Local Models Global Models
  • 49.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project Global Models Local Models
  • 50.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond More Global More Local ML Product Lifecycle ModelCombination
  • 51.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 52.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 53.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 54.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 55.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 56.
    Suggestions for MLPortfolios 1. Progress from global to local models.
  • 57.
    Challenges for MLPortfolios 1. Misplaced modelling. 2. Crude tools.
  • 58.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 59.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 60.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 61.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
  • 62.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
  • 63.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 64.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 65.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 66.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 67.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Where to get your ML Tools?
  • 68.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Then here Where to get your ML Tools?
  • 69.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML
  • 70.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML Proto ML Product
  • 71.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product
  • 72.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product
  • 73.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond
  • 74.
    Suggestions for MLPortfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling.
  • 75.
    Challenges for MLPortfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
  • 76.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 77.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 78.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 79.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project
  • 80.
    @dmurgaPeople by ProSymbolsfrom the Noun Project Music by Iconnic from the Noun Project Two kinds of support: 1. Individual’s Skills 2. Team Structure
  • 81.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Skills Needed
  • 82.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 83.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 84.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 85.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 86.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 87.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML ML EngineerSkills Needed
  • 88.
  • 89.
  • 90.
    @dmurga Roles ‣ Applied MLEng ‣ ML Tool Eng ‣ Core Researcher
  • 91.
    @dmurga Roles ‣ Applied MLEng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Core Researcher
  • 92.
    @dmurga Roles ‣ Applied MLEng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
  • 93.
    @dmurga Roles # Needed‣ AppliedML Eng ‣ Product ‣ Domain ‣ Experiments ‣ Systems ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
  • 94.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Skills Needed
  • 95.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Core ML Researcher Skills Needed
  • 96.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated MLv.non-MLwork Team Structure
  • 97.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad ‣ More collaboration among MLers ‣ Harder to be product-aligned ‣ Possible deliverables: Models Systems Development effort MLv.non-MLwork Team Structure
  • 98.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad ‣ Less collaboration among MLers, so need to be senior ‣ Easier to be product-aligned ‣ MLers likely to do non-ML They need to be flexible PM needs to be skilled to cross-prioritize MLv.non-MLwork
  • 99.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream ‣ Allows more focus when ML progress is more valuable ‣ Prioritization is higher-level: across streams, not tasks ‣ Lots of variety in structure and sharing MLv.non-MLwork
  • 100.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad ‣ Even more focus for “step change” ‣ Need structure for alignment between siblings (eg., for online experimentation) ‣ Rare state since product discovery may be more valuable MLv.non-MLwork
  • 101.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
  • 102.
    Challenges for MLPortfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
  • 103.
    Suggestions for MLPortfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling. 3. Equip individuals and their teams.
  • 104.
    @dmurga Research Discovery Fundamental Delivery Applied PreferredML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
  • 105.
    @dmurgaPeople by ProSymbolsfrom the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 106.
    Stages of MLWorkflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 107.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager
  • 108.
    @dmurga M.V. ML Product No New ML ProtoML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
  • 110.
  • 111.
    Thanks! Questions? David Murgatroyd(@dmurga) Suggestions: What kinds of ways can global and local models combine? What relation does org size have to tool source? How do you train Product Mgrs, Designers, etc., for ML product development? How does the ethics of ML Product Development fit? I hear you’re up for MassTLC’s ML in Action Award? What kind of ML roles are you hiring for in Boston? Hiring in Boston, NYC, London, and Stockholm!