SlideShare a Scribd company logo

Managing Your Machine Learning Portfolio

•
•
David Murgatroyd
David Murgatroyd

Applying machine learning to a particular business need becomes more straightforward with each technological advance. But today’s businesses have a variety of needs which are too numerous to be addressed one-at-a-time and too different to be addressed one-size-fits-all. We examine three significant challenges to building an effective ML portfolio and ways to address them thru the framework of the ML product lifecycle.

Technology
1 of 111
Download to read offline
Managing Your Machine Learning Portfolio David Murgatroyd ( @dmurga) MassTLC ML Dev Day September 4, 2018 (please don’t sue me for copyright violation, Touchstone)
Your customer
Your customer Your ML portfolio
Your product Your customer
Credit: Marta Mandile, DensityDesign Research Lab. CC BY-SA 4.0 Your product Your customer
Your product Your customers
Your product Your customer
Your product Your customer
Your product Your customer
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
A Framework for the ML Product Lifecycle
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Sure it’s possible? Sureit’svaluable?
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Unknown to World Known to World Sure it’s possible? Known to Our Organization Sureit’svaluable?
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product M.V. ML Product
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product No New ML Proto ML Product M.V. ML Product
@dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
@dmurga Research Delivery Applied Alternative ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product Proto ML Product Academic Results M.V. ML Product Industrial Lab Results Discovery Fundamental
@dmurga Research Discovery Fundamental Delivery Applied Preferred ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
Challenges for ML Portfolios 1. Misplaced modelling.
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Global: More data which changes relatively slowly. Local: Less data which changes relatively quickly.
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Local Models Global Models
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Global Models Local Models
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond More Global More Local ML Product Lifecycle ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
Suggestions for ML Portfolios 1. Progress from global to local models.
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools.
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Then here Where to get your ML Tools?
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML Proto ML Product
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond
Suggestions for ML Portfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling.
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
@dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project Two kinds of support: 1. Individual’s Skills 2. Team Structure
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML ML EngineerSkills Needed
@dmurga Roles
@dmurga Roles
@dmurga Roles ‣ Applied ML Eng ‣ ML Tool Eng ‣ Core Researcher
@dmurga Roles ‣ Applied ML Eng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Core Researcher
@dmurga Roles ‣ Applied ML Eng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
@dmurga Roles # Needed‣ Applied ML Eng ‣ Product ‣ Domain ‣ Experiments ‣ Systems ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Core ML Researcher Skills Needed
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated MLv.non-MLwork Team Structure
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad ‣ More collaboration among MLers ‣ Harder to be product-aligned ‣ Possible deliverables: Models Systems Development effort MLv.non-MLwork Team Structure
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad ‣ Less collaboration among MLers, so need to be senior ‣ Easier to be product-aligned ‣ MLers likely to do non-ML They need to be flexible PM needs to be skilled to cross-prioritize MLv.non-MLwork
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream ‣ Allows more focus when ML progress is more valuable ‣ Prioritization is higher-level: across streams, not tasks ‣ Lots of variety in structure and sharing MLv.non-MLwork
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad ‣ Even more focus for “step change” ‣ Need structure for alignment between siblings (eg., for online experimentation) ‣ Rare state since product discovery may be more valuable MLv.non-MLwork
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
Suggestions for ML Portfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling. 3. Equip individuals and their teams.
@dmurga Research Discovery Fundamental Delivery Applied Preferred ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
@dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager
@dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
@dmurga
Thanks! Questions? David Murgatroyd (@dmurga) Suggestions: What kinds of ways can global and local models combine? What relation does org size have to tool source? How do you train Product Mgrs, Designers, etc., for ML product development? How does the ethics of ML Product Development fit? I hear you’re up for MassTLC’s ML in Action Award? What kind of ML roles are you hiring for in Boston? Hiring in Boston, NYC, London, and Stockholm!

Recommended

Leveraging AI the Right Way (for Product Managers)
Leveraging AI the Right Way (for Product Managers)Leveraging AI the Right Way (for Product Managers)
Leveraging AI the Right Way (for Product Managers)
David Murgatroyd
 
Mission-Driven Machine Learning
Mission-Driven Machine LearningMission-Driven Machine Learning
Mission-Driven Machine Learning
David Murgatroyd
 
Iste 3 out of 5 tech trends that bend 2-2014 final
Iste 3 out of 5 tech trends that bend 2-2014 finalIste 3 out of 5 tech trends that bend 2-2014 final
Iste 3 out of 5 tech trends that bend 2-2014 final
Jason Ohler
 
A birds eye view of social media for education
A birds eye view of social media for educationA birds eye view of social media for education
A birds eye view of social media for education
Bernard Goldbach
 
Assignment 1 final
Assignment 1 finalAssignment 1 final
Assignment 1 final
IssybellaLang
 
We need to Talk about Frank
We need to Talk about FrankWe need to Talk about Frank
We need to Talk about Frank
Emma Howell
 
Building Awesome Products (BayCHI)
Building Awesome Products (BayCHI)Building Awesome Products (BayCHI)
Building Awesome Products (BayCHI)
ZURB
 
Life Cycle of an App - From Idea to Monetization
Life Cycle of an App - From Idea to Monetization Life Cycle of an App - From Idea to Monetization
Life Cycle of an App - From Idea to Monetization
Deepu S Nath
 

Recommended

Leveraging AI the Right Way (for Product Managers)
Leveraging AI the Right Way (for Product Managers)Leveraging AI the Right Way (for Product Managers)
Leveraging AI the Right Way (for Product Managers)
David Murgatroyd
 
Mission-Driven Machine Learning
Mission-Driven Machine LearningMission-Driven Machine Learning
Mission-Driven Machine Learning
David Murgatroyd
 
Iste 3 out of 5 tech trends that bend 2-2014 final
Iste 3 out of 5 tech trends that bend 2-2014 finalIste 3 out of 5 tech trends that bend 2-2014 final
Iste 3 out of 5 tech trends that bend 2-2014 final
Jason Ohler
 
A birds eye view of social media for education
A birds eye view of social media for educationA birds eye view of social media for education
A birds eye view of social media for education
Bernard Goldbach
 
Assignment 1 final
Assignment 1 finalAssignment 1 final
Assignment 1 final
IssybellaLang
 
We need to Talk about Frank
We need to Talk about FrankWe need to Talk about Frank
We need to Talk about Frank
Emma Howell
 
Building Awesome Products (BayCHI)
Building Awesome Products (BayCHI)Building Awesome Products (BayCHI)
Building Awesome Products (BayCHI)
ZURB
 
Life Cycle of an App - From Idea to Monetization
Life Cycle of an App - From Idea to Monetization Life Cycle of an App - From Idea to Monetization
Life Cycle of an App - From Idea to Monetization
Deepu S Nath
 
Semantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and StanbolSemantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and Stanbol
All Things Open
 
Social media for hyperlocal tourism
Social media for hyperlocal tourismSocial media for hyperlocal tourism
Social media for hyperlocal tourism
Bernard Goldbach
 
Social media for hyperlocal tourism
Social media for hyperlocal tourismSocial media for hyperlocal tourism
Social media for hyperlocal tourism
Limerick School of Art and Design
 
Designers shouldn't code - The wrong answer to the right question
Designers shouldn't code - The wrong answer to the right questionDesigners shouldn't code - The wrong answer to the right question
Designers shouldn't code - The wrong answer to the right question
Mathias Hellquist
 
Goto Berlin - Migrating to Microservices (Fast Delivery)
Goto Berlin - Migrating to Microservices (Fast Delivery)Goto Berlin - Migrating to Microservices (Fast Delivery)
Goto Berlin - Migrating to Microservices (Fast Delivery)
Adrian Cockcroft
 
How to increase the social impact you make
How to increase the social impact you makeHow to increase the social impact you make
How to increase the social impact you make
Dale Lane
 
A call to give back puppetlabs-corosync to the community
A call to give back puppetlabs-corosync to the communityA call to give back puppetlabs-corosync to the community
A call to give back puppetlabs-corosync to the community
Julien Pivotto
 
The End of Mobile Destinations
The End of Mobile DestinationsThe End of Mobile Destinations
The End of Mobile Destinations
Raimo Van Der Klein
 
PyData: Past, Present Future (PyData SV 2014 Keynote)
PyData: Past, Present Future (PyData SV 2014 Keynote)PyData: Past, Present Future (PyData SV 2014 Keynote)
PyData: Past, Present Future (PyData SV 2014 Keynote)
Peter Wang
 
Puppet and Software Delivery
Puppet and Software DeliveryPuppet and Software Delivery
Puppet and Software Delivery
Julien Pivotto
 
Upgrade Your Offer! How to Sell Business Value
Upgrade Your Offer! How to Sell Business ValueUpgrade Your Offer! How to Sell Business Value
Upgrade Your Offer! How to Sell Business Value
Jakob Persson
 
For, With, Through Design
For, With, Through DesignFor, With, Through Design
For, With, Through Design
Celine Pering
 
Design for Non-Designers
Design for Non-DesignersDesign for Non-Designers
Design for Non-Designers
All Things Open
 
Advancing UX in Your Organization (Fluxible Conference, September 2013)
Advancing UX in Your Organization (Fluxible Conference, September 2013)Advancing UX in Your Organization (Fluxible Conference, September 2013)
Advancing UX in Your Organization (Fluxible Conference, September 2013)
Kimberley Peter
 
The wall falls down: Integrating our online and offline worlds [Confab 2015]
The wall falls down: Integrating our online and offline worlds [Confab 2015]The wall falls down: Integrating our online and offline worlds [Confab 2015]
The wall falls down: Integrating our online and offline worlds [Confab 2015]
Noz Urbina
 
Facebook Marketing - Creatives
Facebook Marketing - CreativesFacebook Marketing - Creatives
Facebook Marketing - Creatives
Instagram
 
Mobiele sites en applicatie op Drupal
Mobiele sites en applicatie op DrupalMobiele sites en applicatie op Drupal
Mobiele sites en applicatie op Drupal
CrossLang / CrossCheck / CrossCast
 
Introduction to Assumptions Mapping - Agile2016
Introduction to Assumptions Mapping - Agile2016Introduction to Assumptions Mapping - Agile2016
Introduction to Assumptions Mapping - Agile2016
David Bland
 
Zilliz - Overview of Generative models in ML
Zilliz - Overview of Generative models in MLZilliz - Overview of Generative models in ML
Zilliz - Overview of Generative models in ML
Zilliz
 
Deepfakes - How they work and what it means for the future
Deepfakes - How they work and what it means for the futureDeepfakes - How they work and what it means for the future
Deepfakes - How they work and what it means for the future
Jarrod Overson
 
How to train your product owner
How to train your product ownerHow to train your product owner
How to train your product owner
David Murgatroyd
 
Technology & Faith: from Coding to Culture
Technology & Faith: from Coding to CultureTechnology & Faith: from Coding to Culture
Technology & Faith: from Coding to Culture
David Murgatroyd
 

More Related Content

Similar to Managing Your Machine Learning Portfolio

Semantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and StanbolSemantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and Stanbol
All Things Open
 
Social media for hyperlocal tourism
Social media for hyperlocal tourismSocial media for hyperlocal tourism
Social media for hyperlocal tourism
Bernard Goldbach
 
Social media for hyperlocal tourism
Social media for hyperlocal tourismSocial media for hyperlocal tourism
Social media for hyperlocal tourism
Limerick School of Art and Design
 
Designers shouldn't code - The wrong answer to the right question
Designers shouldn't code - The wrong answer to the right questionDesigners shouldn't code - The wrong answer to the right question
Designers shouldn't code - The wrong answer to the right question
Mathias Hellquist
 
Goto Berlin - Migrating to Microservices (Fast Delivery)
Goto Berlin - Migrating to Microservices (Fast Delivery)Goto Berlin - Migrating to Microservices (Fast Delivery)
Goto Berlin - Migrating to Microservices (Fast Delivery)
Adrian Cockcroft
 
How to increase the social impact you make
How to increase the social impact you makeHow to increase the social impact you make
How to increase the social impact you make
Dale Lane
 
A call to give back puppetlabs-corosync to the community
A call to give back puppetlabs-corosync to the communityA call to give back puppetlabs-corosync to the community
A call to give back puppetlabs-corosync to the community
Julien Pivotto
 
The End of Mobile Destinations
The End of Mobile DestinationsThe End of Mobile Destinations
The End of Mobile Destinations
Raimo Van Der Klein
 
PyData: Past, Present Future (PyData SV 2014 Keynote)
PyData: Past, Present Future (PyData SV 2014 Keynote)PyData: Past, Present Future (PyData SV 2014 Keynote)
PyData: Past, Present Future (PyData SV 2014 Keynote)
Peter Wang
 
Puppet and Software Delivery
Puppet and Software DeliveryPuppet and Software Delivery
Puppet and Software Delivery
Julien Pivotto
 
Upgrade Your Offer! How to Sell Business Value
Upgrade Your Offer! How to Sell Business ValueUpgrade Your Offer! How to Sell Business Value
Upgrade Your Offer! How to Sell Business Value
Jakob Persson
 
For, With, Through Design
For, With, Through DesignFor, With, Through Design
For, With, Through Design
Celine Pering
 
Design for Non-Designers
Design for Non-DesignersDesign for Non-Designers
Design for Non-Designers
All Things Open
 
Advancing UX in Your Organization (Fluxible Conference, September 2013)
Advancing UX in Your Organization (Fluxible Conference, September 2013)Advancing UX in Your Organization (Fluxible Conference, September 2013)
Advancing UX in Your Organization (Fluxible Conference, September 2013)
Kimberley Peter
 
The wall falls down: Integrating our online and offline worlds [Confab 2015]
The wall falls down: Integrating our online and offline worlds [Confab 2015]The wall falls down: Integrating our online and offline worlds [Confab 2015]
The wall falls down: Integrating our online and offline worlds [Confab 2015]
Noz Urbina
 
Facebook Marketing - Creatives
Facebook Marketing - CreativesFacebook Marketing - Creatives
Facebook Marketing - Creatives
Instagram
 
Mobiele sites en applicatie op Drupal
Mobiele sites en applicatie op DrupalMobiele sites en applicatie op Drupal
Mobiele sites en applicatie op Drupal
CrossLang / CrossCheck / CrossCast
 
Introduction to Assumptions Mapping - Agile2016
Introduction to Assumptions Mapping - Agile2016Introduction to Assumptions Mapping - Agile2016
Introduction to Assumptions Mapping - Agile2016
David Bland
 
Zilliz - Overview of Generative models in ML
Zilliz - Overview of Generative models in MLZilliz - Overview of Generative models in ML
Zilliz - Overview of Generative models in ML
Zilliz
 
Deepfakes - How they work and what it means for the future
Deepfakes - How they work and what it means for the futureDeepfakes - How they work and what it means for the future
Deepfakes - How they work and what it means for the future
Jarrod Overson
 

Similar to Managing Your Machine Learning Portfolio (20)

Semantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and StanbolSemantic Integration with Apache Jena and Stanbol
Semantic Integration with Apache Jena and Stanbol
 
Social media for hyperlocal tourism
Social media for hyperlocal tourismSocial media for hyperlocal tourism
Social media for hyperlocal tourism
 
Social media for hyperlocal tourism
Social media for hyperlocal tourismSocial media for hyperlocal tourism
Social media for hyperlocal tourism
 
Designers shouldn't code - The wrong answer to the right question
Designers shouldn't code - The wrong answer to the right questionDesigners shouldn't code - The wrong answer to the right question
Designers shouldn't code - The wrong answer to the right question
 
Goto Berlin - Migrating to Microservices (Fast Delivery)
Goto Berlin - Migrating to Microservices (Fast Delivery)Goto Berlin - Migrating to Microservices (Fast Delivery)
Goto Berlin - Migrating to Microservices (Fast Delivery)
 
How to increase the social impact you make
How to increase the social impact you makeHow to increase the social impact you make
How to increase the social impact you make
 
A call to give back puppetlabs-corosync to the community
A call to give back puppetlabs-corosync to the communityA call to give back puppetlabs-corosync to the community
A call to give back puppetlabs-corosync to the community
 
The End of Mobile Destinations
The End of Mobile DestinationsThe End of Mobile Destinations
The End of Mobile Destinations
 
PyData: Past, Present Future (PyData SV 2014 Keynote)
PyData: Past, Present Future (PyData SV 2014 Keynote)PyData: Past, Present Future (PyData SV 2014 Keynote)
PyData: Past, Present Future (PyData SV 2014 Keynote)
 
Puppet and Software Delivery
Puppet and Software DeliveryPuppet and Software Delivery
Puppet and Software Delivery
 
Upgrade Your Offer! How to Sell Business Value
Upgrade Your Offer! How to Sell Business ValueUpgrade Your Offer! How to Sell Business Value
Upgrade Your Offer! How to Sell Business Value
 
For, With, Through Design
For, With, Through DesignFor, With, Through Design
For, With, Through Design
 
Design for Non-Designers
Design for Non-DesignersDesign for Non-Designers
Design for Non-Designers
 
Advancing UX in Your Organization (Fluxible Conference, September 2013)
Advancing UX in Your Organization (Fluxible Conference, September 2013)Advancing UX in Your Organization (Fluxible Conference, September 2013)
Advancing UX in Your Organization (Fluxible Conference, September 2013)
 
The wall falls down: Integrating our online and offline worlds [Confab 2015]
The wall falls down: Integrating our online and offline worlds [Confab 2015]The wall falls down: Integrating our online and offline worlds [Confab 2015]
The wall falls down: Integrating our online and offline worlds [Confab 2015]
 
Facebook Marketing - Creatives
Facebook Marketing - CreativesFacebook Marketing - Creatives
Facebook Marketing - Creatives
 
Mobiele sites en applicatie op Drupal
Mobiele sites en applicatie op DrupalMobiele sites en applicatie op Drupal
Mobiele sites en applicatie op Drupal
 
Introduction to Assumptions Mapping - Agile2016
Introduction to Assumptions Mapping - Agile2016Introduction to Assumptions Mapping - Agile2016
Introduction to Assumptions Mapping - Agile2016
 
Zilliz - Overview of Generative models in ML
Zilliz - Overview of Generative models in MLZilliz - Overview of Generative models in ML
Zilliz - Overview of Generative models in ML
 
Deepfakes - How they work and what it means for the future
Deepfakes - How they work and what it means for the futureDeepfakes - How they work and what it means for the future
Deepfakes - How they work and what it means for the future
 

More from David Murgatroyd

How to train your product owner
How to train your product ownerHow to train your product owner
How to train your product owner
David Murgatroyd
 
Technology & Faith: from Coding to Culture
Technology & Faith: from Coding to CultureTechnology & Faith: from Coding to Culture
Technology & Faith: from Coding to Culture
David Murgatroyd
 
Agile Deep Learning
Agile Deep LearningAgile Deep Learning
Agile Deep Learning
David Murgatroyd
 
Choosing a Job for the Right Reasons
Choosing a Job for the Right ReasonsChoosing a Job for the Right Reasons
Choosing a Job for the Right Reasons
David Murgatroyd
 
Managing machine learning
Managing machine learningManaging machine learning
Managing machine learning
David Murgatroyd
 
NLP in the Real World
NLP in the Real WorldNLP in the Real World
NLP in the Real World
David Murgatroyd
 
System combination for HLT
System combination for HLTSystem combination for HLT
System combination for HLT
David Murgatroyd
 
HltCon overview
HltCon overviewHltCon overview
HltCon overview
David Murgatroyd
 
Simple fuzzy name matching in solr
Simple fuzzy name matching in solrSimple fuzzy name matching in solr
Simple fuzzy name matching in solr
David Murgatroyd
 
Linguistic Considerations of Identity Resolution (2008)
Linguistic Considerations of Identity Resolution (2008)Linguistic Considerations of Identity Resolution (2008)
Linguistic Considerations of Identity Resolution (2008)
David Murgatroyd
 
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
David Murgatroyd
 
From Research to Reality: Advances in HLT 2013
From Research to Reality: Advances in HLT 2013From Research to Reality: Advances in HLT 2013
From Research to Reality: Advances in HLT 2013
David Murgatroyd
 

More from David Murgatroyd (12)

How to train your product owner
How to train your product ownerHow to train your product owner
How to train your product owner
 
Technology & Faith: from Coding to Culture
Technology & Faith: from Coding to CultureTechnology & Faith: from Coding to Culture
Technology & Faith: from Coding to Culture
 
Agile Deep Learning
Agile Deep LearningAgile Deep Learning
Agile Deep Learning
 
Choosing a Job for the Right Reasons
Choosing a Job for the Right ReasonsChoosing a Job for the Right Reasons
Choosing a Job for the Right Reasons
 
Managing machine learning
Managing machine learningManaging machine learning
Managing machine learning
 
NLP in the Real World
NLP in the Real WorldNLP in the Real World
NLP in the Real World
 
System combination for HLT
System combination for HLTSystem combination for HLT
System combination for HLT
 
HltCon overview
HltCon overviewHltCon overview
HltCon overview
 
Simple fuzzy name matching in solr
Simple fuzzy name matching in solrSimple fuzzy name matching in solr
Simple fuzzy name matching in solr
 
Linguistic Considerations of Identity Resolution (2008)
Linguistic Considerations of Identity Resolution (2008)Linguistic Considerations of Identity Resolution (2008)
Linguistic Considerations of Identity Resolution (2008)
 
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
Moving beyond-entity-extraction-to-entity-resolution-david-murgatroyd-human-l...
 
From Research to Reality: Advances in HLT 2013
From Research to Reality: Advances in HLT 2013From Research to Reality: Advances in HLT 2013
From Research to Reality: Advances in HLT 2013
 

Recently uploaded

Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
名前 です男
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
DianaGray10
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
TIPNGVN2
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Aggregage
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 

Recently uploaded (20)

Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
 
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionGenerative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 

Managing Your Machine Learning Portfolio

  • 1. Managing Your Machine Learning Portfolio David Murgatroyd ( @dmurga) MassTLC ML Dev Day September 4, 2018 (please don’t sue me for copyright violation, Touchstone)
  • 5. Credit: Marta Mandile, DensityDesign Research Lab. CC BY-SA 4.0 Your product Your customer
  • 10. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 11. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 12. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 13. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 14. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 15. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 16. Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
  • 17. A Framework for the ML Product Lifecycle
  • 18. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Sure it’s possible? Sureit’svaluable?
  • 19. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Unknown to World Known to World Sure it’s possible? Known to Our Organization Sureit’svaluable?
  • 20. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 21. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 22. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 23. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 24. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 25. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 26. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 27. @dmurga Research Discovery Fundamental Delivery Applied Kinds of R&D Work: Examples Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 28. @dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization
  • 29. @dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML
  • 30. @dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product
  • 31. @dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization No New ML Proto ML Product M.V. ML Product
  • 32. @dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product No New ML Proto ML Product M.V. ML Product
  • 33. @dmurga Research Discovery Fundamental Delivery Applied ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
  • 34. @dmurga Research Delivery Applied Alternative ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product Proto ML Product Academic Results M.V. ML Product Industrial Lab Results Discovery Fundamental
  • 35. @dmurga Research Discovery Fundamental Delivery Applied Preferred ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
  • 36. Challenges for ML Portfolios 1. Misplaced modelling.
  • 37. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 38. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 39. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 40. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 41. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 42. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 43. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
  • 44. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
  • 45. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
  • 46. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project
  • 47. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Global: More data which changes relatively slowly. Local: Less data which changes relatively quickly.
  • 48. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Local Models Global Models
  • 49. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project Global Models Local Models
  • 50. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond More Global More Local ML Product Lifecycle ModelCombination
  • 51. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 52. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 53. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 54. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 55. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 56. Suggestions for ML Portfolios 1. Progress from global to local models.
  • 57. Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools.
  • 58. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 59. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 60. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 61. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
  • 62. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Where to get your ML Tools?
  • 63. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 64. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 65. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 66. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 67. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Where to get your ML Tools?
  • 68. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Share starting here Then here Where to get your ML Tools?
  • 69. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML
  • 70. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? No New ML Proto ML Product
  • 71. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product
  • 72. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product
  • 73. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools? M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond
  • 74. Suggestions for ML Portfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling.
  • 75. Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
  • 76. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 77. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 78. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 79. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project
  • 80. @dmurgaPeople by ProSymbols from the Noun Project Music by Iconnic from the Noun Project Two kinds of support: 1. Individual’s Skills 2. Team Structure
  • 81. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Skills Needed
  • 82. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 83. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 84. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 85. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 86. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager Skills Needed
  • 87. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML ML EngineerSkills Needed
  • 90. @dmurga Roles ‣ Applied ML Eng ‣ ML Tool Eng ‣ Core Researcher
  • 91. @dmurga Roles ‣ Applied ML Eng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Core Researcher
  • 92. @dmurga Roles ‣ Applied ML Eng ‣ Product ‣ Domain (logs, text, recs,...) ‣ Experiments ‣ Systems (BE, DE) ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
  • 93. @dmurga Roles # Needed‣ Applied ML Eng ‣ Product ‣ Domain ‣ Experiments ‣ Systems ‣ Algorithms ‣ ML Tool Eng ‣ Infrastructure ‣ Algorithms ‣ Core Researcher
  • 94. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Skills Needed
  • 95. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Applied ML Engineer ML Tools Engineer Core ML Researcher Skills Needed
  • 96. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated MLv.non-MLwork Team Structure
  • 97. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad ‣ More collaboration among MLers ‣ Harder to be product-aligned ‣ Possible deliverables: Models Systems Development effort MLv.non-MLwork Team Structure
  • 98. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad ‣ Less collaboration among MLers, so need to be senior ‣ Easier to be product-aligned ‣ MLers likely to do non-ML They need to be flexible PM needs to be skilled to cross-prioritize MLv.non-MLwork
  • 99. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream ‣ Allows more focus when ML progress is more valuable ‣ Prioritization is higher-level: across streams, not tasks ‣ Lots of variety in structure and sharing MLv.non-MLwork
  • 100. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad ‣ Even more focus for “step change” ‣ Need structure for alignment between siblings (eg., for online experimentation) ‣ Rare state since product discovery may be more valuable MLv.non-MLwork
  • 101. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
  • 102. Challenges for ML Portfolios 1. Misplaced modelling. 2. Crude tools. 3. Under-supported people.
  • 103. Suggestions for ML Portfolios 1. Progress from global to local models. 2. Strategically invest in shared tooling. 3. Equip individuals and their teams.
  • 104. @dmurga Research Discovery Fundamental Delivery Applied Preferred ML Product Lifecycle Development Experimental Hypothesized Unknown Known Sureit’svaluable? Unknown to World Known to World Sure it’s possible? Known to Our Organization Mature ML Product v2 and beyond No New ML Proto ML Product M.V. ML Product
  • 105. @dmurgaPeople by ProSymbols from the Noun Project World by Guilherme Furtado from the Noun Project M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Progress from more global to more local models. ML Product Lifecycle More Global More Local ModelCombination
  • 106. Stages of ML Workflow Offline Evaluation Feature Trans- formation Modeling Model Serving Online Evaluation Data Pre- processing Data set Validation Third Party Open Source or Vendor Your Team Your Organ- ization Where to get your ML Tools?
  • 107. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Drive ML (State-of-the-Art) Drive ML Inform ML Understand ML Designer or User Researcher Other Engineers Data Scientist ML Engineer Product Manager
  • 108. @dmurga M.V. ML Product No New ML Proto ML Product Mature ML Product v2 and beyond Integrated Isolated Central ML Squad Cross-functional Squad Separate ML Workstream Sibling Squad MLv.non-MLwork
  • 109.
  • 111. Thanks! Questions? David Murgatroyd (@dmurga) Suggestions: What kinds of ways can global and local models combine? What relation does org size have to tool source? How do you train Product Mgrs, Designers, etc., for ML product development? How does the ethics of ML Product Development fit? I hear you’re up for MassTLC’s ML in Action Award? What kind of ML roles are you hiring for in Boston? Hiring in Boston, NYC, London, and Stockholm!