The Ethos of Process in ML
Getting Comfortable with Uncomfortable (Again)

process
noun
A series of actions or steps taken in order to
achieve a particular end

Image adapted from The ML Test Score: A Rubric for ML
ProductionReadinessand Technical Debt Reduction

Image credit: InterpretableMachine Learning Book,
https://christophm.github.io/interpretable-ml-book/terminology.html

Image Credit: Jimmy Whitaker,Pachyderm, https://www.pachyderm.com/blog/completing-
machine-learning-loop/

Image adapted from The ML Test Score: A Rubric for ML
ProductionReadinessand Technical Debt Reduction

Tech is easy, process is hard

Tech is easy, process is hard
What are the things we may overlook in our approach to process?

Get comfortable with uncomfortable

Doubt is an
uncomfortable
condition, but
certainty is a
ridiculous one
VOLTAIRE

Models are iterative, so should be the approach to process

Stop trying to justify Data Science $$$

Image Credit: giphy.com

Calculating ROI in ML is hard

Calculating ROI in ML is hard
Too many cost estimation
unknowns
Complex if
product already
exists
Justifying the
experiments that
never get deployed
Investment across a huge range
of the data suite, from
infrastructure to tools, through
to governance and insight
Gains (added value) can come
in many forms, not
necessarily easy to
standardise

Attribution problem
These calculations are complicated because the value isn’t all in one
number, it’s often spread across multiple departments, teams,
business units and programmes of work.
Measuring ROI for a data science project can end up as a project in
itself, which is often difficult to justify.

Image Credit: Andrew Ng,Introductionto Machine Learning in Production,Coursera
Tell compelling ML stories mapped to business metrics

Make maintenance an even bigger part
of your Business As Usual

Steps may get done, but a model will never be
Image Credit: BusinessInsider, https://www.businessinsider.com/how-netflix-has-looked-
over-the-years-2016-4?

Steps may get done, but a model will never be
Be confident and make regular time to:
● Do qualitative and quantitative user research on your models and outputs
● Kill off/deprecate projects that are no longer in use
● Similarly for projects with high maintenance resource/cost and low user
consumption
● Bring projects from MVP to mature
● Reviewing all in production projects, assessing ethics, governance, bias, data
divergence and edge cases, both through automated testing and manual
reliability evaluation

Address your own needs

Image Credit: https://tristramshepard.wordpress.com/2019/01/10/mr-glibblys-
square-world/square-peg-round-hole-gif-gif/

Ensure you have the right people

Data Scientist ML Engineer MLOps Engineer
Data/Analytics Engineer ML Scientist

The “Data Translator”
The “Data Visualiser”
The “Data Explorer”
The “Data Modeler”
The “Data Product
Owner”
Data Scientist ML Engineer MLOps Engineer
Data/Analytics Engineer ML Scientist
The “Data Storyteller”

The “Data Translator”
The “Data Visualiser”
The “Data Explorer”
The “Data Modeler”
The “Data Product
Owner”
Data Scientist ML Engineer MLOps Engineer
Data/Analytics Engineer ML Scientist
The “Data Storyteller”

Delivery Manager
Product Manager
Change Manager
Business Analyst
User Research
Data Scientist ML Engineer MLOps Engineer
Data/Analytics Engineer ML Scientist
User Design &
Experience
Governance &
Strategy
Solution Architect

The choice you made is less important
than having made the choice

Ethics and governance are not steps in
the process, but are fundamental to it

Get ok with people not getting ML

Image adapted from Andy
Scherpenberg,What Companies
ThinkAI Is

But… Organisational alignment is still vital
Image Credit: geek-and-poke.com

process
noun
A series of actions or steps taken in order to
achieve a particular end

process
noun
A series of actions or steps taken in order to
achieve a particular end

process
noun
A series of actions or steps taken in order to
achieve a particular end

process
noun
A series of actions or steps taken in order to
achieve a particular end
outcome

We focus so much on building good
outcomes in Machine Learning.
If we focus more on continuing to
build good process, our outcomes
are more likely to be successful.

Thank you!
Please connect with me:
@LK_Fitzpatrick
Leanne Kim Fitzpatrick