The document summarizes a workshop on continuous delivery for machine learning models. It discusses the challenges of deploying machine learning systems according to continuous delivery principles due to the non-deterministic and unpredictable nature of ML models. The workshop covers exercises to set up ML pipelines using tools like DVC for version control and MLflow for experiment tracking to enable continuous delivery of ML models from development to production. It also demonstrates how to monitor models in production using the EFK stack.

1. CONTINUOUS DELIVERY
FOR MACHINE LEARNING
Workshop | 27th November | ThoughtWorks London

2. 2
Structure of Today’s Workshop
- INTRODUCTION TO THE TOPIC
- EXERCISE 1: SETUP
- EXERCISE 2: DEPLOYMENT PIPELINE
- BREAK
- EXERCISE 3: ML PIPELINE
- DEMO: TRACKING EXPERIMENTS & MODEL MONITORING
©ThoughtWorks 2019

3. 3
Facilitators
©ThoughtWorks 2019
Danilo Sato,
Principal Technology
Consultant
Zehra Kavasoglu,
Data Scientist
James Green,
Data Engineer
Jade Forsberg,
Software Developer

4. 7000+ technologists with 43 offices in 14 countries
Partner for technology driven business transformation
©ThoughtWorks 2019
join.thoughtworks.com

5. #1
in Agile and
Continuous Delivery
100+
books written
©ThoughtWorks 2019

6. ©ThoughtWorks 2019

7. TECHNIQUES
Continuous delivery
for machine
learning (CD4ML)
models
#9
TRIAL
©ThoughtWorks 2019
9
#9

8. CONTINUOUS INTELLIGENCE CYCLE
©ThoughtWorks 2019 8

9. CD4ML isn’t a technology or a
tool; it is a practice and a set of
principles. Quality is built into
software and improvement is
always possible.
But machine learning systems
have unique challenges; unlike
deterministic software, it is
difficult—or impossible—to
understand the behavior of
data-driven intelligent systems.
This poses a huge challenge
when it comes to deploying
machine learning systems in
accordance with CD principles.
9
PRODUCTIONIZING ML IS HARD
Production systems should be:
● Reproducible
● Testable
● Auditable
● Continuously Improving
HOW DO WE APPLY DECADES OF SOFTWARE DELIVERY EXPERIENCE TO
INTELLIGENT SYSTEMS?
©ThoughtWorks 2019

10. CD4ML isn’t a technology or a
tool; it is a practice and a set of
principles. Quality is built into
software and improvement is
always possible.
But machine learning systems
have unique challenges; unlike
deterministic software, it is
difficult—or impossible—to
understand the behavior of
data-driven intelligent systems.
This poses a huge challenge
when it comes to deploying
machine learning systems in
accordance with CD principles.
10
PRODUCTIONIZING ML IS HARD
Production systems should be:
● Reproducible
● Testable
● Auditable
● Continuously Improving
Machine Learning is:
● Non-deterministic
● Hard to test
● Hard to explain
● Hard to improve
HOW DO WE APPLY DECADES OF SOFTWARE DELIVERY EXPERIENCE TO
INTELLIGENT SYSTEMS?
©ThoughtWorks 2019

11. MANY SOURCES OF CHANGE
11©ThoughtWorks 2019
ModelData Code
+ +
Schema
Sampling over Time
Volume
Algorithms
More Training
Experiments
Business Needs
Bug Fixes
Configuration

12. “Continuous Delivery is the ability to get changes of
all types — including new features, configuration
changes, bug fixes and experiments — into
production, or into the hands of users, safely and
quickly in a sustainable way.”
- Jez Humble & Dave Farley
12©ThoughtWorks 2019

13. ©ThoughtWorks 2019
PRINCIPLES OF CONTINUOUS DELIVERY
13
Create a Repeatable, Reliable Process for Releasing
Software
Automate Almost Everything
Build Quality In
Work in Small Batches
Keep Everything in Source Control
Done Means “Released”
Improve Continuously

14. WHAT DO WE NEED IN OUR STACK?
14
Doing CD with Machine Learning is still a hard problem
MODEL
PERFORMANCE
ASSESSMENT
TRACKING
VERSION
CONTROL AND
ARTIFACT
REPOSITORIES
©ThoughtWorks 2019
MODEL
MONITORING
AND
OBSERVABILITY
DISCOVERABLE
AND
ACCESSIBLE
DATA
CONTINUOUS
DELIVERY
ORCHESTRATION
TO COMBINE
PIPELINES
INFRASTRUCTURE
FOR MULTIPLE
ENVIRONMENTS
AND
EXPERIMENTS

15. training
data
Model Building
candidate
models
training
code
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 15

16. training
data
Model Building
Model Evaluation
and
Experimentation
candidate
models
metrics
training
code
chosen
model
test
data
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 16

17. training
data
Model Building
Model Evaluation
and
Experimentation
Productionize
Model
candidate
models
metrics
training
code
chosen
model
productionized
model
test
data
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 17

18. training
data
Model Building
Model Evaluation
and
Experimentation
Productionize
Model
Testing
candidate
models
metrics
training
code
chosen
model
productionized
model
test
code
model
test
data
test
data
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 18

19. training
data
Model Building Deployment
Model Evaluation
and
Experimentation
Productionize
Model
Testing
candidate
models
metrics
training
code
chosen
model
productionized
model
test
code
model
test
data
test
data
application
code
code and
model in
production
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 19

20. training
data
Model Building Deployment
Model Evaluation
and
Experimentation
Productionize
Model
Testing
candidate
models
metrics
Monitoring and
Observability
training
code
chosen
model
productionized
model
test
code
model
test
data
test
data
application
code
code and
model in
production
production
data
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 20

21. training
data
Model Building Deployment
Model Evaluation
and
Experimentation
Productionize
Model
Testing
candidate
models
metrics
Monitoring and
Observability
training
code
chosen
model
productionized
model
test
code
model
test
data
test
data
application
code
code and
model in
production
production
data
codemodeldata
PUTTING EVERYTHING TOGETHER
©ThoughtWorks 2019 21

22. WHAT WE WILL USE IN THIS WORKSHOP
22
There are many options for tools and technologies to implement CD4ML
©ThoughtWorks 2019

23. THE MACHINE
LEARNING PROBLEM
WE ARE EXPLORING
TODAY
©ThoughtWorks 2019 23

24. A REAL BUSINESS PROBLEM
RETAIL / SUPPLY CHAIN
Loss of sales, opportunity cost, stock waste, discounting
REQUIRES
Accurate Demand Forecasting
TYPICAL CHALLENGES
→ Predictions Are Inaccurate
→ Development Takes A Long Time
→ Difficult To Adapt To Market Change Pace
24©ThoughtWorks 2019 24

25. SALES FORECASTING
FOR GROCERY
RETAILER
● 4,000 items
● 50 stores
● 125,000,000 sales transactions
● 4.5 years of data
Make predictions based on data from:
25
TASK:
Predict how many of each
product will be purchased
in each store on a given
date
©ThoughtWorks 2019

26. THE SIMPLIFIED WEB APPLICATION
As a buyer, I want to be able to choose a product and
predict how many units the product will sell at a future date.
26©ThoughtWorks 2019

27. EXERCISE 1: SETUP
https://github.com/ThoughtWorksInc/continuous-intelligence-workshop
27
● Click on instructions → 1-setup.md
● Follow the steps to setup your local development
environment
● Use your assigned User ID
©ThoughtWorks 2019 27

28. DEPLOYMENT PIPELINE
Automates the process of building, testing, and deploying applications to production
28
Application code in
version control
repository
Container image
as deployment
artifact
Deploy container
to production
servers
©ThoughtWorks 2019

29. 29©ThoughtWorks 2019
An Open Source Continuous Delivery server to model and visualise complex workflows

30. Pipeline Group
ANATOMY OF A GOCD PIPELINE
30©ThoughtWorks 2019

31. EXERCISE 2:
DEPLOYMENT
PIPELINE
https://github.com/ThoughtWorksInc/continuous-intelligence-workshop
31
● Click on instructions → 2-deployment-pipeline.md
● Follow the steps to setup your deployment pipeline
● GoCD URL: https://gocd.cd4ml.net
©ThoughtWorks 2019 31

32. BUT NOW WHAT?
● How do we retrain the model more often?
● How to deploy the retrained model to production?
● How to make sure we don’t break anything when
deploying?
● How to make sure that our modeling approach or
parameterization is still the best fit for the data?
● How to monitor our model “in the wild”?
Once your model is in production...
32©ThoughtWorks 2019 32

33. BASIC DATA SCIENCE WORKFLOW
33
Gather data and
extract features
Separate into
training and
validation sets
Train model and
evaluate
performance
©ThoughtWorks 2019

34. SALES FORECAST MODEL TRAINING PROCESS
34©ThoughtWorks 2019
download_data.py
Data splitter.py
Training Data
Validation Data
decision_tree.py
model.pkl
metrics.json
evaluation.py

35. CHALLENGE 1: THESE ARE LARGE FILES
35©ThoughtWorks 2019
download_data.py
Data splitter.py
Training Data
Validation Data
decision_tree.py
model.pkl
metrics.json
evaluation.py

36. CHALLENGE 2: AD-HOC MULTI-STEP PROCESS
36©ThoughtWorks 2019
download_data.py
Data splitter.py
Training Data
Validation Data
decision_tree.py
model.pkl
metrics.json
evaluation.py

37. ● dvc is git porcelain for storing large files using cloud storage
● dvc connects model training steps to create reproducible workflows
SOLUTION: dvc
data science version control
37
master
change-max-depth
try-random-forest
model.pkl
decision_tree.py
model.pkl.dvc
©ThoughtWorks 2019

38. ANATOMY OF A DVC COMMAND
This runs a command and creates a .dvc file. The dvc file points to the
dependencies. The output files are versioned and stored in the cloud by running
dvc push.
When you use the output files (store47-2016.csv) as dependencies for the next
step, a pipeline is automatically created.
You can re-execute an entire pipeline with one command: dvc repro
38
dvc run -d src/download_data.py
-o data/raw/store47-2016.csv python src/download_data.py
©ThoughtWorks 2019 38

39. EXERCISE 3: MACHINE LEARNING
PIPELINE
https://github.com/ThoughtWorksInc/continuous-intelligence-workshop
39
● Click on instructions → 3-machine-learning-
pipeline.md
● Follow the steps on your local development environment
and in GoCD to create your Machine Learning pipeline
©ThoughtWorks 2019 39

40. HOW DO WE TRACK
EXPERIMENTS?
● Which experiments and hypothesis are being explored?
● Which algorithms are being used in each experiment?
● Which version of the code was used?
● How long does it take to run each experiment?
● What parameter and hyperparameters were used?
● How fast are my models learning?
● How do we compare results from different runs?
We need to track the scientific process and evaluate our models:
40©ThoughtWorks 2019 40

41. 41©ThoughtWorks 2019
An Open Source platform for managing end-to-end machine learning lifecycle

42. DEMO: TRACKING EXPERIMENTS
42
● MLflow URL: https://mlflow.cd4ml.net
©ThoughtWorks 2019 42

43. HOW TO LEARN CONTINUOUSLY?
● Track model usage
● Track model inputs to find training-serving skew
● Track model outputs
● Track model interpretability outputs to identify potential
bias or overfit
● Track model fairness to understand how it behaves
against dimensions that could introduce unfair bias
We need to capture production data to improve our models:
43©ThoughtWorks 2019 43

44. EFK STACK
Monitoring and Observability infrastructure
44
Open Source data
collector for unified
logging
Open Source Search
Engine
Open Source web UI
to explore and
visualise data
©ThoughtWorks 2019

45. 45©ThoughtWorks 2019
An Open Source UI that makes it easy to explore and visualise the data index in Elasticsearch

46. DEMO: MODEL MONITORING
46
● Kibana URL: https://kibana.cd4ml.net
©ThoughtWorks 2019 46

47. SUMMARY - WHAT
HAVE WE LEARNED?
©ThoughtWorks 2019 47

48. CD4ML
● Proper data/model versioning tools enable reproducible work to be done in
parallel.
● No need to maintain complex data processing/model training scripts.
● We can then put data science work into a Continuous Delivery workflow.
● Result: Continuous, on-demand AI development and deployment,
from research to production, in an automated fashion.
● Benefit: production AI systems that are always as smart as your data
science team.
48©ThoughtWorks 2019

49. QUESTIONS
©ThoughtWorks 2019 49

50. THANK YOU
Danilo Sato dsato@thoughtworks.com
James Green jagreen@thoughtworks.com
Jade Forsberg jade.forsberg@thoughtworks.com
Zehra Kavasoglu zehra@thoughtworks.com