Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Delivery for Machine Learning

227 views

Published on

Your Data Scientists or Machine Learning experts have developed a machine learning model which runs perfectly in your notebook? Now you want to deploy it into corporate IT to let it run “in the wild”. And a bunch of new problems comes up: How to integrate the work of your data scientists and machine learning experts into the development processes like CI/CD of your corporate IT? How to prevent a “throw-it-over-the-fence” mentality? How to test, monitor and continuously improve your machine learning application “in the wild”?

In a compact workshop, we will discuss the new challenges of integrating machine learning approaches in modern IT development processes and demonstrate our “Continuous Delivery for Machine Learning” (CD4ML) methodology with some live coding examples.

Published in: Technology
  • Be the first to comment

Continuous Delivery for Machine Learning

  1. 1. CONTINUOUS DELIVERY FOR MACHINE LEARNING Workshop | 27th November | ThoughtWorks London
  2. 2. 2 Structure of Today’s Workshop - INTRODUCTION TO THE TOPIC - EXERCISE 1: SETUP - EXERCISE 2: DEPLOYMENT PIPELINE - BREAK - EXERCISE 3: ML PIPELINE - DEMO: TRACKING EXPERIMENTS & MODEL MONITORING ©ThoughtWorks 2019
  3. 3. 3 Facilitators ©ThoughtWorks 2019 Danilo Sato, Principal Technology Consultant Zehra Kavasoglu, Data Scientist James Green, Data Engineer Jade Forsberg, Software Developer
  4. 4. 7000+ technologists with 43 offices in 14 countries Partner for technology driven business transformation ©ThoughtWorks 2019 join.thoughtworks.com
  5. 5. #1 in Agile and Continuous Delivery 100+ books written ©ThoughtWorks 2019
  6. 6. ©ThoughtWorks 2019
  7. 7. TECHNIQUES Continuous delivery for machine learning (CD4ML) models #9 TRIAL ©ThoughtWorks 2019 9 #9
  8. 8. CONTINUOUS INTELLIGENCE CYCLE ©ThoughtWorks 2019 8
  9. 9. CD4ML isn’t a technology or a tool; it is a practice and a set of principles. Quality is built into software and improvement is always possible. But machine learning systems have unique challenges; unlike deterministic software, it is difficult—or impossible—to understand the behavior of data-driven intelligent systems. This poses a huge challenge when it comes to deploying machine learning systems in accordance with CD principles. 9 PRODUCTIONIZING ML IS HARD Production systems should be: ● Reproducible ● Testable ● Auditable ● Continuously Improving HOW DO WE APPLY DECADES OF SOFTWARE DELIVERY EXPERIENCE TO INTELLIGENT SYSTEMS? ©ThoughtWorks 2019
  10. 10. CD4ML isn’t a technology or a tool; it is a practice and a set of principles. Quality is built into software and improvement is always possible. But machine learning systems have unique challenges; unlike deterministic software, it is difficult—or impossible—to understand the behavior of data-driven intelligent systems. This poses a huge challenge when it comes to deploying machine learning systems in accordance with CD principles. 10 PRODUCTIONIZING ML IS HARD Production systems should be: ● Reproducible ● Testable ● Auditable ● Continuously Improving Machine Learning is: ● Non-deterministic ● Hard to test ● Hard to explain ● Hard to improve HOW DO WE APPLY DECADES OF SOFTWARE DELIVERY EXPERIENCE TO INTELLIGENT SYSTEMS? ©ThoughtWorks 2019
  11. 11. MANY SOURCES OF CHANGE 11©ThoughtWorks 2019 ModelData Code + + Schema Sampling over Time Volume Algorithms More Training Experiments Business Needs Bug Fixes Configuration
  12. 12. “Continuous Delivery is the ability to get changes of all types — including new features, configuration changes, bug fixes and experiments — into production, or into the hands of users, safely and quickly in a sustainable way.” - Jez Humble & Dave Farley 12©ThoughtWorks 2019
  13. 13. ©ThoughtWorks 2019 PRINCIPLES OF CONTINUOUS DELIVERY 13 Create a Repeatable, Reliable Process for Releasing Software Automate Almost Everything Build Quality In Work in Small Batches Keep Everything in Source Control Done Means “Released” Improve Continuously
  14. 14. WHAT DO WE NEED IN OUR STACK? 14 Doing CD with Machine Learning is still a hard problem MODEL PERFORMANCE ASSESSMENT TRACKING VERSION CONTROL AND ARTIFACT REPOSITORIES ©ThoughtWorks 2019 MODEL MONITORING AND OBSERVABILITY DISCOVERABLE AND ACCESSIBLE DATA CONTINUOUS DELIVERY ORCHESTRATION TO COMBINE PIPELINES INFRASTRUCTURE FOR MULTIPLE ENVIRONMENTS AND EXPERIMENTS
  15. 15. training data Model Building candidate models training code codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 15
  16. 16. training data Model Building Model Evaluation and Experimentation candidate models metrics training code chosen model test data codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 16
  17. 17. training data Model Building Model Evaluation and Experimentation Productionize Model candidate models metrics training code chosen model productionized model test data codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 17
  18. 18. training data Model Building Model Evaluation and Experimentation Productionize Model Testing candidate models metrics training code chosen model productionized model test code model test data test data codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 18
  19. 19. training data Model Building Deployment Model Evaluation and Experimentation Productionize Model Testing candidate models metrics training code chosen model productionized model test code model test data test data application code code and model in production codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 19
  20. 20. training data Model Building Deployment Model Evaluation and Experimentation Productionize Model Testing candidate models metrics Monitoring and Observability training code chosen model productionized model test code model test data test data application code code and model in production production data codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 20
  21. 21. training data Model Building Deployment Model Evaluation and Experimentation Productionize Model Testing candidate models metrics Monitoring and Observability training code chosen model productionized model test code model test data test data application code code and model in production production data codemodeldata PUTTING EVERYTHING TOGETHER ©ThoughtWorks 2019 21
  22. 22. WHAT WE WILL USE IN THIS WORKSHOP 22 There are many options for tools and technologies to implement CD4ML ©ThoughtWorks 2019
  23. 23. THE MACHINE LEARNING PROBLEM WE ARE EXPLORING TODAY ©ThoughtWorks 2019 23
  24. 24. A REAL BUSINESS PROBLEM RETAIL / SUPPLY CHAIN Loss of sales, opportunity cost, stock waste, discounting REQUIRES Accurate Demand Forecasting TYPICAL CHALLENGES → Predictions Are Inaccurate → Development Takes A Long Time → Difficult To Adapt To Market Change Pace 24©ThoughtWorks 2019 24
  25. 25. SALES FORECASTING FOR GROCERY RETAILER ● 4,000 items ● 50 stores ● 125,000,000 sales transactions ● 4.5 years of data Make predictions based on data from: 25 TASK: Predict how many of each product will be purchased in each store on a given date ©ThoughtWorks 2019
  26. 26. THE SIMPLIFIED WEB APPLICATION As a buyer, I want to be able to choose a product and predict how many units the product will sell at a future date. 26©ThoughtWorks 2019
  27. 27. EXERCISE 1: SETUP https://github.com/ThoughtWorksInc/continuous-intelligence-workshop 27 ● Click on instructions → 1-setup.md ● Follow the steps to setup your local development environment ● Use your assigned User ID ©ThoughtWorks 2019 27
  28. 28. DEPLOYMENT PIPELINE Automates the process of building, testing, and deploying applications to production 28 Application code in version control repository Container image as deployment artifact Deploy container to production servers ©ThoughtWorks 2019
  29. 29. 29©ThoughtWorks 2019 An Open Source Continuous Delivery server to model and visualise complex workflows
  30. 30. Pipeline Group ANATOMY OF A GOCD PIPELINE 30©ThoughtWorks 2019
  31. 31. EXERCISE 2: DEPLOYMENT PIPELINE https://github.com/ThoughtWorksInc/continuous-intelligence-workshop 31 ● Click on instructions → 2-deployment-pipeline.md ● Follow the steps to setup your deployment pipeline ● GoCD URL: https://gocd.cd4ml.net ©ThoughtWorks 2019 31
  32. 32. BUT NOW WHAT? ● How do we retrain the model more often? ● How to deploy the retrained model to production? ● How to make sure we don’t break anything when deploying? ● How to make sure that our modeling approach or parameterization is still the best fit for the data? ● How to monitor our model “in the wild”? Once your model is in production... 32©ThoughtWorks 2019 32
  33. 33. BASIC DATA SCIENCE WORKFLOW 33 Gather data and extract features Separate into training and validation sets Train model and evaluate performance ©ThoughtWorks 2019
  34. 34. SALES FORECAST MODEL TRAINING PROCESS 34©ThoughtWorks 2019 download_data.py Data splitter.py Training Data Validation Data decision_tree.py model.pkl metrics.json evaluation.py
  35. 35. CHALLENGE 1: THESE ARE LARGE FILES 35©ThoughtWorks 2019 download_data.py Data splitter.py Training Data Validation Data decision_tree.py model.pkl metrics.json evaluation.py
  36. 36. CHALLENGE 2: AD-HOC MULTI-STEP PROCESS 36©ThoughtWorks 2019 download_data.py Data splitter.py Training Data Validation Data decision_tree.py model.pkl metrics.json evaluation.py
  37. 37. ● dvc is git porcelain for storing large files using cloud storage ● dvc connects model training steps to create reproducible workflows SOLUTION: dvc data science version control 37 master change-max-depth try-random-forest model.pkl decision_tree.py model.pkl.dvc ©ThoughtWorks 2019
  38. 38. ANATOMY OF A DVC COMMAND This runs a command and creates a .dvc file. The dvc file points to the dependencies. The output files are versioned and stored in the cloud by running dvc push. When you use the output files (store47-2016.csv) as dependencies for the next step, a pipeline is automatically created. You can re-execute an entire pipeline with one command: dvc repro 38 dvc run -d src/download_data.py -o data/raw/store47-2016.csv python src/download_data.py ©ThoughtWorks 2019 38
  39. 39. EXERCISE 3: MACHINE LEARNING PIPELINE https://github.com/ThoughtWorksInc/continuous-intelligence-workshop 39 ● Click on instructions → 3-machine-learning- pipeline.md ● Follow the steps on your local development environment and in GoCD to create your Machine Learning pipeline ©ThoughtWorks 2019 39
  40. 40. HOW DO WE TRACK EXPERIMENTS? ● Which experiments and hypothesis are being explored? ● Which algorithms are being used in each experiment? ● Which version of the code was used? ● How long does it take to run each experiment? ● What parameter and hyperparameters were used? ● How fast are my models learning? ● How do we compare results from different runs? We need to track the scientific process and evaluate our models: 40©ThoughtWorks 2019 40
  41. 41. 41©ThoughtWorks 2019 An Open Source platform for managing end-to-end machine learning lifecycle
  42. 42. DEMO: TRACKING EXPERIMENTS 42 ● MLflow URL: https://mlflow.cd4ml.net ©ThoughtWorks 2019 42
  43. 43. HOW TO LEARN CONTINUOUSLY? ● Track model usage ● Track model inputs to find training-serving skew ● Track model outputs ● Track model interpretability outputs to identify potential bias or overfit ● Track model fairness to understand how it behaves against dimensions that could introduce unfair bias We need to capture production data to improve our models: 43©ThoughtWorks 2019 43
  44. 44. EFK STACK Monitoring and Observability infrastructure 44 Open Source data collector for unified logging Open Source Search Engine Open Source web UI to explore and visualise data ©ThoughtWorks 2019
  45. 45. 45©ThoughtWorks 2019 An Open Source UI that makes it easy to explore and visualise the data index in Elasticsearch
  46. 46. DEMO: MODEL MONITORING 46 ● Kibana URL: https://kibana.cd4ml.net ©ThoughtWorks 2019 46
  47. 47. SUMMARY - WHAT HAVE WE LEARNED? ©ThoughtWorks 2019 47
  48. 48. CD4ML ● Proper data/model versioning tools enable reproducible work to be done in parallel. ● No need to maintain complex data processing/model training scripts. ● We can then put data science work into a Continuous Delivery workflow. ● Result: Continuous, on-demand AI development and deployment, from research to production, in an automated fashion. ● Benefit: production AI systems that are always as smart as your data science team. 48©ThoughtWorks 2019
  49. 49. QUESTIONS ©ThoughtWorks 2019 49
  50. 50. THANK YOU Danilo Sato dsato@thoughtworks.com James Green jagreen@thoughtworks.com Jade Forsberg jade.forsberg@thoughtworks.com Zehra Kavasoglu zehra@thoughtworks.com

×