Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Continuous Deployment for Deep Learning Slide 1 Continuous Deployment for Deep Learning Slide 2 Continuous Deployment for Deep Learning Slide 3 Continuous Deployment for Deep Learning Slide 4 Continuous Deployment for Deep Learning Slide 5 Continuous Deployment for Deep Learning Slide 6 Continuous Deployment for Deep Learning Slide 7 Continuous Deployment for Deep Learning Slide 8 Continuous Deployment for Deep Learning Slide 9 Continuous Deployment for Deep Learning Slide 10 Continuous Deployment for Deep Learning Slide 11 Continuous Deployment for Deep Learning Slide 12 Continuous Deployment for Deep Learning Slide 13 Continuous Deployment for Deep Learning Slide 14 Continuous Deployment for Deep Learning Slide 15 Continuous Deployment for Deep Learning Slide 16 Continuous Deployment for Deep Learning Slide 17 Continuous Deployment for Deep Learning Slide 18 Continuous Deployment for Deep Learning Slide 19 Continuous Deployment for Deep Learning Slide 20 Continuous Deployment for Deep Learning Slide 21 Continuous Deployment for Deep Learning Slide 22 Continuous Deployment for Deep Learning Slide 23 Continuous Deployment for Deep Learning Slide 24 Continuous Deployment for Deep Learning Slide 25 Continuous Deployment for Deep Learning Slide 26 Continuous Deployment for Deep Learning Slide 27 Continuous Deployment for Deep Learning Slide 28 Continuous Deployment for Deep Learning Slide 29 Continuous Deployment for Deep Learning Slide 30 Continuous Deployment for Deep Learning Slide 31 Continuous Deployment for Deep Learning Slide 32 Continuous Deployment for Deep Learning Slide 33 Continuous Deployment for Deep Learning Slide 34 Continuous Deployment for Deep Learning Slide 35 Continuous Deployment for Deep Learning Slide 36 Continuous Deployment for Deep Learning Slide 37 Continuous Deployment for Deep Learning Slide 38 Continuous Deployment for Deep Learning Slide 39 Continuous Deployment for Deep Learning Slide 40 Continuous Deployment for Deep Learning Slide 41 Continuous Deployment for Deep Learning Slide 42 Continuous Deployment for Deep Learning Slide 43
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

Continuous Deployment for Deep Learning

Download to read offline

Continuous integration and deployment has become an increasingly standard and common practice in software development. However, doing this for machine learning models and applications introduces many challenges. Not only do we need to account for standard code quality and integration testing, but how do we best account for changes in model performance metrics coming from changes to code, deployment framework or mechanism, pre- and post-processing steps, changes in data, not to mention the core deep learning model itself?

In addition, deep learning presents particular challenges:
* model sizes are often extremely large and take significant time and resources to train
* models are often more difficult to understand and interpret making it more difficult to debug issues
* inputs to deep learning are often very different from the tabular data involved in most ‘traditional machine learning’ models
* model formats, frameworks and the state-of-the art models and architectures themselves are changing extremely rapidly
* usually many disparate tools are combined to create the full end-to-end pipeline for training and deployment, making it trickier to plug together these components and track down issues.

We also need to take into account the impact of changes on wider aspects such as model bias, fairness, robustness and explainability. And we need to track all of this over time and in a standard, repeatable manner. This talk explores best practices for handling these myriad challenges to create a standardized, automated, repeatable pipeline for continuous deployment of deep learning models and pipelines. I will illustrate this through the work we are undertaking within the free and open-source IBM Model Asset eXchange.

Continuous Deployment for Deep Learning

  1. 1. Continuous Deployment for Deep Learning — Nick Pentreath Principal Engineer @MLnick
  2. 2. About IBM Developer / © 2019 IBM Corporation – @MLnick on Twitter & Github – Principal Engineer, IBM CODAIT (Center for Open-Source Data & AI Technologies) – Machine Learning & AI – Apache Spark committer & PMC – Author of Machine Learning with Spark – Various conferences & meetups 2
  3. 3. CODAIT Improving the Enterprise AI Lifecycle in Open Source Center for Open Source Data & AI Technologies IBM Developer / © 2019 IBM Corporation 3 CODAIT aims to make AI solutions dramatically easier to create, deploy, and manage in the enterprise. We contribute to and advocate for the open-source technologies that are foundational to IBM’s AI offerings. 30+ open-source developers!
  4. 4. Agenda IBM Developer / © 2019 IBM Corporation – Overview of Continuous Integration & Deployment – The Machine Learning Workflow – How is CI/CD for ML Different & Challenges – Model Asset Exchange – Conclusion 4
  5. 5. Continuous Integration & Deployment IBM Developer / © 2019 IBM Corporation 5
  6. 6. Testing IBM Developer / © 2019 IBM Corporation 6 App Data Store Public Front-end Services Back-end Services Service Remote Service Remote Service
  7. 7. Testing IBM Developer / © 2019 IBM Corporation 7 App Data Store Public Front-end Services Back-end Services Service Remote Service Remote Service Unit testing
  8. 8. Testing IBM Developer / © 2019 IBM Corporation 8 App Data Store Public Front-end Services Back-end Services Service Remote Service Remote Service Integration testing
  9. 9. Testing IBM Developer / © 2019 IBM Corporation 9 App Data Store Public Front-end Services Back-end Services Service Remote Service Remote Service Post-deploy / Acceptance testing
  10. 10. Development Process IBM Developer / © 2019 IBM Corporation 10 Build Test Merge Deploy Dependencies
  11. 11. Starting Point IBM Developer / © 2019 IBM Corporation 11 Build Test Merge Dependencies Manual Automated Deploy
  12. 12. Continuous Integration IBM Developer / © 2019 IBM Corporation 12 Build Test Merge Dependencies Manual Automated Deploy
  13. 13. Continuous Deployment IBM Developer / © 2019 IBM Corporation 13 Build Test Merge Dependencies Manual Automated Deploy
  14. 14. Source of changes IBM Developer / © 2019 IBM Corporation 14 Build Test Merge Deploy Dependencies Changes come from: • Our code • Internal dependencies • 3rd party dependencies
  15. 15. The Machine Learning Workflow IBM Developer / © 2019 IBM Corporation 15
  16. 16. Perception IBM Developer / © 2019 IBM Corporation 16
  17. 17. In reality the workflow spans teams … IBM Developer / © 2019 IBM Corporation 17
  18. 18. … and tools … IBM Developer / © 2019 IBM Corporation 18
  19. 19. … and is a small (but critical!) piece of the puzzle *Source: Hidden Technical Debt in Machine Learning Systems IBM Developer / © 2019 IBM Corporation 19
  20. 20. What is a “model”? IBM Developer / © 2019 IBM Corporation 20
  21. 21. Deep learning pipeline IBM Developer / © 2019 IBM Corporation 21 beagle: 0.82 basset: 0.09 bluetick: 0.07 ... Input image Image pre-processing Prediction Decode image Resize Normalization Convert types / format Inference Post-processing [0.2, 0.3, … ] (label, prob) Sort Label map PIL, OpenCV, tf.image, … Custom Python * Logos trademarks of their respective projects
  22. 22. Pipelines, not Models – Deploying (and testing) just the model part of the workflow is not enough – Entire pipeline must be taken into account • Data transforms • Feature extraction & pre-processing • DL / ML model • Prediction transformation – Even ETL is part of the pipeline! – Pipelines in frameworks • scikit-learn • Spark ML pipelines • TensorFlow Transform • pipeliner (R) IBM Developer / © 2019 IBM Corporation 22
  23. 23. Continuous Integration for Machine Learning IBM Developer / © 2019 IBM Corporation 23
  24. 24. Source of changes IBM Developer / © 2019 IBM Corporation 24 Build Test Merge Deploy Dependencies Changes come from: • Our code • Internal dependencies • 3rd party dependencies • Data • Model • Time Data Model
  25. 25. Data IBM Developer / © 2019 IBM Corporation – Data types • Images, video, audio, raw text, unstructured – Size – Schemas * Logos trademarks of their respective projects 25
  26. 26. Models IBM Developer / © 2019 IBM Corporation – Size of models – Resource requirements – Hardware • CPU, GPU, TPU, Mobile, Edge – Need to manage and bridge many different languages, frameworks – Formats – State of the art is changing very rapidly * Logos trademarks of their respective projects 26
  27. 27. Monitoring & Feedback over Time IBM Developer / © 2019 IBM Corporation 27
  28. 28. Monitoring Performance Business Monitoring Software Traditional software monitoring Latency, throughput, resource usage, etc Model performance metrics Traditional ML evaluation measures (accuracy, prediction error, AUC etc) Business metrics Impact of predictions on business outcomes • Additional revenue - e.g. uplift from recommender • Cost savings – e.g. value of fraud prevented • Metrics implicitly influencing these – e.g. user engagement IBM Developer / © 2019 IBM Corporation
  29. 29. Feedback Data Transform TrainDeploy Feedback Adapt An intelligent system must automatically learn from & adapt to the world around it Continual learning Retraining, online learning, reinforcement learning Feedback loops Explicit: models create or directly influence their own training data Implicit: predictions influence behavior in longer-term or indirect ways Humans in the loop IBM Developer / © 2019 IBM Corporation
  30. 30. AI Fairness, Transparency & Security 30IBM Developer / © 2019 IBM Corporation
  31. 31. AI Fairness, Transparency & Security 31IBM Developer / © 2019 IBM Corporation https://github.com/IBM/AIX360/ https://github.com/IBM/AIF360 https://github.com/IBM/adversarial-robustness-toolbox
  32. 32. 32 Model Asset eXchange (MAX) ibm.biz/model-exchange IBM Developer / © 2019 IBM Corporation
  33. 33. What is MAX? 33 - One place for state-of-art open source deep learning models - Wide variety of domains - Tested code and IP - Free and open source - Both trainable and trained versions IBM Developer / © 2019 IBM Corporation
  34. 34. Deployable asset on MAX IBM Developer / © 2019 IBM Corporation 34 Data ExpertiseComputeModel REST API specificationPre Trained ModelI/O processing Deployable Asset on Model Asset Exchange MetadataInferenceSwagger Specification
  35. 35. Deployable asset on MAX IBM Developer / © 2019 IBM Corporation 35 MetadataInferenceSwagger Specification
  36. 36. Deployable asset on MAX IBM Developer / © 2019 IBM Corporation 36
  37. 37. Hosted resources IBM Developer / © 2019 IBM Corporation 37 Model Service Web App Kubernetes
  38. 38. Trainable asset on MAX IBM Developer / © 2019 IBM Corporation 38 Data ExpertiseComputeModel REST API specificationPre Trained ModelI/O processing Deployable Asset on Model Asset Exchange
  39. 39. Trainable asset on MAX IBM Developer / © 2019 IBM Corporation 39 Data ExpertiseComputeModel REST API specificationPre Trained ModelI/O processing Deployable Asset on Model Asset Exchange Standardized scripts & wrappers
  40. 40. MAX and CI/CD IBM Developer / © 2019 IBM Corporation 40 – TravisCI for GitHub repositories – Jenkins for more complex workflows • Deploying long-running instances to Kubernetes • Periodic jobs • Trainable model testing (using Watson Machine Learning service)
  41. 41. Conclusion – ML Pipelines, not just models – Need to take into account • Data • Models • Variation over time • Fairness, explainability, transparency – Containers / Kube? • KubeFlow, MLFlow • Tekton IBM Developer / © 2019 IBM Corporation 41
  42. 42. Thank you IBM Developer / © 2019 IBM Corporation Sign up for IBM Cloud and try Watson Studio: https://ibm.biz/BdznGk codait.org twitter.com/MLnick github.com/MLnick developer.ibm.com 42 Model Asset Exchange: https://ibm.biz/model-exchange
  43. 43. IBM Developer / © 2019 IBM Corporation 43
  • ssusere5a90d

    Dec. 30, 2019

Continuous integration and deployment has become an increasingly standard and common practice in software development. However, doing this for machine learning models and applications introduces many challenges. Not only do we need to account for standard code quality and integration testing, but how do we best account for changes in model performance metrics coming from changes to code, deployment framework or mechanism, pre- and post-processing steps, changes in data, not to mention the core deep learning model itself? In addition, deep learning presents particular challenges: * model sizes are often extremely large and take significant time and resources to train * models are often more difficult to understand and interpret making it more difficult to debug issues * inputs to deep learning are often very different from the tabular data involved in most ‘traditional machine learning’ models * model formats, frameworks and the state-of-the art models and architectures themselves are changing extremely rapidly * usually many disparate tools are combined to create the full end-to-end pipeline for training and deployment, making it trickier to plug together these components and track down issues. We also need to take into account the impact of changes on wider aspects such as model bias, fairness, robustness and explainability. And we need to track all of this over time and in a standard, repeatable manner. This talk explores best practices for handling these myriad challenges to create a standardized, automated, repeatable pipeline for continuous deployment of deep learning models and pipelines. I will illustrate this through the work we are undertaking within the free and open-source IBM Model Asset eXchange.

Views

Total views

547

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

34

Shares

0

Comments

0

Likes

1

×