Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus

119 views

Published on

These are slides from Manasi Vartak's Strata Talk in March 2020 on Robust MLOps with Open-Source.

* Introduction to talk
* What is MLOps?
* Building an MLOps Pipeline
* Real-world Simulations
* Let’s fix the pipeline
* Wrap-up

Published in: Software
  • Be the first to comment

Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus

  1. 1. www.verta.ai Confidential Robust MLOps with Open-Source: ModelDB, Docker, Jenkins, and Prometheus !1 Presented by: Manasi Vartak CEO, Verta.ai Michael Liu Software Engineer, Verta.ai Slack (Q&A): http://bit.ly/modeldb-mlops #webinars
  2. 2. www.verta.ai Confidential About !2 • Open-core MLOps Platform for the full model lifecycle • Model versioning, deployment & ops, monitoring • Built for data science; able to run at large scale Manasi Vartak CEO, Verta.ai Michael Liu Software Engineer, Verta.ai MIT CSAIL Ph.D. UCSD, Cognitive Science Creator of ModelDB — first OSS model management and versioning system Neural-network based audio analysis, everything about Python, Verta client libs
  3. 3. www.verta.ai Confidential Agenda • Part I: Intro to MLOps (15 mins) • Part II: Building an MLOps Pipeline (30 mins) • Part III: Questions (10 mins) !3 Slack (Q&A): http://bit.ly/modeldb-mlops #webinars
  4. 4. www.verta.ai Confidential Models have become Easy to Build !4 from fastai.vision import * from fastai.metrics import accuracy data.normalize(imagenet_stats) learner = create_cnn(data, models.resnet18, metrics=[accuracy], callback_fns=ShowGraph) learner.fit_one_cycle(8, max_lr=slice(1e-3, 1e-2)) learner.save('stage-1')
  5. 5. www.verta.ai Confidential Delivery and Operations of ML-Products is Broken !5 It used to take us 20+ weeks to bring a new version of the model into production. A predictive readmission model that was trained, optimized and deployed at a hospital would start sharply degrading within two to three months.
  6. 6. www.verta.ai Confidential Why is ML Delivery and Ops so Hard? !6
  7. 7. www.verta.ai Confidential!7 Challenge 1. Model Development is empirical & ad-hoc Model 1 Accuracy: 62%
  8. 8. www.verta.ai Confidential!8 Challenge 1. Model Development is empirical & ad-hoc Model 3 Accuracy: 76% val udf1: (Int => Int) = (delayed..) df.withColumn(“timesDelayed”, udf1) RandomForestClassifier
  9. 9. www.verta.ai Confidential!9 Challenge 1. Model Development is empirical & ad-hoc Model 5 Accuracy: 68% val udf1: (Int => Int) = (delayed..) df.withColumn(“timesDelayed”, udf1) RandomForestClassifier credit-default-clean.csv val lrGrid = new ParamGridBuilder() .addGrid(rf.maxDepth, Array(5, 10, 15)) .addGrid(rf.numTrees, Array(50, 100))
  10. 10. www.verta.ai Confidential!10 Challenge 1. Model Development is empirical & ad-hoc Model 50 Accuracy: 82% val udf1: (Int => Int) = (delayed..) df.withColumn(“timesDelayed”, udf1) RandomForestClassifier credit-default-clean.csv val lrGrid = new ParamGridBuilder() .addGrid(rf.maxDepth, Array(5, 10, 15)) .addGrid(rf.numTrees, Array(50, 100)) val labelIndexer1 = new LabelIndexer() val labelIndexer2 = new LabelIndexer() … val udf1: (Int => Int) = (delayed..) val udf2: (String, Int) = … df.withColumn(“timesDelayed”, udf1) .withColumn(“percentPaid”, udf2) .withColumn(“creditUsed”, udf3) val scaler = new StandardScaler() .setInputCol(“features”) …
  11. 11. www.verta.ai Confidential Challenge 2. DS/ML vs. Software are Different worlds !11 • Flexibility • Prototyping • Bespoke code • Robustness • Scale • Generalization
  12. 12. www.verta.ai Confidential Challenge 3. Existing Tools are not ML-Aware !12 Data Drift Resource Utilization Optimizations Interdependencies
  13. 13. www.verta.ai Confidential MLOps: DevOps for ML !13
  14. 14. www.verta.ai Confidential!14 DevOps: Deliver Software Products Faster, More Reliably
  15. 15. www.verta.ai Confidential!15 Cross-Validation LIME Shapley MLOps: Deliver ML Products Faster, More Reliably ??
  16. 16. www.verta.ai Confidential!16 This talk! MLOps: Deliver ML Products Faster, More Reliably
  17. 17. www.verta.ai Confidential Building an MLOps Pipeline with open-source: Docker, Jenkins, Prometheus !17 https://github.com/VertaAI/modeldb/tree/master/demos/webinar-2020-5-6
  18. 18. www.verta.ai Confidential Running Example: TweetTrader !18 Trader DOW NASDAQ AI company using social media analytics to make $$$ … … NLP NLP NLP
  19. 19. www.verta.ai Confidential Let’s help TweetTrader do MLOps !19 Package Step 2: Docker Container Release Step 3: Jenkins Monitor Step 4: Prometheus + Logs Trained Model Step 1: Tweet Model
  20. 20. www.verta.ai Confidential What happens in the wild? !20
  21. 21. www.verta.ai Confidential Scenario: All our new traffic is from Germany !21
  22. 22. www.verta.ai Confidential!22 Challenge 1. Model Development is empirical & ad-hoc Model 50 Accuracy: 82% val udf1: (Int => Int) = (delayed..) df.withColumn(“timesDelayed”, udf1) RandomForestClassifier credit-default-clean.csv val lrGrid = new ParamGridBuilder() .addGrid(rf.maxDepth, Array(5, 10, 15)) .addGrid(rf.numTrees, Array(50, 100)) val labelIndexer1 = new LabelIndexer() val labelIndexer2 = new LabelIndexer() … val udf1: (Int => Int) = (delayed..) val udf2: (String, Int) = … df.withColumn(“timesDelayed”, udf1) .withColumn(“percentPaid”, udf2) .withColumn(“creditUsed”, udf3) val scaler = new StandardScaler() .setInputCol(“features”) …
  23. 23. www.verta.ai Confidential!23
  24. 24. www.verta.ai Confidential Scenario: My colleague has an even better model !24
  25. 25. www.verta.ai Confidential!25
  26. 26. www.verta.ai Confidential What’s missing? !26
  27. 27. www.verta.ai Confidential!27 Cross-Validation LIME Shapley I was kidding; we haven’t solved the ML part
  28. 28. www.verta.ai Confidential!28 DevOps: Deliver Software Products Faster, More Reliably
  29. 29. www.verta.ai Confidential In code, every change that we make is tracked !29
  30. 30. www.verta.ai Confidential In code, every change that we make is tracked !30 java.lang.NullPointerException: null ... ... WebBackend SHA: ed05334
  31. 31. www.verta.ai Confidential What about models? !31 Guten Nacht: Negative Guten Morgen: Negative ... ... NLPModel s3://models/final-bert-March12 ?? Code? Data? Config? Env?
  32. 32. www.verta.ai Confidential What’s missing is ML-specific model versioning • Uniquely identifies a model • Enables user to go back in time and fully recreate a model • Code • Data • Config • Environment • Allows branching, merging, diffs etc. • Versioning that integrates into the ML workflow (e.g., library vs. CLI) !32
  33. 33. www.verta.ai Confidential ModelDB: open-source model versioning !33 ModelDB 2.0: https://github.com/VertaAI/modeldb • Code • Data • Config • Env • Code • Data • Config • Env
  34. 34. www.verta.ai Confidential Let’s fix the pipeline !34
  35. 35. www.verta.ai Confidential Revised MLOps Pipeline !35 Package Step 2: Docker Container Release Step 3: Jenkins Monitor Step 4: Prometheus + Logs Trained Model Step 1: Tweet Model + ModelDB
  36. 36. www.verta.ai Confidential Step 1: Train a Tweet Classification model + use ModelDB for versioning !36
  37. 37. www.verta.ai Confidential Scenario: All our new traffic is from Germany !37
  38. 38. www.verta.ai Confidential Scenario: My colleague has an even better model !38
  39. 39. www.verta.ai Confidential lives! !39 Trader
  40. 40. www.verta.ai Confidential Revised MLOps Pipeline !40 Package Step 2: Docker Container Release Step 3: Jenkins Monitor Step 4: Prometheus + Logs Trained Model Step 1: Tweet Model + ModelDB
  41. 41. www.verta.ai Confidential Summary • Part I: Intro to MLOps • Part II: Building an MLOps Pipeline • Basic pipeline: Docker, Jenkins, Prometheus • Real-world simulations • Pipeline with versioning: ModelDB, Docker, Jenkins, Prometheus !41 https://github.com/VertaAI/modeldb/tree/master/demos/webinar-2020-5-6
  42. 42. www.verta.ai Confidential 3 Takeaways • MLOps is DevOps for ML: it helps you ship ML products faster • Model Versioning : MLOps :: Git : DevOps • Robust OSS MLOps: ModelDB + Docker + Jenkins + Prometheus !42 https://github.com/VertaAI/modeldb/tree/master/demos/webinar-2020-5-6
  43. 43. www.verta.ai Confidential Thanks! https://github.com/VertaAI/modeldb | Today’s talk: modeldb/demos/ | Slack: http://bit.ly/modeldb-mlops !43 Pre-register for our MLOps Salon happening in June! https://info.verta.ai/ml-ops-event

×