Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Building
Reproducible ML
with MLOps and
Metadata
SpeechVision Language
Switchboard
Switchboard
cellular
Meeting
speech
IBM
Switchboard
Broadcast
speech
1993 20172000 2006 ...
Microsoft 365
ML at Microsoft
Research
But ML is HARD!
Building a model
Building
a model
Data ingestion Data analysis
Data
transformation
Data validation Data splitting
Trainer
Model
validation
...
Ok, but, like, I’m
a data scientist. IDGAF
I don’t care
about all that.
Yes You Do!
Cowboys and Ranchers Can Be Friends!
SRE/ML EngineersData Scientist
• Quick iteration
• Frameworks they
understand
• Best ...
MLOps
MLOps = ML + DEV + OPS
Experiment
Data Acquisition
Business Understanding
Initial Modeling
Develop
Modeling
Operate
Contin...
A Pipeline You Say?
Does My Model Actually Work?
SRE/ML EngineersData Scientist
Time to test out
my model…
Laptop The Cloud
Does My Model Actually Work?
SRE/ML EngineersData Scientist
Laptop The Cloud
Looks good to
me! To Production!
What is
happening…
Source Control
Does My Model Actually Work?
SRE/ML EngineersData Scientist
Laptop The Cloud
A Small Example of Issues You Can Have…
• Inappropriate HW/SW stack
• Mismatched driver versions
• Crash looping deploymen...
Does My Model Actually Work?
SRE/ML EngineersData Scientist
Laptop The Cloud
Source Control
Automated
Validation &
Profilin...
But I Can Do All
These Manually…
No.
MLOps is a Platform and a Philosophy
Even if:
• Every data scientist trained...
• And you had all the tools necessary...
•...
That’s MLOps!
Does My Model Actually Work?
SRE/ML EngineersData Scientist
Laptop The Cloud
Source Control
Automated
Validation &
Profilin...
Metadata!
Metadata is ...
A contract for the interface of a service
A historical record of the outcome of a process
3. Structured da...
Does My Model Actually Work?
SRE/ML EngineersData Scientist
Laptop The Cloud
Source Control
Automated
Validation &
Profilin...
Haven’t Convinced
You Yet?
What Did My Customers See?
SRE/ML Engineers
The Cloud
Front End
Model Server
Customer
I’d Like a loan,
please.
Source Cont...
What Did My Customers See?
SRE/ML Engineers
The Cloud
Front End
Model Server
Customer
No.
Source Control
What Did My Customers See?
SRE/ML Engineers
The Cloud
Front End
Model Server
Customer
Ok, but why?
Source Control
Source Control
What Did My Customers See?
SRE/ML Engineers
The Cloud
Front End
Model Server
Customer
Uh oh.
Lawyer
Lawyer
...
It’s Not Just About Explainability!
• Yes, models are complicated
• But, that’s not enough:
• What data did you train on?
...
What Did My Customers See?
SRE/ML Engineers
The Cloud
Front End
Model Server
Customer
Source Control
Automated
Validation ...
32c04681d7573
Automated
Validation &
Profiling
Package
For Rollout
Explain Model
& Look for Bias
Clean/
Minimize
Code
Sane
...
Automated
Validation &
Profiling
Package
For Rollout
Explain Model
& Look for Bias
Clean/
Minimize
Code
Sane
Deployment
Wha...
What Did My Customers See?
SRE/ML Engineers
Front End
Model Server
Customer
Immutable
Metadata Store
32c04681d7573
32c0468...
Metadata Gives You a Repeatable Record
• What data you trained on
• How you transformed it for training
• What the results...
Ok, but you can’t
possibly expect me
to use YAML.
Introducing MLSpecLib
A simple, Python-native library for using with schematized objects
• Extends marshmallow (minimum re...
ENOUGH TALK.
GET TO THE DEMO.
Come Help!
me: David Aronchick (aronchick@gmail.com)
twitter: @aronchick
apps: http://mlops-github.com/
mlspec-lib on pypi: https://p...
Rsqrd AI: How to Design a Reliable and Reproducible Pipeline
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Rsqrd AI: How to Design a Reliable and Reproducible Pipeline

Download to read offline

In this talk, David Aronchick, co-founder of Kubeflow and Microsoft's Head of Open Source ML, talks about designing reproducible and reliable ML pipelines. He speaks about the importance and impact of MLOps and use of metadata in pipelines. He also talks about a library he wrote to help with this problem, MLSpecLib.


**These slides are from a talk given at Rsqrd AI. Learn more at rsqrdai.org**

  • Be the first to like this

Rsqrd AI: How to Design a Reliable and Reproducible Pipeline

  1. 1. Building Reproducible ML with MLOps and Metadata
  2. 2. SpeechVision Language Switchboard Switchboard cellular Meeting speech IBM Switchboard Broadcast speech 1993 20172000 2006 2010 5.1% Switchboard speech recognition test 96% RESNET vision test 152 layers 88.5% SQuAD reading comprehension test 69.9% MT research system 2016 Object recognition Human parity 2017 Speech recognition Human parity 2018 Machine reading comprehension Human parity 2018 Machine translation Human parity Microsoft ML breakthroughs
  3. 3. Microsoft 365 ML at Microsoft Research
  4. 4. But ML is HARD!
  5. 5. Building a model
  6. 6. Building a model Data ingestion Data analysis Data transformation Data validation Data splitting Trainer Model validation Training at scale LoggingRoll-out Serving Monitoring
  7. 7. Ok, but, like, I’m a data scientist. IDGAF I don’t care about all that.
  8. 8. Yes You Do!
  9. 9. Cowboys and Ranchers Can Be Friends! SRE/ML EngineersData Scientist • Quick iteration • Frameworks they understand • Best of breed tools • No management headaches • Unlimited scale • Reuse of tooling and platforms • Corporate compliance • Observability • Uptime
  10. 10. MLOps
  11. 11. MLOps = ML + DEV + OPS Experiment Data Acquisition Business Understanding Initial Modeling Develop Modeling Operate Continuous Delivery Data Feedback Loop System + Model Monitoring + Testing Continuous Integration Continuous Deployment ML
  12. 12. A Pipeline You Say?
  13. 13. Does My Model Actually Work? SRE/ML EngineersData Scientist Time to test out my model… Laptop The Cloud
  14. 14. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Looks good to me! To Production!
  15. 15. What is happening… Source Control Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud
  16. 16. A Small Example of Issues You Can Have… • Inappropriate HW/SW stack • Mismatched driver versions • Crash looping deployment • Data/model versioning [Nick Walsh] • Non-standard images/OS version • Pre-processing code doesn’t match production pre-processing • Production data doesn’t match training/test data • Output of the model doesn’t match application expectations • Hand-coded heuristics better than model [Adam Laiacano] • Model freshness (train on out-of-date data/input shape changed) • Test/production statistics/population shape skew • Overfitting on training/test data • Bias introduction (or not tested) • Over/under HW provisioning • Latency issues Or It Just Doesn’t Work! At All! • Permissions/certs • Failure to obey health checks • Killed production model before roll out of new/in wrong order • Thundering herd for new model • Logging to the wrong location • Storage for model not allocated properly/accessible by deployment tooling • Route to artifacts not available for download • API signature changes not propagated/expected • Cross-data center latency • Expected benefit doesn’t materialize (e.g. multiple components in the app change simultaneously) • Get wrong/no traffic because A/B config didn’t roll out • No CI/CD; manual changes untracked [Jon Peck] • Get too much traffic too soon (expected to canary/exponential roll out) • Outliers not predicted [MikeBSilverman] • Change was a good change, but didn’t communicate with the rest of the team (so you must roll back) • No dates! (date to measure impact/improvement against a pre-agreed measure; date scheduled to assess data changes) [Mary Branscombe] • LACK OF DOCUMENTATION!! (the problem, the testing, the solution, lots more) [Terry Christiani] • Successful model causes pain elsewhere in the organization (e.g. detecting faults previously missed) [Mark Round] • Lack of visibility into real-time model behavior (detecting data drift, live data distribution vs train data, etc) [Nick Walsh]
  17. 17. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment Nice. Nice. ✔
  18. 18. But I Can Do All These Manually…
  19. 19. No.
  20. 20. MLOps is a Platform and a Philosophy Even if: • Every data scientist trained... • And you had all the tools necessary... • And they all worked together... • And your SREs understood ML modeling... • And and and and ... You’d still need a permanent, repeatable record of what you did
  21. 21. That’s MLOps!
  22. 22. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment Nice. Nice. ✔ What goes here?
  23. 23. Metadata!
  24. 24. Metadata is ... A contract for the interface of a service A historical record of the outcome of a process 3. Structured data that allows for (more) reliable automated workflows 4. And much much more...
  25. 25. Does My Model Actually Work? SRE/ML EngineersData Scientist Laptop The Cloud Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment Nice. Nice. ✔
  26. 26. Haven’t Convinced You Yet?
  27. 27. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer I’d Like a loan, please. Source Control
  28. 28. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer No. Source Control
  29. 29. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Ok, but why? Source Control
  30. 30. Source Control What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Uh oh. Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer Lawyer LawyerLawyer
  31. 31. It’s Not Just About Explainability! • Yes, models are complicated • But, that’s not enough: • What data did you train on? • How did you transform/exclude outliers? • What are the data statistics? • Did anything change between code and production? • What model did you actually serve (to this person)? • Metadata can help!
  32. 32. What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Source Control Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment
  33. 33. 32c04681d7573 Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Source Control Immutable Metadata Store b151f8e65b32a c7f4e7607b4b7 0ef1d58921d89 e2e1e994c4251 786c8e57a6d51 9ce88802f0759 32c04681d7573
  34. 34. Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment What Did My Customers See? SRE/ML Engineers The Cloud Front End Model Server Customer Source Control Immutable Metadata Store b151f8e65b32a c7f4e7607b4b7 0ef1d58921d89 e2e1e994c4251 786c8e57a6d51 9ce88802f0759 32c04681d7573 Why didn’t I get a loan? 32c04681d7573
  35. 35. What Did My Customers See? SRE/ML Engineers Front End Model Server Customer Immutable Metadata Store 32c04681d7573 32c04681d7573 Automated Validation & Profiling Package For Rollout Explain Model & Look for Bias Clean/ Minimize Code Sane Deployment The Cloud Source Control b151f8e65b32a c7f4e7607b4b7 0ef1d58921d89 e2e1e994c4251 786c8e57a6d51 9ce88802f0759 32c04681d7573
  36. 36. Metadata Gives You a Repeatable Record • What data you trained on • How you transformed it for training • What the results of the training were • What kind of fairness tests you ran • How those results compared with previous results • How you rolled it out • Which version a customer saw • And, and, and ... All Automatically! (Mostly)
  37. 37. Ok, but you can’t possibly expect me to use YAML.
  38. 38. Introducing MLSpecLib A simple, Python-native library for using with schematized objects • Extends marshmallow (minimum rewriting) • Comes with some standard schemas in the box • It started with ML but it works for anything But wait there’s more! • Read/write serialized objects natively with Python (using dot notation and everything) - No YAML! No JSON! • User friendly, trivially extensible schema language - including importing from a remote store • “Lazy” enforcement (at load/save time only) • Code-gen for the REALLY lazy (like me)
  39. 39. ENOUGH TALK. GET TO THE DEMO.
  40. 40. Come Help!
  41. 41. me: David Aronchick (aronchick@gmail.com) twitter: @aronchick apps: http://mlops-github.com/ mlspec-lib on pypi: https://pypi.org/project/mlspeclib/ mlspec-lib on github: https://github.com/mlspec/mlspec-lib THANK YOU!

In this talk, David Aronchick, co-founder of Kubeflow and Microsoft's Head of Open Source ML, talks about designing reproducible and reliable ML pipelines. He speaks about the importance and impact of MLOps and use of metadata in pipelines. He also talks about a library he wrote to help with this problem, MLSpecLib. **These slides are from a talk given at Rsqrd AI. Learn more at rsqrdai.org**

Views

Total views

949

On Slideshare

0

From embeds

0

Number of embeds

864

Actions

Downloads

4

Shares

0

Comments

0

Likes

0

×