Download as PDF, PPTX
![What do we mean by MLOps?
“MLOps (a compound of “machine learning” and “operations”)
is a practice for collaboration and communication between
data scientists and operations professionals to help manage
production ML lifecycle.” [1]
[1] https://en.wikipedia.org/wiki/MLOps](https://image.slidesharecdn.com/20adriangonzalez-martin-201124200310/85/Seamless-MLOps-with-Seldon-and-MLflow-9-320.jpg)
![What is MLflow?
➔ Open Source project initially started by Databricks
➔ Now part of the LFAI
“MLflow is an open source platform to manage the ML
lifecycle, including experimentation, reproducibility,
deployment, and a central model registry.” [2]
[2] https://mlflow.org/](https://image.slidesharecdn.com/20adriangonzalez-martin-201124200310/85/Seamless-MLOps-with-Seldon-and-MLflow-13-320.jpg)
![Deploying models is hard
Analysis Tools
Serving
Infrastructure
Monitoring
Machine
Resource
Management
Process
Management
Tools
Analysis Tools
Serving
Infrastructure
Monitoring
Machine
Resource
Management
Process
Management Tools
ML
Code
Data Verification
Data Collection
Feature Extraction
Configuration
Seldon can help with these...
➔ Serving infrastructure
➔ Scaling up (and down!)
➔ Monitoring models
➔ Updating models
[3] Hidden Technical Debt in Machine Learning Systems, NIPS 2015](https://image.slidesharecdn.com/20adriangonzalez-martin-201124200310/85/Seamless-MLOps-with-Seldon-and-MLflow-20-320.jpg)
![What is Seldon Core?
➔ Open Source project created by Seldon
“An MLOps framework to package, deploy, monitor and manage
thousands of production machine learning models” [3]
[3] https://github.com/SeldonIO/seldon-core](https://image.slidesharecdn.com/20adriangonzalez-martin-201124200310/85/Seamless-MLOps-with-Seldon-and-MLflow-21-320.jpg)
![How does Seldon Core work?
Advanced Deployment Models
https://docs.seldon.io/projects/seldon-core/en/latest/examples/mlflow_server_ab_test_ambassador.html
a-b-deployment.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
name: wines-classifier
spec:
name: wines-classifier
predictors:
- graph:
children: []
implementation: MLFLOW_SERVER
modelUri: gs://seldon-models/mlflow/model-a
name: wines-classifier
name: model-a
replicas: 1
traffic: 50
- graph:
children: []
implementation: MLFLOW_SERVER
modelUri: gs://seldon-models/mlflow/model-b
name: wines-classifier
name: model-b
replicas: 1
traffic: 50
➔ A/B Tests
◆ We’ll see this in the demo!
➔ Shadow Deployments](https://image.slidesharecdn.com/20adriangonzalez-martin-201124200310/85/Seamless-MLOps-with-Seldon-and-MLflow-31-320.jpg)
![Demo a-b-deployment.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
name: wines-classifier
spec:
name: wines-classifier
predictors:
- graph:
children: []
implementation: MLFLOW_SERVER
modelUri: gs://seldon-models/mlflow/model-a
name: wines-classifier
name: model-a
replicas: 1
traffic: 50
- graph:
children: []
implementation: MLFLOW_SERVER
modelUri: gs://seldon-models/mlflow/model-b
name: wines-classifier
name: model-b
replicas: 1
traffic: 50
Serving
Model A
Model B
{
{](https://image.slidesharecdn.com/20adriangonzalez-martin-201124200310/85/Seamless-MLOps-with-Seldon-and-MLflow-39-320.jpg)
Uploaded byDatabricks
Seamless MLOps with Seldon and MLflow
The document provides an overview of seamless MLOps using Seldon and MLflow. It discusses how MLOps is challenging due to the wide range of requirements across the ML lifecycle. MLflow helps with training by allowing experiment tracking and model versioning. Seldon Core helps with deployment by providing servers to containerize models and infrastructure for monitoring, A/B testing, and feedback. The demo shows training models with MLflow, deploying them to Seldon for A/B testing, and collecting feedback to optimize models.
More Related Content
Similar to Seamless MLOps with Seldon and MLflow
![[DSC Europe 25] Borko Kozomora - Optimizing business workflows with advances ...](https://cdn.slidesharecdn.com/ss_thumbnails/hbgekyb0txw0xpo4yfml-borko-kozomora-leading-ai-transformation-260122103838-cc29ee38-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...](https://cdn.slidesharecdn.com/ss_thumbnails/1zovd6fgsycdg4wvgvls-milos-belcevic-product-professionals-journey-to-full-stack-product-developer-260123083019-d993120d-thumbnail.jpg?width=640&height=640&fit=bounds)
Seamless MLOps with Seldon and MLflow
- 1. Seamless MLOps withSeldon and MLflow Adaptive. Intelligent. Agile. Intuitive. Alive. Inspiring.
- 2. About me Adrian Gonzalez-Martin MachineLearning Engineer agm@seldon.io @kaseyo23 github.com/adriangonz
- 3. About Seldon We arehiring! seldon.io/careers/
- 4. Outline ➔ Why isMLOps hard? ➔ Training with MLflow ➔ Serving with Seldon ➔ Demo!
- 5. Why is MLOpshard?
- 6. Notebooks (by themselves)don’t scale! What do we mean by MLOps? Untitled.ipynb Untitled_3.ipynb Untitled_2.ipynb
- 7. And training isnot the end goal! What do we mean by MLOps? Data Processing Training Serving DISCLAIMER: This is just a high-level overview!
- 8. There is alarger ML lifecycle What do we mean by MLOps? Data Processing Training Serving Data Lineage Workflow Management Metadata Management Experiment Tracking Hyperparameter Optimisation Packaging Monitoring Explainability Audit Trails Data Labelling Distributed Training DISCLAIMER: This is just a high-level overview!
- 9. What do wemean by MLOps? “MLOps (a compound of “machine learning” and “operations”) is a practice for collaboration and communication between data scientists and operations professionals to help manage production ML lifecycle.” [1] [1] https://en.wikipedia.org/wiki/MLOps
- 10. Why is MLOpshard? Managing the ML lifecycle is hard! ➔ Wide heterogeneous requirements ➔ Technically challenging (e.g. monitoring) ➔ Needs to scale up across every ML model! ➔ Organizational challenge ◆ The lifecycle needs to jump across many walls
- 11. How can wemake MLOps better? Data Processing Training Serving
- 12.
- 13. What is MLflow? ➔Open Source project initially started by Databricks ➔ Now part of the LFAI “MLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry.” [2] [2] https://mlflow.org/
- 14. What is MLflow? MLflow Project MLflow Tracking S3,Minio, DBFS, GCS, etc. Model V1 Model V2 Model V3 1. run experiment and track results 2. “log” trained model 2.1. trained model goes into a persistent store MLflow Registry 3. model registry keeps track of model iterations
- 15. ➔ MLflow Project ◆Defines environment, parameters and model’s interface. ➔ MLflow Tracking ◆ API to track experiment results and hyperparameters. ➔ MLflow Model ◆ Snapshot / version of the model. ➔ MLflow Registry ◆ Keeps track of model metadata What is MLflow?
- 16. How does MLflowwork? MLproject name: mlflow-talk conda_env: conda.yaml entry_points: main: parameters: alpha: float l1_ratio: {type: float, default: 0.1} command: "python train.py {alpha} {l1_ratio}" $ mlflow run ./training -P alpha=0.5 $ mlflow run ./training -P alpha=1.0 MLproject file and training
- 17. MLmodel artifact_path: model flavors: python_function: data: model.pkl env:conda.yaml loader_module: mlflow.sklearn python_version: 3.6.9 sklearn: pickled_model: model.pkl serialization_format: cloudpickle sklearn_version: 0.19.1 run_id: 5a6be5a1ef844783a50a6577745dbdc3 utc_time_created: '2019-10-02 14:21:15.783806' How does MLflow work? MLmodel snapshot
- 18. How does MLflowwork?
- 19.
- 20. Deploying models ishard Analysis Tools Serving Infrastructure Monitoring Machine Resource Management Process Management Tools Analysis Tools Serving Infrastructure Monitoring Machine Resource Management Process Management Tools ML Code Data Verification Data Collection Feature Extraction Configuration Seldon can help with these... ➔ Serving infrastructure ➔ Scaling up (and down!) ➔ Monitoring models ➔ Updating models [3] Hidden Technical Debt in Machine Learning Systems, NIPS 2015
- 21. What is SeldonCore? ➔ Open Source project created by Seldon “An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models” [3] [3] https://github.com/SeldonIO/seldon-core
- 22. Model C Model A Feature Transformation API (REST, gRPC) ModelB A/B Test Multi Armed Bandit Direct traffic to the most optimal model Outlier Detection Key features to identify outlier anomalies (Fraud, KYC) Explanation Why is the model doing what it’s doing? Complex Seldon Core Inference Graph From model binary Or language wrapper Into fully fledged microservice Model A API (REST, gRPC) Simple Seldon Core Inference Graph 1. Containerise 2. Deploy 3. Monitor What is Seldon Core?
- 23. ➔ Abstraction forMachine Learning deployments: SeldonDeployment CRD ◆ Simple enough to deploy a model only pointing to stored weights ◆ Powerful enough to keep full control over the created resources ➔ Pre-built inference servers for a subset of ML frameworks ◆ Ability to write custom ones ➔ A/B tests, shadow deployments, etc. ➔ Integrations with Alibi explainers, outlier detectors, etc. ➔ Tools and integrations for monitoring, logging, scaling, etc. What is Seldon Core? SeldonDeployment CRD to manage ML deployments
- 24. What is SeldonCore? On-prem Cloud Native ➔ Built on top of Kubernetes Cloud Native APIs. ➔ All major cloud providers are supported. ➔ On-prem providers such as OpenShift are also supported. https://docs.seldon.io/projects/seldon-core/en/latest/examples/notebooks.html
- 25. deployment.yaml apiVersion: machinelearning.seldon.io/v1 kind: SeldonDeployment metadata: name:example-model spec: name: example predictors: - componentSpecs: - spec: containers: - image: model:0.1 name: my-model - image: transformer:0.1 name: input-transformer - image: combiner:0.1 name: model-combiner graph: name: input-transformer type: TRANSFORMER children: - name: model-combiner type: COMBINER children: - name: my-model type: MODEL - name: classifier implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-a name: default replicas: 1 SeldonDeployment Pod input-transformer my-model classifier model-combinerorchestrator kubectl apply -f deployment.yaml Orchestrate requests between components init-container Downloads model artifacts How does Seldon Core work?
- 26. ➔ Pre-packaged serversfor common ML frameworks. How does Seldon Core work? Inference Servers model-a SeldonDeployment CR model-b SeldonDeployment CR model-a model-b.bst . . . https://docs.seldon.io/projects/seldon-core/en/latest/servers/overview.html
- 27. How does SeldonCore work? Monitoring https://docs.seldon.io/projects/seldon-core/en/latest/examples/notebooks.html
- 28. ➔ Seldon leveragesKNative for async pipelines which handle: ◆ Outlier detection (through Alibi Detect) ◆ Drift detection (through Alibi Detect) ◆ Custom metrics How does Seldon Core work? Monitoring https://docs.seldon.io/projects/seldon-core/en/latest/examples/notebooks.html
- 29. How does SeldonCore work? Auditability https://docs.seldon.io/projects/seldon-core/en/latest/examples/notebooks.html
- 30. How does SeldonCore work? Explainability https://github.com/seldonio/alibi
- 31. How does SeldonCore work? Advanced Deployment Models https://docs.seldon.io/projects/seldon-core/en/latest/examples/mlflow_server_ab_test_ambassador.html a-b-deployment.yaml apiVersion: machinelearning.seldon.io/v1alpha2 kind: SeldonDeployment metadata: name: wines-classifier spec: name: wines-classifier predictors: - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-a name: wines-classifier name: model-a replicas: 1 traffic: 50 - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-b name: wines-classifier name: model-b replicas: 1 traffic: 50 ➔ A/B Tests ◆ We’ll see this in the demo! ➔ Shadow Deployments
- 32.
- 33. 6.5 2.3 ? ➔We want to predict wine quality for new wines ➔ We want to listen to feedback from customers Demo Wine e-commerce
- 34. Demo Fixed Acidity Volatile Acidity Citric Acid ... Sulphates AlcoholQuality 7 0.27 0.36 0.45 8.8 6 6.3 0.3 0.34 0.49 9.5 7 8.1 0.28 0.4 0.44 10.1 1 7.2 0.23 0.32 0.4 9.9 2 7.2 0.23 0.32 0.4 9.9 5 ... Wine quality dataset
- 35. ➔ Linear regressionwith L1 and L2 regularisers. ➔ Two hyperparameters: Demo ElasticNet
- 36. Demo Wine Project MLflow Tracking GCS Model A Model B 1. run experiment andtrack results 2. “log” trained model 2.1. trained model goes into a persistent store Training
- 37. Demo Hyperparameter setting? ➔ Traintwo versions of ElasticNet ➔ It’s not clear which one would be better in production ➔ Deploy both and do A/B test based on user’s feedback Model A Model B ?Which one is best?
- 38. Demo Serving SeldonDeployment CR model-a model-b GCS ModelA Model B A/B Test Router
- 39. Demo a-b-deployment.yaml apiVersion: machinelearning.seldon.io/v1alpha2 kind:SeldonDeployment metadata: name: wines-classifier spec: name: wines-classifier predictors: - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-a name: wines-classifier name: model-a replicas: 1 traffic: 50 - graph: children: [] implementation: MLFLOW_SERVER modelUri: gs://seldon-models/mlflow/model-b name: wines-classifier name: model-b replicas: 1 traffic: 50 Serving Model A Model B { {
- 40. Demo Feedback SeldonDeployment CR model-a model-b A/BTest Router Feedback (reward signal) ➔ Reward signal based on “proxy metric”: ◆ Sales ◆ User rating
- 41. Feedback Demo ➔ We canbuild a rough reward signal using the squared error
- 42.
- 43.
- 44.
- 45. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.