Learn how to deploy your model to production in 30 minutes and.. • Reduce costs and man power with auto scaling • Load balanced the traffic • Natively monitored by Kubernetes • Update your model continuously: canary deployments, blue/green deployments

1. WORKSHOP
Deploy ML models to production using Kubernetes
How to create and deploy your ML models to kubernetes in minutes
Leah Kolben, CTO
@leah4kosh
leah@cnvrg.io

2. whoami
• Developer/Data scientist => CTO
• cnvrg.io = built by data scientists, for data scientists to help teams:
• Get from data to models to production in the most efficient and fast way
• bridge science and engineering

3. agenda
• Introduction – previous webinars
• Kubernetes overview
• Why we need Kubernetes for production models
• Production components
• LIVE workshop
• Summary

4. Introduction
• Previous webinars:
• Train your ML models on Kubernetes
• Run ETL jobs using spark on Kubernetes
• Today: “My model is ready, now what?”
• Use flask to create a service of your model
• Use Kubernetes to push the service to production
• Use Kubernetes for auto scaling, load balancer and more

5. Kubernetes - recap
• Provides a runtime environment for Docker containers
• Provides an abstraction layer for containers to run on
• Deploy as micro services
• All services are natively load balanced
• Can scale up and down dynamically
• Monitor the health of the containers
• Schedule runs and cron-jobs
• Use the same API across EVERY cloud provider and bare metal!

6. • Goals:
• Quickly deploy your ML model as a service
• Reduce costs and man power with auto scaling
• Load balanced the traffic
• Natively monitored by Kubernetes
• Update your model continuously: canary deployments, blue/green deployments
ML in production

7. Architecture
• Write the predict functionality
• Flask for wrapping the model as a REST API service
• Docker image to wrap the API service requirements
• Use a Deployment for creating the service in the cluster
• Use a service to expose the deployment outside of the cluster
• Optional:
• Gunicorn
• Nginx
• Replica set

8. Flask- example

9. Deployment - example

10. service- example

11. Let’s do it!

12. Using cnvrg to deploy ML models
• Your only responsibility is to write the predict function
• end-2-end pipelines
• Manage traffic using istio.io
• Manage rolling updates
• Use continual learning to track the health of the models
• Monitor the models using grafana, kibana & prometheus
• Automatic auto scaling

13. DEMO

14. Summary
• Kubernetes is the becoming the standard way to deploy models
• Overview Kubernetes
• Overview of what’s ML in production
• Architecture overview
• Live demo
• Deploy, manage and retrain models using cnvrg

15. Thanks!
https://cnvrg.io
info@cnvrg.io
+972-506-660186