Once a model is deployed, you have a responsibility to ensure its reliability and performance in production. That means that in addition to system monitoring, you should be checking and monitoring its ML health and vitals such as accuracy, bias, and variance as new data comes in. In this online workshop we’ll discuss how to build a system to monitor your machine learning model in production on Kubernetes. You’ll learn to keep track of different models and their model performance over time, and how to set up custom alerts for your models. We’ll discuss what types of variants to monitor, and how to measure its performance. Join CTO of cnvrg.io, Leah Kolben in this hands-on workshop on critical practices for monitoring your machine learning models in production. Using the power of Kubernetes, we’ll build a complete system for model tracking that ensures high performing models in production. Watch the full presentation with video and audio here: https://info.cnvrg.io/monitor-machine-learning-model-workshop What you’ll learn: - Why we monitor models in production - The critical vitals to track and monitor performance - How to set up automated alerts - How to set up Kubernetes for monitoring - Use tools like Grafana and Kibana to monitor and visualize your system and ML health To watch the full live presentation click here: https://info.cnvrg.io/monitor-machine-learning-model-workshop

1. WORKSHOP
Monitor ML models in production
How to monitor and manage the health of your ml models
Leah Kolben, CTO
@leah4kosh
leah@cnvrg.io

2. whoami
• Developer/Data scientist => CTO
• cnvrg.io = built by data scientists, for data scientists to help teams:
• Get from data to models to production in the most efficient and fast way
• bridge science and engineering

3. agenda
• Introduction – recap previous webinars
• Kubernetes overview
• Why should we monitor our models
• Monitor tools
• LIVE workshop
• Summary

4. Introduction
• Previous webinars:
• Train your ML models on Kubernetes
• Run ETL jobs using spark on Kubernetes
• Deploy your ML model in production
• Today: “My model is in production, now what?”
• Use Grafana to monitor your model metrics – CPU & Memory
• Use Kibana to monitor your ML model logs
• Use elasicsearch to index your model and create alerts

5. Kubernetes - recap
• Provides a runtime environment for Docker containers
• Provides an abstraction layer for containers to run on
• Deploy as micro services
• All services are natively load balanced
• Can scale up and down dynamically
• Monitor the health of the containers
• Schedule runs and cronjobs
• Use the same API across EVERY cloud provider and bare metal!

6. • Goals:
• Quickly deploy your ML model as a service
• Reduce costs and man power with auto scaling
• Load balanced the traffic
• Natively monitored by Kubernetes
• Update your model continuously: canary deployments, blue/green deployments
ML in production - recap

7. Why should we monitor our models?
• Able to track you model performance
• Prevent model drift
• Monitor auto scaling and high load of traffic
• Know when to retrain your model and updat it
• Monitor updates – validate new version is better the old one
• Keep you production models updated and relevant!

8. Monitor tools
• Use the EKS stack on kubernetes
• Use open source tools that can be quickly installed on kuberentes using helm
• Use grafana to monitor the resources you rmodel uses
• Use Kibana to show and track you input/output
• Use elasticsearch to index the logs
• Use elasalert for tagging and creating alerts regarding the health of your models

9. Grafana
• Open source analytics and monitoring for a variety of databases
• Natively connects to the EKS stack
• Use custom dashboards to track your model usage

10. Grafana

11. Grafana – custom dashboard

12. Kibana
• Part of the EKS stack
• Log all input and output
• Log internal logs
• Search and find prediction outputs
• Visualize and understand the data

13. Kibana

14. Elasticsearch & elasalert
• Store and index all input/output of you models
• Get sense of you data
• Query your data
• Create custom rules on the data to monitor the health of your model
• Trigger custom webhooks upon alerts – retrain CI/CD pipelines

15. Let’s do it!

16. Using cnvrg to deploy & monitor your models
• Your only responsibility is to write the predict function
• end-2-end pipelines
• EKS stack is deployed automatically
• Dedicated Grafana for each endpoint
• Dedicated kibana for each endpoint
• Alerts and triggers out of the box
• Continual learning support

17. DEMO

18. Summary
• Kubernetes is the becoming the standard way to deploy models
• Overview Kubernetes
• Overview of what’s ML in production
• Monitor – why and how
• Live demo
• Deploy, monitor and retrain models using cnvrg

19. Thanks!
https://cnvrg.io
info@cnvrg.io
+972-506-660186