Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Production and Beyond: Deploying and Managing Machine Learning Models

5,392 views

Published on

Presenters: Rajat Arya, Alice Zheng

Published in: Technology
  • Be the first to comment

Production and Beyond: Deploying and Managing Machine Learning Models

  1. 1. Production and Beyond: Rajat Arya, Senior Product Manager Alice Zheng, Director of Data Science 1 Deploying and Managing Machine Learning Models
  2. 2. Choosing between deployed models. What is Production? Evaluation Monitoring Deployment Management Easily serve live predictions. Measuring quality of deployed models. Tracking model quality & operations.
  3. 3. Lifecycle of ML in Production Evaluation Monitoring Deployment Management
  4. 4. Lifecycle of ML in Production Evaluation Monitoring Deployment Management
  5. 5. The Setup Suppose we are building a website with product recommendations, trained using Amazon reviews. • 34.6M reviews • 2.4M products • 6.6M users
  6. 6. Deployment System Model Historical Data Predictions Live Data Batch training Real-time predictions Feedback
  7. 7. Deployment System Model Historical Data Batch training Real-time predictions input recommendations
  8. 8. Batch Training: DIY • Use entire cluster efficiently • Scale nodes up or down • No data transfer • Operation metrics, dashboard, alarming 8 Scalable Distributed Easy to schedule, launch, and monitor Co-located with Data Model Historical Data Batch training
  9. 9. Dato Distributed Dato Distributed Architecture GraphLab Create
  10. 10. Dato Distributed In one line of code launch a long-running cluster of machines to do parallel / distributed execution from GraphLab Create.
  11. 11. Deployment System Model Historical Data Predictions Live Data Batch training Real-time predictions
  12. 12. Real-time Predictions: DIY • REST endpoints, language independence • Fast predictions, with caching • Replicated models • Alarming, Metrics • Scale up or down • Easy to update / deploy models 12 Low Latency Ease of Integration Scalable Fault Tolerant Maintainable Predictions Live Data Real-time predictions
  13. 13. Dato Predictive Services Architecture Dato Predictive Services website, mobile, browser, etc REST API DIST. CACHE MODEL
  14. 14. Dato Predictive Services In one line of code launch a fault-tolerant, scalable, robust, and maintainable cluster, putting a service-oriented architecture on machine learning models.
  15. 15. Dato Predictive Services Benchmark Amazon review dataset (34.6M reviews, 2.4mi users, 6.6mi products). Measured end to end latency Average: < 65ms P99: < 100ms AWS m3.xlarge instances • 4 cores • 15 GB RAM 3 Node deployment Moderate Load Generated
  16. 16. Recommendation System Model Historical Data Predictions Live Data Batch training Real-time predictions
  17. 17. Lifecycle of ML in Production Evaluation Monitoring Deployment Management
  18. 18. What happens after (initial) deployment
  19. 19. ML production life cycle Evaluation Monitoring Deployment Management
  20. 20. After deployment Evaluate and track metrics over time. React to feedback from deployed models. MonitoringManagementEvaluation
  21. 21. ML in production - 101 Model Historical Data Predictions Live Data Feedback Batch training Real-time predictions
  22. 22. ML in production - 101 Model Historical Data Real-time predictionsBatch training Predictions Model 2 Live Data
  23. 23. Key questions • When to update a model? • How to choose between existing models? • Answer: continuous evaluation and testing
  24. 24. What is evaluation? Predictions Metric + Evaluation What data? Which metric?
  25. 25. Evaluating a recommender Model Historical Data Predictions Live Data Ranking loss User engagement
  26. 26. Evaluating a recommender Model Historical Data Predictions Live Data Ranking loss User engagementOffline evaluation: When to update model Online evaluation: Choosing between models
  27. 27. Updating ML models Why update? • Trends and user tastes change over time • Model performance drops When to update? • Track statistics of data over time • Monitor both offline & online metrics on live data • Update when offline metric diverges from online metrics
  28. 28. Choosing between ML models Model 2 Model 1 2000 visits 10% CTR Group A Everybody gets Model 2 2000 visits 30% CTR Group B Strategy 1: A/B testing—select the best model and use it all the time
  29. 29. Choosing between ML models A statistician walks into a casino… Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500 Play this 85% of the time Play this 10% of the time Play this 5% of the time Multi-armed bandits
  30. 30. Choosing between ML models A statistician walks into an ML production environment Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500 Use this 85% of the time (Exploitation) Use this 10% of the time (Exploration) Use this 5% of the time (Exploration) Model 1 Model 2 Model 3
  31. 31. MAB vs. A/B testing Why MAB? • Continuous optimization, “set and forget” • Maximize overall reward Why A/B test? • Simple to understand • Single winner • Tricky to do right
  32. 32. Other production considerations • Versioning • Logging • Provenance • Dashboards • Reports “Machine learning: The high interest rate credit card of technical debt,” D. Sculley et al, Google, 2014 “Two big challenges in machine learning,” Leon Bottou, ICML 2015 invited talk
  33. 33. Conclusions Evaluation Monitoring Deployment Management Dato Distributed & Dato Predictive Services A/B testing, multi-armed bandits & much more Dato – one stop shop for all stages of the ML lifecycle Simple, platform agnostic interface @datoinc, #DataSmt

×