After initial deployment, machine learning models require continuous evaluation, monitoring, and management over their lifetime. Key aspects include tracking offline and online metrics to evaluate model performance over time, reacting to feedback from deployed models, and choosing the best model through techniques like A/B testing or multi-armed bandits. Proper evaluation, monitoring, and management are essential for maintaining high-quality machine learning systems in production.

1. What happens after (initial) deployment

2. ML production life cycle
Evaluation
Monitoring
Deployment
Management

3. After deployment
Evaluate and track metrics over time.
React to feedback from deployed models.
MonitoringManagementEvaluation

4. ML in production - 101
Model
Historical
Data
Predictions
Live
Data
Feedback
Batch training Real-time predictions

5. ML in production - 101
Model
Historical
Data
Real-time predictionsBatch training
Predictions
Model 2
Live
Data

6. Key questions
• When to update a model?
• How to choose between existing models?
• Answer: continuous evaluation and testing

7. What is evaluation?
Predictions Metric
+
Evaluation
What data?
Which metric?

8. Evaluating a recommender
Model
Historical
Data
Predictions
Live
Data
Ranking
loss
User
engagement

9. Evaluating a recommender
Model
Historical
Data
Predictions
Live
Data
Ranking
loss
User
engagementOffline evaluation:
When to update model
Online evaluation:
Choosing between models

10. Updating ML models
Why update?
• Trends and user tastes change over time
• Model performance drops
When to update?
• Track statistics of data over time
• Monitor both offline & online metrics on live data
• Update when offline metric diverges from online metrics

11. Choosing between ML models
Model 2
Model 1
2000 visits
10% CTR
Group A
Everybody gets
Model 2
2000 visits
30% CTR
Group B
Strategy 1: A/B testing—select the best model and use it all the time

12. Choosing between ML models
A statistician walks into a casino…
Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500
Play this 85% of
the time
Play this 10% of
the time
Play this 5% of the
time
Multi-armed
bandits

13. Choosing between ML models
A statistician walks into an ML production environment
Pay-off $1:$1000 Pay-off $1:$200 Pay-off $1:$500
Use this 85% of
the time
(Exploitation)
Use this 10% of
the time
(Exploration)
Use this 5% of the
time
(Exploration)
Model 1 Model 2 Model 3

14. MAB vs. A/B testing
Why MAB?
• Continuous optimization, “set and forget”
• Maximize overall reward
Why A/B test?
• Simple to understand
• Single winner
• Tricky to do right

15. Other production considerations
• Versioning
• Logging
• Provenance
• Dashboards
• Reports
“Machine learning: The high interest rate credit card of technical debt,” D. Sculley et al, Google, 2014
“Two big challenges in machine learning,” Leon Bottou, ICML 2015 invited talk

16. Conclusions
Evaluation
Monitoring
Deployment
Management
Dato Distributed
&
Dato Predictive Services
A/B testing,
multi-armed bandits
& much more
Dato – one stop shop for all stages of the ML life cycle
Simple, platform agnostic interface
@datoinc, #DataSmt