Vigil: Effective end-to-end monitoring for large-scale recommender systems at Glance
•
0 likes•49 views
The success of large-scale recommender systems hinges upon their ability to deliver accurate and timely recommendations to a diverse user base. At Glance, we deliver snackable personalized content to the lock screens of 200M smartphones. In this context, continuous monitoring is paramount as it safeguards data integrity, detects drifts, addresses evolving user preferences, optimizes system downtime, and ultimately augments the system’s effectiveness and user satisfaction. In this talk, we delve into Vigil, a set of monitoring practices developed to provide comprehensive end-to-end monitoring of recommender systems at Glance. These practices revolve around three key pillars: mitigating developer fatigue, ensuring precise predictions, and establishing a centralized monitoring framework. By adopting these practices, we have observed an 18\% increase in user engagement, a 30% reduction in compute cost, a 26% drop in downtime, and a surge in developer productivity demonstrated by a 45% decrease in turnaround time.
Recommended
Recommended
More Related Content
Similar to Vigil: Effective end-to-end monitoring for large-scale recommender systems at Glance
Similar to Vigil: Effective end-to-end monitoring for large-scale recommender systems at Glance (20)
Recently uploaded
Recently uploaded (20)
Vigil: Effective end-to-end monitoring for large-scale recommender systems at Glance
- 1. VIGIL Effective end-to-end monitoring for large-scale recommender systems at Glance
- 2. We'll be your speakers for today's featured talk on monitoring practices. Brace yourselves for an enjoyable session!! Hello! Priyansh Manisha Page 02 |
- 4. 3 in 4 Android phones have Glance Lock Screen 230+ M users Page 04 |
- 6. Page 06 |
- 7. Page 07 |
- 8. Page 07 |
- 9. Page 07 |
- 10. Page 07 |
- 11. Page 07 |
- 12. Page 07 |
- 13. ML MONITORING
- 28. HOW ML MONITORING IS DIFFERENT FROM SYSTEM MONITORING? Page 10 |
- 29. VIGIL MONITORING FRAMEWORK @ GLANCE
- 30. VIGIL FRAMEWORK Page 12 | QUALITY RELIABILITY & ROBUSTNESS DEVELOPER PRODUCTIVITY OPTIMIZE RESOURCE & COST PROACTIVE ALERTING & SYSTEM MONITORING DEPENDENCY & IMPACT MONITORING TESTING & PERFORMANCE MONITORING DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS CENTRALIZED VIEW & DEPENDENCY GRAPHS ALERTING MODEL PERFORMANCE MONITORING
- 31. VIGIL FRAMEWORK Page 12 | QUALITY RELIABILITY & ROBUSTNESS DEVELOPER PRODUCTIVITY OPTIMIZE RESOURCE & COST PROACTIVE ALERTING & SYSTEM MONITORING DEPENDENCY & IMPACT MONITORING TESTING & PERFORMANCE MONITORING DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS CENTRALIZED VIEW & DEPENDENCY GRAPHS ALERTING MODEL PERFORMANCE MONITORING
- 32. VIGIL FRAMEWORK Page 12 | QUALITY RELIABILITY & ROBUSTNESS DEVELOPER PRODUCTIVITY OPTIMIZE RESOURCE & COST PROACTIVE ALERTING & SYSTEM MONITORING DEPENDENCY & IMPACT MONITORING TESTING & PERFORMANCE MONITORING DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS CENTRALIZED VIEW & DEPENDENCY GRAPHS ALERTING MODEL PERFORMANCE MONITORING
- 33. CHALLENGES Drift in data y = f(x) Page 13 | SET - 1
- 34. Changing data distribution Page 14 |
- 35. Changing user interest & behavior Page 15 |
- 36. VIGIL TECHNIQUES 01 02 03 DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS 04 MODEL PERFORMANCE MONITORING 05 CENTRALIZED VIEW & DEPENDENCY GRAPHS 06 ALERTING Page 16 |
- 37. Distance metrics Error-based metrics Statistical tests Data sketches DRIFT DETECTION Page 17 | QUALITY, REOSURCE & COST OPTIMIZATION
- 38. CHALLENGES Issues in data Bias and fairness concerns Incorrect or outdated features SET - 2 Page 18 |
- 39. Data Quality and Hygiene Areas of impact Inaccurate Data Data duplicati on Missing values Bias Incorrect features Recommendation High Low Low High High Business metrics High Low Low High High User trust Low Low High High High Areas of Impact Matrix Page 19 |
- 40. VIGIL TECHNIQUES Page 20 | 01 02 03 DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS 04 MODEL PERFORMANCE MONITORING 05 CENTRALIZED VIEW & DEPENDENCY GRAPHS 06 ALERTING
- 41. Data validation Statistical measures Feature consistency Outlier detection DATA QUALITY & HYGIENE Page 21| RELIABILITY & ROBUSTNESS
- 45. VIGIL TECHNIQUES Page 25 | 01 02 03 DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS 04 MODEL PERFORMANCE MONITORING 05 CENTRALIZED VIEW & DEPENDENCY GRAPHS 06 ALERTING
- 46. COMPONENT LEVEL METRICS Logging component-specific errors Log-based metric dashboards Latency SLAs Load testing Page 26 | RELIABILITY & ROBUSTNESS, DEVLEOPER PRODUCTIVITY
- 47. CHALLENGES Lack of Actionable Insights Adapt to evolving user preferences Overfitting or underfitting Page 27 | SET - 4
- 48. VIGIL TECHNIQUES Page 31 | 01 02 03 DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS 04 MODEL PERFORMANCE MONITORING 05 CENTRALIZED VIEW & DEPENDENCY GRAPHS 06 ALERTING
- 49. MODEL PERFORMANCE MONITORING Continuous evaluation Comparison with expected behavior A/B testing Page 29 | QUALITY
- 50. CHALLENGES Lack of visibility Bottlenecks and performance issues Dependency tracking Page 30 | SET - 5
- 51. VIGIL TECHNIQUES Page 28 | 01 02 03 DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS 04 MODEL PERFORMANCE MONITORING 05 CENTRALIZED VIEW & DEPENDENCY GRAPHS 06 ALERTING
- 52. CENTRALIZED VIEW & DEPENDENCY GRAPHS Expressway Platform Dependency graphs Page 32 | DEVELOPER PRODUCTIVITY
- 53. CHALLENGES Delayed detection Poor system performance Inability to address problems quickly Resource utilization Page 33 | SET - 6
- 54. VIGIL TECHNIQUES Page 34| 01 02 03 DRIFT DETECTION DATA HYGIENE CHECKS COMPONENT SPECIFIC METRICS 04 MODEL PERFORMANCE MONITORING 05 CENTRALIZED VIEW & DEPENDENCY GRAPHS 06 ALERTING
- 55. ALERTING Threshold-based alerts Incident management systems Page 35 | DEVELOPER PRODUCTIVITY, RESOURCE & COST OPTIMIZATION
- 56. IMPACT
- 58. IMPACT 45% Reduction in turnaround time 30% Reduction in system costs Page 37 |
- 59. IMPACT 45% Reduction in turnaround time 30% Reduction in system costs 26% Decrease in system downtime Page 37 |
- 60. QUICK QUESTION: WHAT'S NEW ABOUT THESE PRACTICES? DON'T WE HAVE SOME OF THEM ALREADY? Page 38 |
- 61. EXPRESSWAY MONITORING PLATFORM @ GLANCE
- 62. Expressway objectives 1 Provide a centralized monitoring view of various system components. 2 3 Offer a holistic perspective on component interconnectedness. Streamlined onboarding process. Page 40 |
- 63. Page 41 |
- 64. Page 42 |
- 65. CONCLUSION
- 66. CONCLUSION Page 44 | 1 Effective system component management 2 3 Improved prediction accuracy Enhanced developer productivity 4 Optimized resource utilization 5 Centralized platform for monitoring
- 67. THANKS