More Related Content
Similar to RecSysOps: Best Practices for Operating a Large-Scale Recommender System
RecSysOps: Best Practices for Operating a Large-Scale Recommender System
- 1. RecSysOps: Best Practices forOperating a Large-Scale Recommender System Ehsan (Mohammad) Saberian, Justin Basilico RecSys 2021 2021-09-27 @ehsan_saberian, @JustinBasilico, @NetflixResearch
- 2. Large scale RecSysis a complex operation
- 3. RecSys environment isdynamic It changes every second!
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10. How to ensurethat our RecSys is working correctly?
- 11. RecSysOps: Lessons we learnedwhile operating a large RecSys
- 12. Benefits: Reduce firefighting time Focuson innovation Build trust with our stakeholders
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. Detect issues quickly Themost challenging part There are endless potential issues Some of them we don’t know yet!
- 19. Detection lesson 1: Implementall the known best practices Unit test, Integration test MLOps: data/metrics check CICD, regular retraining
- 20. Detection lesson 1: Implementall the known best practices Unit test, Integration test MLOps: data/metrics check CICD, regular retraining
- 21. Detection lesson 2: Monitorend-to-end your own way
- 22. Detection lesson 2: Monitorend-to-end your own way Don’t rely only on partner teams’ audits What does correct data look like from your perspective?
- 23. Detection lesson 3: Understandyour stakeholders’ concerns Stakeholders: members and items
- 24. Detection lesson 3.1: Everytime a member plays something that is ranked low by the model; it is a potential issue
- 25. Detection lesson 3.1: Everytime a member plays something that is ranked low by the model; it is a potential issue Monitor and analyze them Get inspiration for future innovations
- 26. Detection lesson 3.2: Engagewith teams responsible for items and understand their concerns
- 27. Detection lesson 3.2: Engagewith teams responsible for items and understand their concerns Is an item cold-started properly? Is production bias hurting an item?
- 28. Detection lesson 3.2: Engagewith teams responsible for items and understand their concerns Build tools to detect their concerns and integrated them in your system
- 29.
- 30. Can you predict issuesbefore they happen?
- 31. Netflix case: Is itpossible to predict if an item is going to cold-start properly 7 days before its launch date?
- 32. Yes, we cantrain a model to predict production model’s behaviour on day of launch Flag any item with unexpected prediction and investigate
- 33. Prediction Lesson: Try topredict issues before they happen
- 34.
- 35. Step 1: Reproduce issuein isolation
- 36. Step 1: Reproduce issuein isolation Need sufficient advanced logging
- 37. Step 2: Input dataissue or model issue?
- 38. Step 2.1: Input dataissue? Are input values right?
- 39. Step 2.1: Input dataissue? Are input values right? Trick: use similar items or members to estimate range of typical values
- 40. Step 2.1: Input dataissue? Are input values right? Example: language of an item was set up incorrectly
- 41. Step 2.2: Model issue? Needto inspect/interpret model
- 42. Step 2.2: Model issue? Needto inspect/interpret model There are many tools, SHAP, LIME ..
- 43. Step 2.2: Model issue? Needto inspect/interpret model Example: missing values were handled incorrectly
- 44. Diagnosis lessons: Set uplogging to reproduce issue Develop tools to check validity of inputs Develop tools to inspect models
- 45.
- 46. It’s like softwareengineering Hotfixes and long term solutions
- 47. Hotfixes in MLare challenging! Models are highly optimized and hotfix modification will lead to suboptimality
- 48. Resolution lesson 1: Havea collection of hotfixes ready Understand their costs and trade-offs
- 49. Resolution lesson 2: Withevery issue, make RecsysOps better
- 50.
- 51. Final lesson: Make RecSysOpsfrictionless
- 52. Run checks ona regular basis If human judgment is needed, make all required information ready Be able to deploy hotfixes with couple of clicks
- 53.