Lucas Bernardi "Рекомендаційні системи: Погляд із середини"
•Download as PPTX, PDF•
0 likes•112 views
This document discusses recommender systems at Booking.com. It notes that Booking.com has over 28 million listings, 5.6 million homes and apartments, and serves over 1.5 million room nights per day. It discusses that recommender systems are complex, as they must consider the target audience, framing, metrics, data used, and how to define user satisfaction. Recommender systems are also hard, as they must provide recommendations in under 10 milliseconds while accommodating availability and cold start issues. The document emphasizes that offline metrics are just a health check, and that experiments must be run to discover causal effects. It stresses the importance of latency, UX, simplicity over complexity, and moving beyond simple ranking to augmented
Recommended
Recommended
More Related Content
Similar to Lucas Bernardi "Рекомендаційні системи: Погляд із середини"
Similar to Lucas Bernardi "Рекомендаційні системи: Погляд із середини" (20)
More from Lviv Startup Club
More from Lviv Startup Club (20)
Recently uploaded
Recently uploaded (20)
Lucas Bernardi "Рекомендаційні системи: Погляд із середини"
- 1. Recommender Systems: A view from the trenches Lucas Bernardi - Principal Data Scientist lucas.bernardi@booking.com
- 2. ● 28+ million reported listings ● 5.6+ million are homes, apartments and other unique places to stay ● 141+ thousands destinations ● 1.5+ million room nights/day ● Terabytes of data every day ● 200+ Machine Learning Models Deployed Mission to empower people to experience the world
- 3. Recommender Systems. Accommodations Default Ranking Click History Similar Accommodations Different Accommodations Dates Alternative Dates No Dates Available Dates Destinations Plain Recommendations Autocomplete Similar Destinations Multi-city Trip Destinations Nearby Destinations More Filters Districts Room Types Policies
- 4. What do we want to recommend? Recommender Systems are Complex. Target Item Audience Framing KPI Data Feedback Who are we recommending? When? What is the role of the recs? How do we measure effectiveness? What data are we going to use? How do we define user satisfaction?
- 5. Recommender Systems are Hard. Latency Compute recs in less than 10ms Availability Hotels run out of rooms Cold Start New Users New Hotels New Types Everyday Most people only travel once a year Sparsity Complexity Items are related to each other
- 6. Offline metrics are just a Health Check. Offline Metric KPI Relative Improvements
- 7. Offline metrics are just a Health Check. Offline Metric KPI Relative Improvements
- 8. Offline metrics are just a Health Check. Selection Bias Models are trained using a non random sample of the population of interest Offline Metric KPI Relative Improvements Feedback Loops Our Recommender System changes the system Non Stationarity Suddenly everyone goes to Russia It works in my computer but Constraints Hotels run out of rooms
- 9. Offline metrics are just a Health Check. Evaluate Systems as a whole Including the model, the UI, and the audience Expose wrong intuitions More clicks is better, is it? Show Improvement Direction Analysing experiment results we can get concrete ideas about what to do next Discover Causal Effects Experiments are the ultimate piece of evidence for causality Run Experiments Offline Metric KPI Relative Improvements
- 10. Latency A very Important Commercial Metric Another very Important Commercial Metric Latency is Critical.
- 11. Latency is Critical. Precomputed vs Online Latency A very Important Commercial Metric Another very Important Commercial Metric Sparse vs Dense Black Box vs White Box Client Side vs Server Side Work closely with Engineers
- 12. Latency is Critical. Decouple Training from Prediction Centralized Repository of Machine Learned Models Latency A very Important Commercial Metric Another very Important Commercial Metric Every model runs within latency constraints, or does not run at all Wide range of model packaging options Centralized Monitoring of the system as a whole
- 13. UX Matters. 2015
- 16. UX Matters. Work closely with UX Experts, Designers and Copywriters Impact Time 2015 2017
- 17. Simplicity Bias. Tree Based Models Matrix Factorization Neural NetsMarkov ChainsGLMs Complexity
- 18. Simplicity Bias. Tree Based Models Matrix Factorization Neural NetsMarkov ChainsGLMs Simple beats Complex Success Complexity
- 19. Beyond Ranking.
- 20. Beyond Ranking. Augmented Recommender Systems
- 21. A view from the trenches. Run Experiments Latency is Critical UX Matters Simple beats Complex Augmented Recommenders