Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lucas Bernardi "Рекомендаційні системи: Погляд із середини"

14 views

Published on

Data Science Practice

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Lucas Bernardi "Рекомендаційні системи: Погляд із середини"

  1. 1. Recommender Systems: A view from the trenches Lucas Bernardi - Principal Data Scientist lucas.bernardi@booking.com
  2. 2. ● 28+ million reported listings ● 5.6+ million are homes, apartments and other unique places to stay ● 141+ thousands destinations ● 1.5+ million room nights/day ● Terabytes of data every day ● 200+ Machine Learning Models Deployed Mission to empower people to experience the world
  3. 3. Recommender Systems. Accommodations Default Ranking Click History Similar Accommodations Different Accommodations Dates Alternative Dates No Dates Available Dates Destinations Plain Recommendations Autocomplete Similar Destinations Multi-city Trip Destinations Nearby Destinations More Filters Districts Room Types Policies
  4. 4. What do we want to recommend? Recommender Systems are Complex. Target Item Audience Framing KPI Data Feedback Who are we recommending? When? What is the role of the recs? How do we measure effectiveness? What data are we going to use? How do we define user satisfaction?
  5. 5. Recommender Systems are Hard. Latency Compute recs in less than 10ms Availability Hotels run out of rooms Cold Start New Users New Hotels New Types Everyday Most people only travel once a year Sparsity Complexity Items are related to each other
  6. 6. Offline metrics are just a Health Check. Offline Metric KPI Relative Improvements
  7. 7. Offline metrics are just a Health Check. Offline Metric KPI Relative Improvements
  8. 8. Offline metrics are just a Health Check. Selection Bias Models are trained using a non random sample of the population of interest Offline Metric KPI Relative Improvements Feedback Loops Our Recommender System changes the system Non Stationarity Suddenly everyone goes to Russia It works in my computer but Constraints Hotels run out of rooms
  9. 9. Offline metrics are just a Health Check. Evaluate Systems as a whole Including the model, the UI, and the audience Expose wrong intuitions More clicks is better, is it? Show Improvement Direction Analysing experiment results we can get concrete ideas about what to do next Discover Causal Effects Experiments are the ultimate piece of evidence for causality Run Experiments Offline Metric KPI Relative Improvements
  10. 10. Latency A very Important Commercial Metric Another very Important Commercial Metric Latency is Critical.
  11. 11. Latency is Critical. Precomputed vs Online Latency A very Important Commercial Metric Another very Important Commercial Metric Sparse vs Dense Black Box vs White Box Client Side vs Server Side Work closely with Engineers
  12. 12. Latency is Critical. Decouple Training from Prediction Centralized Repository of Machine Learned Models Latency A very Important Commercial Metric Another very Important Commercial Metric Every model runs within latency constraints, or does not run at all Wide range of model packaging options Centralized Monitoring of the system as a whole
  13. 13. UX Matters. 2015
  14. 14. UX Matters. 2015 2017
  15. 15. UX Matters. 2015 2017 Time
  16. 16. UX Matters. Work closely with UX Experts, Designers and Copywriters Impact Time 2015 2017
  17. 17. Simplicity Bias. Tree Based Models Matrix Factorization Neural NetsMarkov ChainsGLMs Complexity
  18. 18. Simplicity Bias. Tree Based Models Matrix Factorization Neural NetsMarkov ChainsGLMs Simple beats Complex Success Complexity
  19. 19. Beyond Ranking.
  20. 20. Beyond Ranking. Augmented Recommender Systems
  21. 21. A view from the trenches. Run Experiments Latency is Critical UX Matters Simple beats Complex Augmented Recommenders
  22. 22. Thank you! booking.ai

×