Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable

914 views

Published on

When we talk about search and recommendations, we frequently talk about them as polar opposites. In this talk, I will discuss the different technology components we employ when thinking about Search and Recommendations at OpenTable. We will describe how we can use the same technology stack to power the many different modes of decision-making users have, ranging from helping people locate a restaurant they have in mind, to browsing based on a general set of criteria, to suggesting ideas for what might be best for a given context.

Published in: Software

RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable

  1. 1. Unifying the Problem of Search and Recommendations at OpenTable Jeremy Schiff, Ph.D. RecSys 2015 09/20/2015
  2. 2. BEFORE DURING AFTER DINERSRESTAURANTS Understanding & Evolving Attracting & Planning OpenTable: Deliver great experiences at every step, based on who you are Proprietary 3
  3. 3. OpenTable in Numbers • Our network connects diners with more than 32,000 restaurants worldwide. • Our diners have spent approximately $35 billion at our partner restaurants. • OpenTable seats more than 17 million diners each month. • Every month, OpenTable diners write more than 475,000 restaurant reviews 4
  4. 4. OpenTable Data Ecosystem Search (Context & Intent) Restaurant Profile (Decision Confidence) Reservation History (Verifying the Loop) Reviews (Verifying the Loop) User’s Location Search Location Date, Time Query Reviews, Ratings (Overall, Food, Noise Level, etc) Seating Logs Photos, Reviews, Ratings, Menus
  5. 5. OpenTable Data Ecosystem Search (Context & Intent) Restaurant Profile (Decision Confidence) Reservation History (Verifying the Loop) Reviews (Verifying the Loop) User’s Location Search Location Date, Time Query Reviews, Ratings (Overall, Food, Noise Level, etc) Seating Logs Photos, Reviews, Ratings, Menus User Interaction Logs
  6. 6. So what are recommendations?
  7. 7. So what are recommendations?
  8. 8. What’s the Goal Minimizing Engineering Time to Improve The Metric that Matters • Make it Easy to Measure • Make it Easy to Iterate • Reduce Iteration Cycle Times 9
  9. 9. Pick Your Business Metric Revenue, Conversions • OpenTable • Amazon Retention, Engagement • Netflix • Pandora • Spotify 10
  10. 10. Importance of A/B Testing • If you don’t measure it, you can’t improve it • Metrics Drive Behavior • Continued Forward Progress 11
  11. 11. The Optimization Loops Introspect Offline Learning Online Learning Hours Days Weeks 12
  12. 12. The ingredients of a spectacular dining experience… 13
  13. 13. … and a spectacularly bad one 14
  14. 14. Examples of Topics (using MF) 15
  15. 15. Edit via the Header & Footer menu in PowerPoint 1616
  16. 16. LeadTimeLeadTime Distance Distance New York Dallas Lead Time 95% 95%
  17. 17. Query Logs • Effective mechanism for understanding what users are trying to do • Reducing 0 result queries - Anecdote: should we support zipcodes next?
  18. 18. Search to Recommendations Continuum • Common Themes - Ranking always tries to move key metric (like conversion) - Always leverage implicit signals (time of day, day of week, location, etc) - User Control vs. Paradox of Choice Advantage Example Stage Item Count Search User Control $$, French, Takes Credit Card Retrieval Many Browse Use Case Control Great View / Romantic Ranking Many Recommend Data-Driven Flexibility Best around me Ranking Few
  19. 19. Differences in Recommender Usage Right now vs. Planning Cost of Being Wrong Search vs. Recommendations 20
  20. 20. Search vs. Recommendations Collaborative Filtering Models • Personalized • Without Context Search • Leverage Context • Using CF as One of Many Inputs
  21. 21. Search & Recommendation Stack Query Interpretation Retrieval Ranking – Item & Explanation Index Building Context for Query & User Model Building Explanation Content Visualization Collaborative Filters Item / User Metadata 22
  22. 22. Using Context, Frequency & Sentiment • Context - Implicit: Location, Time, Mobile/Web - Explicit: Query • High End Restaurant for Dinner - Low Frequency, High Sentiment • Fast, Mediocre Sushi for Lunch - High Frequency, Moderate Sentiment 23
  23. 23. Offline Models with Limited Data • Minimize Confusing User Experience • Little to No Data - Heuristics  Encoding Product Expectations • Eg: Romantic Dates are not $. Sushi is not good for Breakfast • Limited Data - Data-Informed  Eg: Analyze what Cuisines Users Click on when they Query for Lunch
  24. 24. Offline Models with Significant Data • Compensate for Sparseness • As Signals Improve, Popular -> Personalized • OpenTable Example - Context: User Location, Searched Location, Query, etc. • Learning to Rank - E [ Revenue | Query, Position, Item, User ] - E [ Engagement | Query, Position, Item, User ] - Regression, RankSVM, LambdaMart…
  25. 25. The Metric Gap 26 Training Test Training Error Generalization Error RMSE Precision @ K Stage Example
  26. 26. The Metric Gap 27 Training Error Generalization Error RMSE Precision @ K Stage Example Generalization Gap Training Test
  27. 27. The Metric Gap 28 Training Error Generalization Error RMSE Precision @ K Stage Example Learning to Rank Training Test
  28. 28. The Metric Gap Training Error Generalization Error A/B Metric RMSE Precision @ K Conversion Learning to Rank Stage Example Offline (Hours) Online (Weeks)
  29. 29. The Metric Gap 30 Training Error Generalization Error A/B Metric RMSE Precision @ K Conversion Learning to Rank Stage Example Offline (Hours) Online (Weeks) Offline -> Online Gap
  30. 30. Online Learning – Overview • Naïve Online Learning is A/B testing - Try different sets of parameters, pick the winner • Multi-Arm Bandit - Exploiting the parameter sets that do well - Exploring parameters that we don’t understand well yet (high variance)
  31. 31. Online Learning – Implementation • Iteration Loop - Add Sets of Parameters - Explore vs. Exploit Current Parameters • Validate Online Learning with A/B testing • Note: Tradeoff in Time to Statistical Significance
  32. 32. Example – Start with 1 arm Parameter Metric
  33. 33. Example – Resample arm Parameter Metric
  34. 34. Example – Determine 2nd Arm Parameter Metric
  35. 35. Example – Select Arm Parameter Metric
  36. 36. Example – Improve Arm’s Estimation Parameter Metric
  37. 37. Example – Select Arm Parameter Metric
  38. 38. Example – Learn from Arm Parameter Metric
  39. 39. Example – Determine New Arm Parameter Metric
  40. 40. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark)
  41. 41. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark)
  42. 42. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark) User Interaction Logs (Kafka) Frontends & Backend Services
  43. 43. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark) User Interaction Logs (Kafka) Online Learning Frontends & Backend Services
  44. 44. Training DataFlow Collaborative Filter Service (Realtime) Collaborative Filter HyperParameter Tuning (Batch with Spark) Collaborative Filter Training (Batch with Spark) Search Service (Realtime) Search HyperParameter Tuning (Batch with Spark) Search Training (Batch with Spark) User Interaction Logs (Kafka) Online Learning Frontends & Backend Services A/B Validation
  45. 45. Compelling Recommendations 46
  46. 46. Recommendation Explanations • Amazon • Ness • Netflix • Ness - Social 47
  47. 47. Summarizing Content • Essential for Mobile • Balance Utility With Trust? - Summarize, but surface raw data • Example: - Initially, read every review - Later, use average star rating 48
  48. 48. Summarizing Restaurant Attributes 49
  49. 49. Dish Recommendation • What to try once I have arrived? 50
  50. 50. Thanks! Jeremy Schiff, Ph.D. jschiff@opentable.com Other OpenTable Members @ RecSys: Sudeep Das & Pablo Delgado

×