Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Recommendation Architecture
Jeremy Schiff
MLConf 2015
03/27/2015
BEFORE	
   DURING	
  	
   AFTER	
  
DINERS	
  RESTAURANTS	
  
Understanding	
  
&	
  Evolving	
  
A2rac4ng	
  &	
  
Planni...
OpenTable in Numbers
• Our network connects diners with more than
32,000 restaurants worldwide.
• Our diners have spent mo...
Recommendations 
>>
Collaborative Filtering
4	
  
So what are recommendations?
5	
  
Building Recommendation Systems
• Importance of A/B
Testing
• Generating
Recommendations
• Recommendation
Explanations


6...
What’s the Goal

Minimizing Engineering Time to Improve The
Metric that Matters

• Make it Easy to Measure
• Make it Easy ...
Importance of A/B Testing
• If you don’t measure it,
you can’t improve it

 
 
 

• Metrics Drive Behavior
• Continued For...
Pick Your Business Metric
Revenue, Conversions 
• OpenTable 
• Amazon 

Engagement
• Netflix
• Pandora
• Spotify
9	
  
Measuring & The Iteration Loop
A/B	
  	
  
Tes4ng	
  
Weeks	
  
Measure	
  
10	
  
Measuring & The Iteration Loop
Op4mize	
  	
  
Models	
  
A/B	
  	
  
Tes4ng	
  
Days	
   Weeks	
  
Predict	
   Measure	
 ...
Measuring & The Iteration Loop
Analyze	
  &	
  	
  
Introspect	
  
Op4mize	
  	
  
Models	
  
A/B	
  	
  
Tes4ng	
  
Hours...
Ranking Objectives
Objectives: 
• Training Error
- Minimize Loss Function
§  Often Convex
• Generalization Error
- Precis...
Training, Generalization, and Online Error
• Training: Train on your specific dataset
- Dealing with Sparseness
• Test/Gene...
Fundamental Differences in Usage

Right now vs. Planning

Cost of Being Wrong

Search vs. Recommendations


15	
  
Recommendation Stack
Query	
  Interpreta4on	
  
Retrieval	
  
Ranking	
  –	
  Item	
  &	
  Explana4on	
  
Index	
  
Buildi...
Using Context, Frequency & Sentiment
•  Context
- Implicit: Location, Time, Mobile/Web
- Explicit: Query
•  High End Resta...
How to use this data
• Frequency Data: 
- General: Popularity
- Personalized: Implicit CF
• Sentiment Data: 
- General: Go...
Ranking
Phase 1: Bootstrap through heuristics

Phase 2: Learn to Rank
•  Many models
- E [ Revenue | Query, Position, Item...
Training Example
•  Context Free (Collaborative Filtering)
- Train for Content Based and Collaborative Filtering models.
-...
Training DataFlow
Collabora4ve	
  Filter	
  
Service	
  
(Real4me)	
  
Collabora4ve	
  Filter	
  
HyperParameter	
  Tuning...
Training DataFlow
Collabora4ve	
  Filter	
  
Service	
  
(Real4me)	
  
Collabora4ve	
  Filter	
  
HyperParameter	
  Tuning...
Training DataFlow
Collabora4ve	
  Filter	
  
Service	
  
(Real4me)	
  
Collabora4ve	
  Filter	
  
HyperParameter	
  Tuning...
Compelling Recommendations
24	
  
Recommendation Explanations
•  Amazon

•  Ness
•  Netflix

•  Ness - Social
25	
  
Summarizing Content

• Essential for Mobile
• Balance Utility With Trust?
- Summarize, but surface raw
data

• Example: 
-...
Summarizing Restaurant Attributes
27	
  
Dish Recommendation
• What to try once I have arrived?

28	
  
Edit	
  via	
  the	
  Header	
  &	
  Footer	
  menu	
  in	
  
PowerPoint	
  
29	
  29	
  
Analyzing Review Content
30	
  
The ingredients of a spectacular

dining experience…
31	
  
… and a spectacularly bad one
32	
  
Content Features

Pandora
• Music Genome Project

Natural Language Processing
• Topics & Tags
33	
  
Topic Modeling Methods
We applied two main topic
modeling methods:
• Latent Dirichlet Allocation
(LDA) 
- (Blei et al. 200...
The food was great! I loved the view of the
sailboats.

Bag of Words Model
food	
   great	
   chicken	
   sailboat	
   vie...
Topics with NMF using TF-IDF
Word	
  1	
   Word	
  …	
   Word	
  N	
  
Review	
  1	
   0.8	
   0.9	
   0	
  
Review	
  …	
...
Describing Restaurants as Topics
Each	
  review	
  for	
  a	
  
given	
  restaurant	
  	
  
has	
  certain	
  topic	
  
di...
Examples of Topics
38	
  
Varying Topic By Region
•  San Francisco
•  `

•  London
•  Chicago

•  New York
39	
  
Building Recommendation Systems
• Importance of A/B
Testing
• Generating
Recommendations
• Recommendation
Explanations

40...
Thanks!
Jeremy Schiff
jschiff@opentable.com
Upcoming SlideShare
Loading in …5
×

Jeremy Schiff, Senior Manager, Data Science, OpenTable at MLconf NYC

1,775 views

Published on

Recommendation Architecture: Understanding the Components of a Personalized Recommendation System: When we typically talk about recommendation systems, we focus on specific novel algorithms and formulations for performing collaborative filtering. However, building a system to recommend items to a user in a personalized way often involves many more components than just a collaborative filter; it requires a much broader ecosystem of functionality, tools, and development pipelines. This presentation will discuss an holistic approach to building recommendation systems including 1) how A/B testing works with machine learning to iterate toward better recommendations, 2) how to couple an information-retrieval based search stack with collaborative filtering to capture user intent in a personalized way, and 3) making recommendations more relevant and interpretable.

Published in: Technology
  • Be the first to comment

Jeremy Schiff, Senior Manager, Data Science, OpenTable at MLconf NYC

  1. 1. Recommendation Architecture Jeremy Schiff MLConf 2015 03/27/2015
  2. 2. BEFORE   DURING     AFTER   DINERS  RESTAURANTS   Understanding   &  Evolving   A2rac4ng  &   Planning   OpenTable: Deliver great experiences at 
 every step, based on who you are Proprietary   2  
  3. 3. OpenTable in Numbers • Our network connects diners with more than 32,000 restaurants worldwide. • Our diners have spent more than $30 billion at our partner restaurants. • OpenTable seats more than 16 million diners each month. • Every month, OpenTable diners write more than 450,000 restaurant reviews 3  
  4. 4. Recommendations >> Collaborative Filtering 4  
  5. 5. So what are recommendations? 5  
  6. 6. Building Recommendation Systems • Importance of A/B Testing • Generating Recommendations • Recommendation Explanations 6  
  7. 7. What’s the Goal Minimizing Engineering Time to Improve The Metric that Matters • Make it Easy to Measure • Make it Easy to Iterate • Reduce Iteration Cycle Times 7  
  8. 8. Importance of A/B Testing • If you don’t measure it, you can’t improve it • Metrics Drive Behavior • Continued Forward Progress 8  
  9. 9. Pick Your Business Metric Revenue, Conversions • OpenTable • Amazon Engagement • Netflix • Pandora • Spotify 9  
  10. 10. Measuring & The Iteration Loop A/B     Tes4ng   Weeks   Measure   10  
  11. 11. Measuring & The Iteration Loop Op4mize     Models   A/B     Tes4ng   Days   Weeks   Predict   Measure   11  
  12. 12. Measuring & The Iteration Loop Analyze  &     Introspect   Op4mize     Models   A/B     Tes4ng   Hours   Days   Weeks   Insights   Predict   Measure   12  
  13. 13. Ranking Objectives Objectives: • Training Error - Minimize Loss Function §  Often Convex • Generalization Error - Precision at K • A/B Metric - Conversion / Engagement 13  
  14. 14. Training, Generalization, and Online Error • Training: Train on your specific dataset - Dealing with Sparseness • Test/Generalization: How does it generalize to unseen data? - Hyper-Parameter Tuning • Online: How does it perform in the wild - Model interaction effects between recommend items (diversity)
  15. 15. Fundamental Differences in Usage Right now vs. Planning Cost of Being Wrong Search vs. Recommendations 15  
  16. 16. Recommendation Stack Query  Interpreta4on   Retrieval   Ranking  –  Item  &  Explana4on   Index   Building   Context  for  Query  &  User     Model   Building   Explana4on   Content   Visualiza4on   Collabora4ve   Filters   Item  /  User   Metadata   16  
  17. 17. Using Context, Frequency & Sentiment •  Context - Implicit: Location, Time, Mobile/Web - Explicit: Query •  High End Restaurant for Dinner - Low Frequency, High Sentiment •  Fast, Mediocre Sushi for Lunch - High Frequency, Moderate Sentiment 17  
  18. 18. How to use this data • Frequency Data: - General: Popularity - Personalized: Implicit CF • Sentiment Data: - General: Good Experience - Personalized: Explicit CF • Good Recommendation - Use both to drive your Business Metric 18  
  19. 19. Ranking Phase 1: Bootstrap through heuristics Phase 2: Learn to Rank •  Many models - E [ Revenue | Query, Position, Item, User ] - E [ Engagement | Query, Position, Item, User ] - Regression, RankSVM, LambdaMart… •  Modeling Diversity is Important 19  
  20. 20. Training Example •  Context Free (Collaborative Filtering) - Train for Content Based and Collaborative Filtering models. - Create an Ensemble Model - Perform Hyper-Parameter Tuning for each model •  With Context (Search) - Train a model using query (implicit & explicit) §  Includes Context-Free Model - Perform Hyper-Parameter Tuning •  Evaluate Model using A/B - Change models, objective functions, etc.
  21. 21. Training DataFlow Collabora4ve  Filter   Service   (Real4me)   Collabora4ve  Filter   HyperParameter  Tuning     (Batch  with  Spark)   Collabora4ve  Filter   Training   (Batch  with  Spark)  
  22. 22. Training DataFlow Collabora4ve  Filter   Service   (Real4me)   Collabora4ve  Filter   HyperParameter  Tuning     (Batch  with  Spark)   Collabora4ve  Filter   Training   (Batch  with  Spark)   Search  Service   (Real4me)   Search  HyperParameter   Tuning     (Batch  with  Spark)   Search  Training   (Batch  with  Spark)  
  23. 23. Training DataFlow Collabora4ve  Filter   Service   (Real4me)   Collabora4ve  Filter   HyperParameter  Tuning     (Batch  with  Spark)   Collabora4ve  Filter   Training   (Batch  with  Spark)   Search  Service   (Real4me)   Search  HyperParameter   Tuning     (Batch  with  Spark)   Search  Training   (Batch  with  Spark)   User  Interac4on  Logs   (Ka_a)   A/B  Tes4ng  Dashboards   Other  Services  
  24. 24. Compelling Recommendations 24  
  25. 25. Recommendation Explanations •  Amazon •  Ness •  Netflix •  Ness - Social 25  
  26. 26. Summarizing Content • Essential for Mobile • Balance Utility With Trust? - Summarize, but surface raw data • Example: - Initially, read every review - Later, use average star rating 26  
  27. 27. Summarizing Restaurant Attributes 27  
  28. 28. Dish Recommendation • What to try once I have arrived? 28  
  29. 29. Edit  via  the  Header  &  Footer  menu  in   PowerPoint   29  29  
  30. 30. Analyzing Review Content 30  
  31. 31. The ingredients of a spectacular
 dining experience… 31  
  32. 32. … and a spectacularly bad one 32  
  33. 33. Content Features Pandora • Music Genome Project Natural Language Processing • Topics & Tags 33  
  34. 34. Topic Modeling Methods We applied two main topic modeling methods: • Latent Dirichlet Allocation (LDA) - (Blei et al. 2003) • Non-negative Matrix Factorization (NMF) - (Aurora et al. 2012) 34  
  35. 35. The food was great! I loved the view of the sailboats. Bag of Words Model food   great   chicken   sailboat   view   service   1   1   0   1   1   0   35  
  36. 36. Topics with NMF using TF-IDF Word  1   Word  …   Word  N   Review  1   0.8   0.9   0   Review  …   0.6   0   0.8   Review  N   0.9   0   0.8   Reviews   X   Words   Reviews   X   Topics   Topics   X   Words   36  
  37. 37. Describing Restaurants as Topics Each  review  for  a   given  restaurant     has  certain  topic   distribuCon   Combining  them,   we  idenCfy  the  top   topics  for  that   restaurant.   Topic 01! Topic 02! Topic 03! Topic 04! Topic 05! Topic 01! Topic 02! Topic 03! Topic 04! Topic 05! Topic 01! Topic 02! Topic 03! Topic 04! Topic 05! review  1   review  2   review  N   .   .   .   Topic 01! Topic 02! Topic 03! Topic 04! Topic 05! Restaurant   37  
  38. 38. Examples of Topics 38  
  39. 39. Varying Topic By Region •  San Francisco •  ` •  London •  Chicago •  New York 39  
  40. 40. Building Recommendation Systems • Importance of A/B Testing • Generating Recommendations • Recommendation Explanations 40  
  41. 41. Thanks! Jeremy Schiff jschiff@opentable.com

×