Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Recommendations at Zillow
Similar to Recommendations at Zillow (20)
Recently uploaded
Recently uploaded (20)
Recommendations at Zillow
- 1. 1 ZILLOW | TRULIA | STREETEASY | HOTPADS ZILLOW | TRULIA | STREETEASY | HOTPADS December 14, 2016 Shruti Kamath (shrutik) & Nicholas Stevens (nicholass) Recommendations at Zillow
- 3. 3 Zillow Group’s mission stretches across brands Build the world's largest, most trusted and vibrant home-related marketplace.
- 6. 6 Goal: Help all of our users seamlessly, efficiently & delightfully discover homes that match her family’s needs & lifestyle.
- 7. 7 Tons of Zillow product areas for recommendations ● Communication Channels ○ Email ○ SMS ○ Push ● Home Details Pages ○ Homes like this ● Search ○ Personalized Search ● Home Page ● Beyond home recommendations ○ Content Recommendations ○ Mortgages ○ Agents
- 8. 8 Tons of Zillow product areas for recommendations ● Communication Channels ○ Email ○ SMS ○ Push ● Home Details Pages ○ Homes like this ● Search ○ Personalized Search ● Home Page ● Beyond home recommendations ○ Content Recommendations ○ Mortgages ○ Agents
- 9. 9 Why now? important user flows + user and home data + proven algorithms
- 11. 11 We want to test & validate models quickly Offline Testing Online A/B Testing 100% Data set Evaluation Metric Models Infrastructure Metrics
- 12. 12 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving
- 13. 13 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving ● User Interactions (Clicks, saves, photo views, etc) ● Home Details (Price, Location, beds, etc)
- 14. 14 The shape of Zillow’s User-Item matrix is unique Users Items Traditional
- 15. 15 The shape of Zillow’s User-Item matrix is unique Users Traditional Zillow Items
- 16. 16 The shape of Zillow’s User-Item matrix is unique Users Traditional Items
- 17. 17 The shape of Zillow’s User-Item matrix is unique Users Zillow Items
- 18. 18 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Was this relevant?
- 19. 19 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving
- 20. 20 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Feature Description uid unique id of user pid Property id first_visit timestamp or 0 num_views sigmoid(#views) time_spent time on page num_contacts # leads sent num_saves # saves on zpid num_shares # shares on zpid num_photos # photos viewed
- 21. 21 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Users Homes Relevant/Not Relevant [0...1]
- 22. 22 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving
- 23. 23 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving ● Similar user preferences ● Needs: user & items interaction ● Doesn’t need: domain knowledge
- 24. 24 Collaborative Filtering example USERAUSERB CLICK HISTORY (or other signals of interest) MACHINE SUGGESTIONS
- 25. 25 Many options to test collaborative filtering ● Open Source: ○ Spark’s collab filtering via ALS (spark.mllib.recommendation). ○ LensKit: Java (documentation) ○ PredictionIO: Spark, HBase and Spray. Acquired by SalesForce (site) ○ Seldon: Java based, built on Spark (site) ○ GraphLab: SDK available in Java, Python, C++, etc (guide) ● Recs as a Service: Amazon ML, GraphLab, Google ML, Azure, etc ● Algorithms from papers: ○ Restricted Boltzman Machines & SVD++ (what Netflix was using in 2014) ○ Trulia’s Wedge Counting
- 26. 26 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving ● Similar user preferences ● Needs: user & items interaction ● Doesn’t need: domain knowledge
- 27. 27 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving ● Specific home features ● Needs: domain knowledge ● Doesn’t need: interaction
- 28. 28 Content-based modeling example CATALOG OF TAGGED ITEMSSHOWED INTEREST MACHINE SUGGESTIONS Fruit, Berry, Red Fruit, Berry, Red
- 29. 29 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Type Features (categorical variables) Bath 0_bath, 0.5_bath, 1_Bath, 1.5_bath, 2_bath, 2.5_bath, 3_bath Bed 0_bed, 1_bed, 2_bed, 3_bed, 4_bed, 5_bed Price 100_125_price, 125_150_price, 150_175_price Use Code condo, single_family, farm_land Zipcode zip_98109
- 30. Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving ● Types of properties ● Stage of buying (early vs late) ● Region differences ● Communication preferences
- 31. 31 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving
- 32. 32 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Email: New Rec’s! (A) Home1 ModelX Home2 ModelX Home3 ModelX Email: New Rec’s! (B) Home1 ModelY Home2 ModelY Home3 ModelY v
- 33. 33 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Email: New Rec’s! Home1 ModelX Home3 ModelX Email: New Rec’s! Home1 ModelY Home2 ModelY Home3 ModelY v Home2 ModelX
- 34. 34 Background on Interleaving ● Blog introduction to interleaving, 2008: http://glinden.blogspot.com/2008/11/testing-rankers-by-interleaving-search.html ● Paper comparing interleaving to other, more traditional methods (2010): https://www.microsoft.com/en-us/research/wp-content/uploads/2010/07/fp146-radlinski.pdf ○ 5,000 judged queries ~= 50,000 user impressions. ● Paper with a comprehensive review of interleaving in 2012: http://www.cs.cornell.edu/people/tj/publications/chapelle_etal_12a.pdf ● Comparison of specific interleaving algorithms (click aggregation schemes, 2013): https://www.microsoft.com/en-us/research/wp-content/uploads/2013/02/Radlinski_Optimized_WSDM2013.pdf.pdf ● Big guide on evaluation techniques, including interleaving: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/06/ftir-online-evaluation-final-journal.pdf
- 35. 35 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving Email: New Rec’s! Home1 ModelX Home3 ModelZ Email: New Rec’s! Home1 ModelZ Home2 ModelY Home3 ModelY v Home2 ModelX
- 36. 36 Components of Home Rec’s Data set Relevant/Not Relevant Evaluation Metric Collaborative Filtering Content-based Other A/B Infrastructure Serving Infrastructure Metrics Interleaving
- 38. Home Recommendations architecture RECOMMENDATION API (Java) Zillow Group Data Lake (S3 / Kinesis) Property Featurization (Spark EMR) User Profiles (Spark EMR) Ranking (Spark EMR) Wedge Counting Collaborative Filtering (Spark EMR) Property Aggregate Features (Spark EMR) Data Collection Systems (Java/Python/SQL)
- 39. 39 Airflow
- 41. 41 Machine Learning pipeline - ETL
- 42. 42 Our top data sources Data Source Volume per day Location Frequency User-Property Interactions Google Analytics & Kinesis stream 1 TB S3 Batch & Realtime Property Third-Party, County, Banks, Users, Listings 12 GB S3 Realtime Posting History MLS, Brokers, Listings 12 GB S3 Realtime
- 43. 43 Data processing timestamp Google Analytics User Events training validation test Aggregated user events property Subset 1 Subset n Filter subset 1 n 1 n uid zpid timestamp price device event event type event label . . . uid zpid dwell time num_shares . . uid zpid dwell time ... zipid countyid ...
- 44. 44 Subset 1 Subset n Filter region uid zpid dwell time ... zipid countyid ... Filter zuids, zpids Subset 1 Subset n Training data & Subset 1 Subset n Prediction data Data processing
- 45. 45 Scala is a Spark first class citizen ● Functional ○ Composability (building blocks) ○ Easy to Parallelize ● Type Safe ● Ship features and improvements quickly ● Deployment of code through the JVM is easier ● Using Datasets API ○ Compile-time safety for syntax and analysis errors ○ Structured Streaming
- 46. 46 Datasets (Spark 2.0) is very promising Pros: ● Compile time safety. ● Domain specific operations ● Composable (output of one process is the input into another). ● Uses Catalyst Optimizer ● Structured Streaming Cons: ● Dataset API’s are still in experimental phase ● Only supported for primitive types and case classes. ● Joins are not type-safe. ● Complex types such as Iterators are not supported in the current version and require custom encoders
- 47. 47 We still use RDDs where Dataset is challenging Dataset RDD (Resilient Distributed Dataset)
- 48. 48 Machine Learning pipeline - Training
- 49. 49 Machine Learning pipeline - Training
- 51. 51 Machine Learning pipeline - Prediction
- 52. 52 Machine Learning pipeline - Prediction
- 54. 54 Machine Learning pipeline - Evaluation
- 56. 56 Offline Metrics Description Precision rk = # recommended properties in test set in top k Recall n = total properties in the test set Freshness # of listings recommended with modified date < d days old in top k Recs Coverage # unique listings recommended across all users in the top k Our top model evaluation metrics
- 58. 58 Other problems we’re tackling ● Classifiers for listing descriptions ● Deep learning on listing images ● Structured streaming on Spark 2.0 ● Real-time scoring We’re hiring for Data Scientists & Engineers! zillow.com/jobs Questions? shrutik@zillow & nicholass@zillow
- 59. 59 ● Imri S ● Alex C ● Shruti K ● Nicholas S Recommendations team Data Science ● Hao X ● Purvag P ● Kurtis M ● XiaoXi G ● Ming-Li L ● Jason C ● Andy W ● Eric T ● Andrew M And others! Personalization Product team
- 60. 60 That’s it!