Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elena Grewal, Data Science Manager, Airbnb at MLconf SF 2016

1,486 views

Published on

Before the Model: How Machine Learning Products Start, with Examples from Airbnb: Often the most important part of building a machine learning product is the formulation of the problem; the most elegant model is rendered useless without the right application and model architecture. Airbnb is an online marketplace for accommodations which has found many interesting applications for machine learning products by taking a data driven approach to investment in Machine learning products. Come hear about how the Airbnb team generates and vets ideas for machine learning products and tailors the product to business problems, with some examples of success and lessons learned along the way.

Published in: Technology

Elena Grewal, Data Science Manager, Airbnb at MLconf SF 2016

  1. 1. Before the Model: How Machine Learning Products Start Elena Grewal / November 11, 2016 / @elenatej
  2. 2. Machine Learning Products @ Airbnb ● Two sided marketplace: Each guest and host are unique. ● ML at its core is around personalization and we use it in all aspects of our product. ● Teams which have ML products: host growth, guest growth, search, pricing, customer support, many more.
  3. 3. Machine Learning at all steps of using Airbnb
  4. 4. Lifecycle of a Machine Learning Product Sizing Opportunity and Scope Model Architecture Data Pipelines and Processing Model Optimization Production Implementation & Evaluation
  5. 5. Initial formulation of the problem is key to success Sizing Opportunity and Scope Data Pipelines and Processing Model Optimization Production Implementation & Evaluation Model Architecture
  6. 6. You need to have the right target metric(s)
  7. 7. Pricing
  8. 8. Way back in 2014 we did an offsite Question: “What do you think is the highest impact project our team can undertake in the next year?” Answer: “Pricing” (we also ate pizza in a baller Airbnb home)
  9. 9. Step 1: Make the Case for Working on Pricing - Highlight all the ways that prices matter - The impact of price on booking + rebooking - Price filter usage - Variations by market 50 slide deck presented to executives Buy time! A project like this takes ~6 months to see any results
  10. 10. Step 1: Make the Case for Working on Pricing
  11. 11. Step 2: Model Architecture - Before Current model predicted price using nearby Airbnb homes - Location, Listing characteristics, Recency This mimicked host behavior
  12. 12. Step 2: Model Architecture - After New metric: Bookings Price suggestion based on probability of booked on given day - Much more flexible - Prices for each date - Interesting UX opportunities Added model layer for adoption of prices. Team of 15 on it now!
  13. 13. Learnings ● Target metric = business outcome (NOT the precision/recall of your model) ● Up front analysis of potential impact of ML product achieves the buy in to work on a project for the needed time ○ More important - you have a better idea of whether it’s the right thing to work on ● User behavior should be considered in model architecture Make time for thinking about machine learning products.
  14. 14. Search
  15. 15. Ranking model could optimize for ‘click through’ But those might not be the right fit for the trip at hand
  16. 16. Ranking model could optimize for guest ‘contact’ But what if the guest is rejected?
  17. 17. Solution: Optimize for a combination of outcomes Machine Learned ranker, using Gradient Boosted Model (GBM)
  18. 18. Learnings ● Target metric = business outcome ○ Traditional target metrics don’t always apply ● Think carefully about the value of different potential business outcomes - solution may be a combination of outcomes
  19. 19. Business Travel
  20. 20. How did it start We noticed that we didn’t have as many business travelers Hypothesis: business travelers have different needs than leisure travelers Can we design products specifically for business travelers?
  21. 21. Step 1: Size the Opportunity Problem: We didn’t know who was a business traveler and who wasn’t. To personalize, we needed to show segments had meaningful differences Collected initial label from 1%
  22. 22. Step 2: Model architecture In this case, our goal was to target business travelers with customized content to increase business travel penetration Simple model, where we predicted if you were a business traveler or not.
  23. 23. Learnings ● Start with hypothesis ● Collect labeled data ● Build a simple product to start - see how it works
  24. 24. Machine Learning Infrastructure
  25. 25. Prior state of the world - Teams develop multiple ML infrastructure with different versions of features - ML in production requires engineering expertise - While many teams are using ML the process is painful Meta before the model
  26. 26. Step 1: Sizing the opportunity & scope 1. Generate ideas for adding 65 new ML products -> multiplier opportunity for building shareable components 2. ‘Back of the envelope’ potential impact on metrics 3. Team proposal with clear deliverables i. # of users participating in ML ii. Reduced time and effort to build ML products iii. Enable easy model eval Feature Discovery Data Acquisition Feature Engineering Model Training Model Scoring
  27. 27. Step 2: In progress! We have added support for Tensorflow and are now supporting a couple models in production with new infra Interesting challenges: how to represent a listing in an extensible way - what features will apply to many different models? This is where we are going in the future.
  28. 28. Step 2: In progress! - Added support for TensorFlow (enabling deep learning at scale) - Interesting challenges: how to represent a listing in an extensible way - what features will apply to many different models? - This is where we are going in the future images text Categorical attributes
  29. 29. Guiding principles Target metric Analyze user behavior Architect Model Opportunity for personalization, impact on metric, user interaction with ML product UX Set up is the most important part. Start simple and iterate. Focus on moving a business metric with ML product
  30. 30. Appendix
  31. 31. Life cycle of a machine learning product ● Opportunity and Scope: Tailoring a data product solution to a business problem (e.g. scoping optimizing improved pricing recommendation model as a solution to hosts setting the right price) ● Model Architecture: Figuring out high-level labels, feature choice and modeling approach ● Data pipelines/processing: Process raw data to features and labels. ● Model implementation: Building v1 of the model - typically done at scale and setting up infrastructure is needed - can be easy with off the shelf packages but harder if bigger ones ● Model optimization: ○ Offline evaluation: Where does the model fall? ○ Model performance: Optimize model to improve overall predictive power to resolve fail points (feature transformation, regularisation, etc) ● Productionizing: Scoring model (online or offline), piping features to model, piping scores to production. ● Online Evaluation: experimentation
  32. 32. For this talk ● Opportunity and Scope: Tailoring a data product solution to a business problem (e.g. scoping optimizing improved pricing recommendation model as a solution to hosts setting the right price) ● Model Architecture: Figuring out high-level labels, feature choice and modeling approach ● Data pipelines/processing: Process raw data to features and labels. ● Model implementation: Building v1 of the model - typically done at scale and setting up infrastructure is needed - can be easy with off the shelf packages but harder if bigger ones ● Model optimization: ○ Offline evaluation: Where does the model fall? ○ Model performance: Optimize model to improve overall predictive power to resolve fail points (feature transformation, regularisation, etc) ● Productionizing: Scoring model (online or offline), piping features to model, piping scores to production. ● Online Evaluation: experiment! Creating the kaggle competition
  33. 33. Why do we care about this ● You can have a great modle optimizing it perfectly but if the framing isn’t right it doesn’t matter ● This is often the most important part of buildling a machine learning product. ● Going to go over a few examples now of where this goes wrong ○ You don’t have the right business problem ○ You aren’t thinking about the way users adopt ○ You don’t know the size of the impact / when to personalize
  34. 34. Ways a ML product can begin ● Structured: You have a metric you’d like to improve - you think of a machine learning product that could help ● Unstructured: You’re playing around with new data, you have some ideas - brainstorm etc A company that builds successful ML products will create incentives and space for innovation in both instances
  35. 35. Importance of a metric ● For any machine learning challenge you need to have a metric that you are optimizing against. Otherwise you will be unable to evaluate the value of a machine learning product to your business and to your users. ● OKR structure ● Bookings over time - we have a goal of 100 how do we get it there? Get a lesson out of every case study E.g. Worth training off explainability
  36. 36. Pricing ● When we first started there was a model that used the most important characteristics about a listing, like the number of rooms and beds, the neighboring properties, and certain amenities, like a parking space or even a pool. And then essential looked at nearby listings with close similarities to suggest a price ● Simulated what users were doing on their own and automated, and you could throw more features and do better clustering ● Didn’t take into account demand, not flexible. and most importantly wasn’t formulated in a way that would optimize against the right metric Add the work up front to prove we should invest 6 months - 12 peopel on it now. All from a data science offsite Indirectly it was whether they accepted or not. Standard recommender is did they take my suggestions. 15 people working on it - huge lever - ux - designers testing those changes. summarziation/highlights It was against the metric of traffice. Things to do in san francisco. SEO. this is what this is for.
  37. 37. Search Slides from Lisa
  38. 38. Biz Travel - Personalization 1) figure out if there is a personalization opportunity. 2) get labeled data. Biz travle. Our hypothesis is that biz travelers are looking for something different than leisure. Is there actually an opportunity there? FIrst you need some labels. Take 1% of traffic and prompt users to tell us if you are traveling for business or leisure. Then you have labeled data. Now we have user attributes and we can see if there is a difference and can we predict if someone is traveling for business or leisure. Trip attributes were also super important. Entire home. Weekdays. Biz travelers usually look at the city level at pseicific address and you’re not starting big and zooming in. Search attributes. Price. Wifi. Then you can build a model and deploy. Show the right business travel promotion. A banner on the booking page to sign up for business travel for the people who are likely for it. A promotion of 100% would cannibalize the promotion space. P5 banners. That gives virality effect where they can sign up> Yahoo is sign up company. Google is its a long tail of small business similar with facebook. Airbnb core product is better for small medium businesses. Next time someone else signs up with the same company its legit and has more than one person. Then we can send an email to those people to ask your travel managers - directly billed to company, find the right listing. Data science is being used to find the long tail that we wouldn’t have found direct sales. Shared itinerary with other people - growth experiment so other people sign up.
  39. 39. Machine learning infrastructure Creating generalized infrastructure so we can do it all ● Making the case for machine learning infrastructure. Machine learning infrastructure. Holistic representation of a listing. Where we are going in the future.
  40. 40. Case studies ● Early motivation is looking at our main metric. Search was a very hand tuned in the past. Pricing. Its not easy! Accuracy is what I can improve but that metric moving is harder. You can improve the performance intrinsically but then you deploy and it looks like the improvement doesn’t lead to the improvement you think of. For example smart pricing you don’t like the suggestion. You’re lowballing. Take into account people’s behavior and how users respond to an improvement. ● The simpler model is often a lot more effective. Better to build something quickly see how it performs and then see if it can be revisit. Can reference the post on coming from academia. ● Ticket routing and user issues - had hard set rules that were very rigid - is you are in this bucket we implemented a probabilistic model that figures automatically what we can do. Go from manual rules to a learned model. Rules failing and then moving to ‘softer’ approaches that are probabilistic. One pattern. We look at signals when the user comes in - surface these links vs those links. Like biz travel. We were ignoring a strong signal that was the text of a ticket. Improve accuracy and also increase volume and optimize precision and recall. Could address CX staffing accordingly. Route more to directly and its ok if they can’t solve it and it takes time to send it back to Airbnb. Impossible to do in previous world. High level talking point - these models give us more flexibility to adapt to the changing dymanics of our business. Set of rules are much harder to tweak. Models give a lot more flexibility. ● Using machine learning to not just build model for predictive performance but to inform analysis. Chao yang on host quality. 30% are worse. Build model on 70%. Learn a model to predict ratings in other bucket. Lead model. PX model. Customizing how users interact with our website using signals available. ● Making the case for machine learning infrastructure. Machine learning infrastructure. Holistic representation of a listing. Where we are going in the future.
  41. 41. Slide Title Here Optional subtitle goes here ● Cereal Entrepreneur: Creative. Embraces constraints. Solution-oriented. Tenacious. ● Be a Host: Collaborative. Anticipates the needs of others. Prepared. Authentic. Listens. ● Embrace the Adventure: Flexible. Risk tolerant. Always learning. Curious. Open-minded. ● Simplify: Distills a problem to its essence. Makes and communicates clear decisions. ● Champion the Mission: Passionate. Committed. Optimistic. ● Every Frame Matters: Thinks holistically. Rigorous about quality. Appreciates the details and prioritizes the right ones.
  42. 42. Slide Title Here Optional subtitle goes here
  43. 43. Slide Title Here Optional subtitle goes here Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed risus arcu, lacinia a aliquet in, vulputate ac turpis. Donec elit elit, consectetur at hendrerit a, porta ac elit. Vivamus efficitur lacus nec ex porttitor lacinia at et nulla.
  44. 44. Your text overlay goes here
  45. 45. Your text overlay goes here Your text overlay goes here
  46. 46. Rausch Hackberry Kazan Babu Lima Beach Ebisu Tirol Foggy Hoff Brand Colors
  47. 47. Product Icons
  48. 48. Iconic Lists
  49. 49. Four Items Iconic List List Item 1 List Item 2 List Item 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit Sed risus arcu, lacinia a aliquet in, vulputate turpis Donec elit elit, consectetur at hendrerit a, porta ac elit Vivamus efficitur lacus nec ex porttitor lacinia at et nulla List Item 4
  50. 50. Five Items Iconic List List Item 1 List Item 2 List Item 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit Sed risus arcu, lacinia a aliquet in, vulputate turpis Donec elit elit, consectetur at hendrerit a, porta ac elit Vivamus efficitur lacus nec ex porttitor lacinia at et nulla List Item 4 List Item 5 Lorem ipsum dolor sit amet, consectetur adipiscing elit
  51. 51. Timelines
  52. 52. Three Items Timeline Time 1 Time 2 Time 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit Sed risus arcu, lacinia a aliquet in, vulputate turpis Donec elit elit, consectetur at hendrerit a, porta ac elit
  53. 53. Four Items Timeline Time 1 Time 2 Time 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit Sed risus arcu, lacinia a aliquet in, vulputate turpis Donec elit elit, consectetur at hendrerit a, porta ac elit Vivamus efficitur lacus nec ex porttitor lacinia at et nulla Time 4
  54. 54. Five Items Timeline Time 1 Time 2 Time 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit Sed risus arcu, lacinia a aliquet in, vulputate turpis Donec elit elit, consectetur at hendrerit a, porta ac elit Vivamus efficitur lacus nec ex porttitor lacinia at et nulla Time 4 Time 5 Lorem ipsum dolor sit amet, consectetur adipiscing elit
  55. 55. Four Items with Box Callout Timeline This is a box callout. Text is fully editable and you can move it around to different dots. Time 1 Time 2 Time 3 Time 4
  56. 56. Six Items with Box Callout Timeline Time 1 Time 2 Time 3 Time 4 Time 5 This is a box callout. Text is fully editable and you can move it around to different dots. Time 6
  57. 57. Map of Airbnb Offices
  58. 58. Portland San Francisco Los Angeles Toronto New York Miami Sao Paulo Dubli n London Paris Barcelona Berlin Milan Copenhagen New Delhi Seoul Beijing Tokyo Sydney Singapore Washington, DC Map of Airbnb Offices 2016
  59. 59. Charts
  60. 60. Column Chart Charts Jan Feb Mar Apr 30 May 10 20 30 40 0 20 25 10 40
  61. 61. Column Chart with Highlight Charts Jan Feb Mar Apr 30 May 10 20 30 40 0 20 25 10 40
  62. 62. Column Chart - Multicolor Charts Jan Feb Mar Apr 30 May 10 20 30 40 0 20 25 10 40
  63. 63. Bar Chart Charts 10 20 30 40 Apr Mar Feb Jan May 0 30 20 25 10 40
  64. 64. Bar Chart with Highlight Charts 10 20 30 40 Apr Mar Feb Jan May 0 30 20 25 10 40
  65. 65. Bar Chart - Multicolor Charts 10 20 30 40 Apr Mar Feb Jan May 0 30 20 25 10 40
  66. 66. Line Chart Charts Jan Feb Mar Apr May 10 20 30 40 0 Item 1 Item 2 Item 3
  67. 67. Line Chart with Data Points Charts Jan Feb Mar Apr May 10 20 30 40 0 Item 1 Item 2 Item 3

×