The document discusses the processes involved in developing machine learning products at Airbnb, emphasizing the importance of personalization and the careful formulation of problems before implementation. It covers various stages such as opportunity sizing, model architecture, data processing, and the need for a well-defined target metric that aligns with business outcomes. Key learnings highlight the significance of understanding user behavior and optimizing for various business metrics when designing machine learning solutions.

1. Before the Model:
How Machine Learning
Products Start
Elena Grewal / November 11, 2016 / @elenatej

2. Machine Learning Products @ Airbnb
● Two sided marketplace: Each guest and host are unique.
● ML at its core is around personalization and we use it in all
aspects of our product.
● Teams which have ML products: host growth, guest growth,
search, pricing, customer support, many more.

3. Machine Learning at all steps of using Airbnb

4. Lifecycle of a Machine Learning Product
Sizing
Opportunity and
Scope
Model
Architecture
Data Pipelines
and Processing
Model
Optimization
Production
Implementation
& Evaluation

5. Initial formulation of the problem is key to success
Sizing
Opportunity and
Scope
Data Pipelines
and Processing
Model
Optimization
Production
Implementation
& Evaluation
Model
Architecture

6. You need to have the
right target metric(s)

7. Pricing

9. Way back in 2014 we did an offsite
Question: “What do you think is the highest impact project our
team can undertake in the next year?”
Answer: “Pricing”
(we also ate pizza in a baller Airbnb home)

10. Step 1: Make the Case for Working on Pricing
- Highlight all the ways that prices matter
- The impact of price on booking + rebooking
- Price filter usage
- Variations by market
50 slide deck presented to executives
Buy time! A project like this takes ~6 months to see any results

11. Step 1: Make the Case for Working on Pricing

12. Step 2: Model Architecture - Before
Current model predicted price using nearby Airbnb homes
- Location, Listing characteristics, Recency
This mimicked host behavior

13. Step 2: Model Architecture - After
New metric: Bookings
Price suggestion based on probability of booked on given day
- Much more flexible
- Prices for each date
- Interesting UX opportunities
Added model layer for adoption of prices. Team of 15 on it now!

14. Learnings
● Target metric = business outcome (NOT the precision/recall of
your model)
● Up front analysis of potential impact of ML product achieves
the buy in to work on a project for the needed time
○ More important - you have a better idea of whether it’s the
right thing to work on
● User behavior should be considered in model architecture
Make time for thinking about machine learning products.

15. Search

17. Ranking model could optimize for ‘click through’
But those might
not be the right fit
for the trip at hand

18. Ranking model could optimize for guest ‘contact’
But what if the
guest is rejected?

19. Solution: Optimize for a combination of outcomes
Machine Learned ranker, using Gradient Boosted Model (GBM)

20. Learnings
● Target metric = business outcome
○ Traditional target metrics don’t always apply
● Think carefully about the value of different potential business
outcomes - solution may be a combination of outcomes

21. Business Travel

22. How did it start
We noticed that we didn’t have as many business travelers
Hypothesis: business travelers have different needs than leisure
travelers
Can we design products specifically for business travelers?

24. Step 1: Size the Opportunity
Problem: We didn’t know who was a business traveler and who
wasn’t.
To personalize, we needed to show segments had meaningful
differences
Collected initial label from 1%

25. Step 2: Model architecture
In this case, our goal was to target business travelers with
customized content to increase business travel penetration
Simple model, where we predicted if you were a business traveler
or not.

26. Learnings
● Start with hypothesis
● Collect labeled data
● Build a simple product to start - see how it works

27. Machine Learning Infrastructure

28. Prior state of the world
- Teams develop multiple ML infrastructure with different
versions of features
- ML in production requires engineering expertise
- While many teams are using ML the process is painful
Meta before the model

29. Step 1: Sizing the opportunity & scope
1. Generate ideas for adding 65 new ML products -> multiplier
opportunity for building shareable components
2. ‘Back of the envelope’ potential impact on metrics
3. Team proposal with clear deliverables
i. # of users participating in ML
ii. Reduced time and effort to build ML products
iii. Enable easy model eval
Feature
Discovery
Data
Acquisition
Feature
Engineering
Model Training Model Scoring

30. Step 2: In progress!
We have added support for Tensorflow and are now supporting a
couple models in production with new infra
Interesting challenges: how to represent a listing in an extensible
way - what features will apply to many different models?
This is where we are going in the future.

31. Step 2: In
progress!
- Added support for
TensorFlow
(enabling deep
learning at scale)
- Interesting
challenges: how to
represent a listing in
an extensible way -
what features will
apply to many
different models?
- This is where we are
going in the future
images
text
Categorical
attributes

32. Guiding principles
Target metric Analyze user behavior Architect Model
Opportunity for
personalization, impact on
metric, user interaction with
ML product UX
Set up is the most
important part.
Start simple and iterate.
Focus on moving a business
metric with ML product

34. Appendix

35. Life cycle of a machine learning product
● Opportunity and Scope: Tailoring a data product solution to a business problem (e.g. scoping optimizing
improved pricing recommendation model as a solution to hosts setting the right price)
● Model Architecture: Figuring out high-level labels, feature choice and modeling approach
● Data pipelines/processing: Process raw data to features and labels.
● Model implementation: Building v1 of the model - typically done at scale and setting up infrastructure is
needed - can be easy with off the shelf packages but harder if bigger ones
● Model optimization:
○ Offline evaluation: Where does the model fall?
○ Model performance: Optimize model to improve overall predictive power to resolve fail points
(feature transformation, regularisation, etc)
● Productionizing: Scoring model (online or offline), piping features to model, piping scores to production.
● Online Evaluation: experimentation

36. For this talk
● Opportunity and Scope: Tailoring a data product solution to a business problem (e.g. scoping optimizing
improved pricing recommendation model as a solution to hosts setting the right price)
● Model Architecture: Figuring out high-level labels, feature choice and modeling approach
● Data pipelines/processing: Process raw data to features and labels.
● Model implementation: Building v1 of the model - typically done at scale and setting up infrastructure is
needed - can be easy with off the shelf packages but harder if bigger ones
● Model optimization:
○ Offline evaluation: Where does the model fall?
○ Model performance: Optimize model to improve overall predictive power to resolve fail points
(feature transformation, regularisation, etc)
● Productionizing: Scoring model (online or offline), piping features to model, piping scores to production.
● Online Evaluation: experiment!
Creating the kaggle competition

37. Why do we care about this
● You can have a great modle optimizing it perfectly but if the framing isn’t right it doesn’t matter
● This is often the most important part of buildling a machine learning product.
● Going to go over a few examples now of where this goes wrong
○ You don’t have the right business problem
○ You aren’t thinking about the way users adopt
○ You don’t know the size of the impact / when to personalize

38. Ways a ML product can begin
● Structured: You have a metric you’d like to improve - you think of a machine learning product that could
help
● Unstructured: You’re playing around with new data, you have some ideas - brainstorm etc
A company that builds successful ML products will create incentives and space for innovation in both
instances

39. Importance of a metric
● For any machine learning challenge you need to have a metric that you are optimizing against.
Otherwise you will be unable to evaluate the value of a machine learning product to your business and
to your users.
● OKR structure
● Bookings over time - we have a goal of 100 how do we get it there?
Get a lesson out of every case study
E.g. Worth training off explainability

40. Pricing
● When we first started there was a model that used the most important characteristics about a listing, like
the number of rooms and beds, the neighboring properties, and certain amenities, like a parking space or
even a pool. And then essential looked at nearby listings with close similarities to suggest a price
● Simulated what users were doing on their own and automated, and you could throw more features and do
better clustering
● Didn’t take into account demand, not flexible. and most importantly wasn’t formulated in a way that
would optimize against the right metric
Add the work up front to prove we should invest
6 months - 12 peopel on it now. All from a data science offsite
Indirectly it was whether they accepted or not. Standard recommender is did they take my suggestions.
15 people working on it - huge lever - ux - designers testing those changes.
summarziation/highlights
It was against the metric of traffice. Things to do in san francisco. SEO. this is what this is for.

41. Search
Slides from Lisa

42. Biz Travel - Personalization
1) figure out if there is a personalization opportunity. 2) get labeled data. Biz travle. Our hypothesis is that biz travelers are looking for
something different than leisure. Is there actually an opportunity there? FIrst you need some labels. Take 1% of traffic and prompt users
to tell us if you are traveling for business or leisure. Then you have labeled data. Now we have user attributes and we can see if there is a
difference and can we predict if someone is traveling for business or leisure. Trip attributes were also super important. Entire home.
Weekdays. Biz travelers usually look at the city level at pseicific address and you’re not starting big and zooming in. Search attributes.
Price. Wifi. Then you can build a model and deploy. Show the right business travel promotion. A banner on the booking page to sign up
for business travel for the people who are likely for it. A promotion of 100% would cannibalize the promotion space. P5 banners. That
gives virality effect where they can sign up> Yahoo is sign up company. Google is its a long tail of small business similar with facebook.
Airbnb core product is better for small medium businesses. Next time someone else signs up with the same company its legit and has
more than one person. Then we can send an email to those people to ask your travel managers - directly billed to company, find the right
listing. Data science is being used to find the long tail that we wouldn’t have found direct sales. Shared itinerary with other people - growth
experiment so other people sign up.

43. Machine learning infrastructure
Creating generalized infrastructure so we can do it all
● Making the case for machine learning infrastructure. Machine learning infrastructure. Holistic
representation of a listing. Where we are going in the future.

44. Case studies
● Early motivation is looking at our main metric. Search was a very hand tuned in the past. Pricing. Its not
easy! Accuracy is what I can improve but that metric moving is harder. You can improve the
performance intrinsically but then you deploy and it looks like the improvement doesn’t lead to the
improvement you think of. For example smart pricing you don’t like the suggestion. You’re lowballing.
Take into account people’s behavior and how users respond to an improvement.
● The simpler model is often a lot more effective. Better to build something quickly see how it performs
and then see if it can be revisit. Can reference the post on coming from academia.
● Ticket routing and user issues - had hard set rules that were very rigid - is you are in this bucket we
implemented a probabilistic model that figures automatically what we can do. Go from manual rules to
a learned model. Rules failing and then moving to ‘softer’ approaches that are probabilistic. One
pattern. We look at signals when the user comes in - surface these links vs those links. Like biz travel.
We were ignoring a strong signal that was the text of a ticket. Improve accuracy and also increase
volume and optimize precision and recall. Could address CX staffing accordingly. Route more to directly
and its ok if they can’t solve it and it takes time to send it back to Airbnb. Impossible to do in previous
world. High level talking point - these models give us more flexibility to adapt to the changing dymanics
of our business. Set of rules are much harder to tweak. Models give a lot more flexibility.
● Using machine learning to not just build model for predictive performance but to inform analysis. Chao
yang on host quality. 30% are worse. Build model on 70%. Learn a model to predict ratings in other
bucket. Lead model. PX model. Customizing how users interact with our website using signals available.
● Making the case for machine learning infrastructure. Machine learning infrastructure. Holistic
representation of a listing. Where we are going in the future.

45. Slide Title Here
Optional subtitle goes here
● Cereal Entrepreneur: Creative. Embraces
constraints. Solution-oriented. Tenacious.
● Be a Host: Collaborative. Anticipates the needs of
others. Prepared. Authentic. Listens.
● Embrace the Adventure: Flexible. Risk tolerant.
Always learning. Curious. Open-minded.
● Simplify: Distills a problem to its essence. Makes
and communicates clear decisions.
● Champion the Mission: Passionate. Committed.
Optimistic.
● Every Frame Matters: Thinks holistically. Rigorous
about quality. Appreciates the details and
prioritizes the right ones.

46. Slide Title Here
Optional subtitle goes here

47. Slide Title Here
Optional subtitle goes here
Lorem ipsum dolor sit amet,
consectetur adipiscing elit. Sed
risus arcu, lacinia a aliquet in,
vulputate ac turpis.
Donec elit elit, consectetur at
hendrerit a, porta ac elit.
Vivamus efficitur lacus nec ex
porttitor lacinia at et nulla.

49. Your text overlay goes here

51. Your text overlay
goes here
Your text overlay
goes here

53. Rausch Hackberry Kazan Babu Lima
Beach Ebisu Tirol Foggy Hoff
Brand Colors

59. Product Icons

61. Iconic Lists

62. Four Items
Iconic List
List Item 1 List Item 2 List Item 3
Lorem ipsum dolor
sit amet,
consectetur
adipiscing elit
Sed risus arcu,
lacinia a aliquet in,
vulputate turpis
Donec elit elit,
consectetur at
hendrerit a,
porta ac elit
Vivamus efficitur
lacus nec ex
porttitor lacinia at et
nulla
List Item 4

63. Five Items
Iconic List
List Item 1 List Item 2 List Item 3
Lorem ipsum
dolor sit amet,
consectetur
adipiscing elit
Sed risus arcu,
lacinia a aliquet
in, vulputate
turpis
Donec elit elit,
consectetur at
hendrerit a,
porta ac elit
Vivamus
efficitur
lacus nec ex
porttitor lacinia
at et nulla
List Item 4 List Item 5
Lorem ipsum
dolor sit amet,
consectetur
adipiscing elit

64. Timelines

65. Three Items
Timeline
Time 1 Time 2 Time 3
Lorem ipsum dolor sit amet,
consectetur adipiscing elit
Sed risus arcu, lacinia a
aliquet in, vulputate turpis
Donec elit elit, consectetur
at hendrerit a, porta ac elit

66. Four Items
Timeline
Time 1 Time 2 Time 3
Lorem ipsum dolor
sit amet, consectetur
adipiscing elit
Sed risus arcu, lacinia
a aliquet in,
vulputate turpis
Donec elit elit,
consectetur at
hendrerit a, porta ac
elit
Vivamus efficitur
lacus nec ex porttitor
lacinia at et nulla
Time 4

67. Five Items
Timeline
Time 1 Time 2 Time 3
Lorem ipsum
dolor sit amet,
consectetur
adipiscing elit
Sed risus arcu,
lacinia a aliquet
in, vulputate
turpis
Donec elit elit,
consectetur at
hendrerit a,
porta ac elit
Vivamus
efficitur
lacus nec ex
porttitor lacinia
at et nulla
Time 4 Time 5
Lorem ipsum
dolor sit amet,
consectetur
adipiscing elit

68. Four Items with Box Callout
Timeline
This is a box
callout. Text is
fully editable
and you can
move it around
to different
dots.
Time 1 Time 2 Time 3 Time 4

69. Six Items with Box Callout
Timeline
Time 1 Time 2 Time 3 Time 4 Time 5
This is a box
callout. Text is
fully editable
and you can
move it around
to different
dots.
Time 6

70. Map of Airbnb Offices

71. Portland
San Francisco
Los Angeles
Toronto
New York
Miami
Sao Paulo
Dubli
n
London
Paris
Barcelona
Berlin
Milan
Copenhagen
New Delhi
Seoul
Beijing
Tokyo
Sydney
Singapore
Washington, DC
Map of Airbnb Offices
2016

72. Charts

73. Column Chart
Charts
Jan Feb Mar Apr
30
May
10
20
30
40
0
20
25
10
40

74. Column Chart with Highlight
Charts
Jan Feb Mar Apr
30
May
10
20
30
40
0
20
25
10
40

75. Column Chart - Multicolor
Charts
Jan Feb Mar Apr
30
May
10
20
30
40
0
20
25
10
40

76. Bar Chart
Charts
10 20 30 40
Apr
Mar
Feb
Jan
May
0
30
20
25
10
40

77. Bar Chart with Highlight
Charts
10 20 30 40
Apr
Mar
Feb
Jan
May
0
30
20
25
10
40

78. Bar Chart - Multicolor
Charts
10 20 30 40
Apr
Mar
Feb
Jan
May
0
30
20
25
10
40

79. Line Chart
Charts
Jan Feb Mar Apr May
10
20
30
40
0
Item 1
Item 2
Item 3

80. Line Chart with Data Points
Charts
Jan Feb Mar Apr May
10
20
30
40
0
Item 1
Item 2
Item 3