Jen Wang, Wayfair Data Science Team, Projects, and Case Study -- Uplift Modeling for Driving Incremental Revenue in Display Remarketing

1. Northeastern University, 4 April 2018
Jen Wang
Wayfair Data Science Team, Projects, and Case Study --
Uplift Modeling for Driving Incremental Revenue in Display Remarketing

2. 2
Wayfair: e-Commerce Tech Company
Our typical customer:
35 to 65 year old
woman with annual
household income of
$50k to $250k;
comScore median
household income of
$82k

3. 3
A Clear Online Leader in Home Goods
OtherDirect Retail

4. 4
Main goals of seminar
1. Data Science Team in Wayfair – How Data Science work is
organized at Wayfair, and the different types of projects we work on
2. Marketing Data Science – How Data Science projects are aligned
against different points of the marketing funnel
3. Case Studies in MKT DS – Uplift Modeling for Driving Incremental
Revenue in Display Remarketing

5. 5
Data Science Team in Wayfair

6. 6
Venn Diagram for Wayfair Data Science
Commonalities
across
companies
Trade-offs
Research
Application Engineering
• Develop & apply machine learning algorithms to find
answers to business problems
• Great range of algorithm complexity (from linear / logistic
regression to deep learning), but always need sufficient,
“big” data to get good results
• Standard set of technical tools, from R / Python for
scripting to Spark for big data processing
• Innovate by creating new algorithms /
approaches to solve problems
• Typically “bet big”, but doesn’t always pay
off
• Innovate by efficiently adapting existing
approaches to solve problems
• Can sometimes lead to more incremental
progress, tricky to build for long-term
• Typically use business rules & simpler
algorithms
• Focus on robustness & scalability first, then
modeling
Business
Problem
Solving
Engineering
Research /
Modeling /
ML
Warning:
No Unicorns!!!
Modeling
• Build “right” model first, then whittle away to get
in form ready for production
• Need to be mindful of 80/20

7. 7
Data Science Groups at Wayfair
DS Infrastructure
DS Operation
Catalog
Optimization
NLP / CNN
Competitive
Intelligence
NLP
DS Marketing
Customer
Scoring &
Bidding
B2B
Uplift Model Text Mining
E.g. E.g.
Business
Problem
Solving
Engineering
Research /
Modeling /
ML
Business
Problem
Solving
Engineering
Research /
Modeling /
ML
DS Product
Recommenda-
-tion System
Visual Search
Reinforcement
learning / CF
E.g.
CNN
Business
Problem
Solving
Engineering
Research /
Modeling /
ML

8. 8
Marketing Data Science: Business Problems

9. 9
The key objective of Marketing Data Science is aligned to maximize return on marketing
investment by optimizing budget allocation, channel strategy and customer journey touchpoint
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Retargeting
MKT
Budget Maximize MROI =
𝑅𝑒𝑣𝑒𝑛𝑢𝑒
𝐴𝑑 𝐶𝑜𝑠𝑡
By…
1. Guide the right budget allocation across
channels in our marketing portfolio
2. Provide channel level tactical guidance
towards delivering the right message to the
right customer through the right channel
3. Lineup the marketing treatments in the right
sequence and right cadence along the
customer life journey
Customer Life
Time Value
$
T1 T2 TN
Systematic View of Marketing Marketing DS Objective

10. 10
(1/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Retargeting
MKT
Budget
1. Guide the right budget allocation across
channels in our marketing portfolio
Q for DS:
How would you measure the revenue
contributed by different channels?
Customer Life
Time Value
$
T1 T2 TN
Order Attribution base on
Incremental Value

11. 11
(2/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Retargeting
MKT
Budget
2. Provide channel level tactical guidance
towards delivering the right message to the
right customer through the right channel
Q for DS:
• How to decide which TV channel
should be invested with more ads?
Customer Life
Time Value
$
T1
TV Targeting
T2 TN

12. 12
(3/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Retargeting
MKT
Budget
2. Provide channel level tactical guidance
towards delivering the right message to the
right customer through the right channel
Q for DS:
• How much we should bid on each
google keyword?
Customer Life
Time Value
$
T1
Keyword Bidding
T2 TN

13. 13
(4/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Retargeting
MKT
Budget
2. Provide channel level tactical guidance
towards delivering the right message to the
right customer through the right channel
Q for DS:
• How much we should bid the ads on
each customer?
Customer Life
Time Value
$
T1
Display Ads
T2 TN

14. 14
(5/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Retargeting
MKT
Budget
3. Schedule the marketing treatments in the
right sequence and right cadence along the
customer life journey
Q for DS:
• How often we should send the
marketing emails to customers?
Customer Life
Time Value
$
T1 T2 TN

15. 15
Case Study:
Uplift Modeling to Drive Incremental Revenue in Display Remarketing

16. 16
Customer Scoring
Uplift modeling for prediction of incremental revenue as base bid for each customer
𝒚=
Ad-inventory Scoring
Click-through rate prediction as bid modifier across Internet
Data Science Solutions
𝑦1 = P(buy | Ad) - P(buy | no Ad)
y2 = $Rev(buy | Ad)
Expected incr. Rev
Uplift
Base Bid
Case Study – Uplift Modeling in Display Remarketing
Display Remarketing
Why is it challenging?
• Billions of bidding opportunities across
Internet per day
• Real-time bidding
• Customer-level prediction
• Causal effects of Ad targeting?
*numbers are Illustrative only
Final Bid!
X
𝐶𝑇𝑅 =
𝐶𝑖𝑐𝑘𝑠
𝐼𝑚𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛𝑠
Bid Modifier

17. 17
Modeling and Evaluation
• Random Targeting to Collect Data
• Uplift Modeling
• Score (Predicted Uplift) = P(buy | Ad) - P(buy | no Ad)
• Uplift = Test CVR - Control CVR
Control Group:
PSA
Test Group:
Ads Seen
Case Study – Uplift Modeling in Display Remarketing
Background
• A method for modeling and predicting
causal effects
• Target most incremental (or persuadable)
customers
• Obama Camp persuaded millions of
voters with Uplift Modeling in 2012
Persuadables Sure Things
Lost Causes Sleeping Dogs
Will Convert if Not Treated
WillConvertifTreated
No Yes
NoYes
NumberofIncrementalCustomers
10 20 30 40 50 60 70 80 90 100
20 30
1040
01020
Number of Customers Targeted
&
Random Targeting
Perfect Uplift Model
Good Uplift Model
Perfect Conversion Model
Good Conversion Model

18. 18
Jen Wang’s Journey to Data Science
Ph.D. in
(Biophysical)
Chemistry
Postdoc in
Drug Design
Health Data
Science
Fellow
Marketing
Data
Scientist

19. 19
Wayfair: We Are Hiring!
• Wayfair DS Career: https://www.wayfaircareers.com/
• Wayfair DS Blog: http://tech.wayfair.com/category/data-science/
If you are interested in Wayfair data science...
Happy to answer any question you have…
• LinkedIn: https://www.linkedin.com/in/jenzhenwang/