Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Northeastern University, 4 April 2018
Jen Wang
Wayfair Data Science Team, Projects, and Case Study --
Uplift Modeling for ...
2
Wayfair: e-Commerce Tech Company
Our typical customer:
35 to 65 year old
woman with annual
household income of
$50k to $...
3
A Clear Online Leader in Home Goods
OtherDirect Retail
4
Main goals of seminar
1. Data Science Team in Wayfair – How Data Science work is
organized at Wayfair, and the different...
5
Data Science Team in Wayfair
6
Venn Diagram for Wayfair Data Science
Commonalities
across
companies
Trade-offs
Research
Application Engineering
β€’ Devel...
7
Data Science Groups at Wayfair
DS Infrastructure
DS Operation
Catalog
Optimization
NLP / CNN
Competitive
Intelligence
NL...
8
Marketing Data Science: Business Problems
9
The key objective of Marketing Data Science is aligned to maximize return on marketing
investment by optimizing budget a...
10
(1/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Reta...
11
(2/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Reta...
12
(3/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Reta...
13
(4/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Reta...
14
(5/5) Example of Marketing Data Science Project
MKT Channel A
E.g. TV
MKT Channel B
E.g. Search
MKT Channel X
E.g. Reta...
15
Case Study:
Uplift Modeling to Drive Incremental Revenue in Display Remarketing
16
Customer Scoring
Uplift modeling for prediction of incremental revenue as base bid for each customer
π’š=
Ad-inventory Sc...
17
Modeling and Evaluation
β€’ Random Targeting to Collect Data
β€’ Uplift Modeling
β€’ Score (Predicted Uplift) = P(buy | Ad) -...
18
Jen Wang’s Journey to Data Science
Ph.D. in
(Biophysical)
Chemistry
Postdoc in
Drug Design
Health Data
Science
Fellow
M...
19
Wayfair: We Are Hiring!
β€’ Wayfair DS Career: https://www.wayfaircareers.com/
β€’ Wayfair DS Blog: http://tech.wayfair.com...
Upcoming SlideShare
Loading in …5
×

Wayfair's Data Science Team and Case Study: Uplift Modeling

Jen Wang, Wayfair Data Science Team, Projects, and Case Study -- Uplift Modeling for Driving Incremental Revenue in Display Remarketing

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Wayfair's Data Science Team and Case Study: Uplift Modeling

  1. 1. Northeastern University, 4 April 2018 Jen Wang Wayfair Data Science Team, Projects, and Case Study -- Uplift Modeling for Driving Incremental Revenue in Display Remarketing
  2. 2. 2 Wayfair: e-Commerce Tech Company Our typical customer: 35 to 65 year old woman with annual household income of $50k to $250k; comScore median household income of $82k
  3. 3. 3 A Clear Online Leader in Home Goods OtherDirect Retail
  4. 4. 4 Main goals of seminar 1. Data Science Team in Wayfair – How Data Science work is organized at Wayfair, and the different types of projects we work on 2. Marketing Data Science – How Data Science projects are aligned against different points of the marketing funnel 3. Case Studies in MKT DS – Uplift Modeling for Driving Incremental Revenue in Display Remarketing
  5. 5. 5 Data Science Team in Wayfair
  6. 6. 6 Venn Diagram for Wayfair Data Science Commonalities across companies Trade-offs Research Application Engineering β€’ Develop & apply machine learning algorithms to find answers to business problems β€’ Great range of algorithm complexity (from linear / logistic regression to deep learning), but always need sufficient, β€œbig” data to get good results β€’ Standard set of technical tools, from R / Python for scripting to Spark for big data processing β€’ Innovate by creating new algorithms / approaches to solve problems β€’ Typically β€œbet big”, but doesn’t always pay off β€’ Innovate by efficiently adapting existing approaches to solve problems β€’ Can sometimes lead to more incremental progress, tricky to build for long-term β€’ Typically use business rules & simpler algorithms β€’ Focus on robustness & scalability first, then modeling Business Problem Solving Engineering Research / Modeling / ML Warning: No Unicorns!!! Modeling β€’ Build β€œright” model first, then whittle away to get in form ready for production β€’ Need to be mindful of 80/20
  7. 7. 7 Data Science Groups at Wayfair DS Infrastructure DS Operation Catalog Optimization NLP / CNN Competitive Intelligence NLP DS Marketing Customer Scoring & Bidding B2B Uplift Model Text Mining E.g. E.g. Business Problem Solving Engineering Research / Modeling / ML Business Problem Solving Engineering Research / Modeling / ML DS Product Recommenda- -tion System Visual Search Reinforcement learning / CF E.g. CNN Business Problem Solving Engineering Research / Modeling / ML
  8. 8. 8 Marketing Data Science: Business Problems
  9. 9. 9 The key objective of Marketing Data Science is aligned to maximize return on marketing investment by optimizing budget allocation, channel strategy and customer journey touchpoint MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget Maximize MROI = 𝑅𝑒𝑣𝑒𝑛𝑒𝑒 𝐴𝑑 πΆπ‘œπ‘ π‘‘ By… 1. Guide the right budget allocation across channels in our marketing portfolio 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel 3. Lineup the marketing treatments in the right sequence and right cadence along the customer life journey Customer Life Time Value $ T1 T2 TN Systematic View of Marketing Marketing DS Objective
  10. 10. 10 (1/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 1. Guide the right budget allocation across channels in our marketing portfolio Q for DS: How would you measure the revenue contributed by different channels? Customer Life Time Value $ T1 T2 TN Order Attribution base on Incremental Value
  11. 11. 11 (2/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel Q for DS: β€’ How to decide which TV channel should be invested with more ads? Customer Life Time Value $ T1 TV Targeting T2 TN
  12. 12. 12 (3/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel Q for DS: β€’ How much we should bid on each google keyword? Customer Life Time Value $ T1 Keyword Bidding T2 TN
  13. 13. 13 (4/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel Q for DS: β€’ How much we should bid the ads on each customer? Customer Life Time Value $ T1 Display Ads T2 TN
  14. 14. 14 (5/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 3. Schedule the marketing treatments in the right sequence and right cadence along the customer life journey Q for DS: β€’ How often we should send the marketing emails to customers? Customer Life Time Value $ T1 T2 TN
  15. 15. 15 Case Study: Uplift Modeling to Drive Incremental Revenue in Display Remarketing
  16. 16. 16 Customer Scoring Uplift modeling for prediction of incremental revenue as base bid for each customer π’š= Ad-inventory Scoring Click-through rate prediction as bid modifier across Internet Data Science Solutions 𝑦1 = P(buy | Ad) - P(buy | no Ad) y2 = $Rev(buy | Ad) Expected incr. Rev Uplift Base Bid Case Study – Uplift Modeling in Display Remarketing Display Remarketing Why is it challenging? β€’ Billions of bidding opportunities across Internet per day β€’ Real-time bidding β€’ Customer-level prediction β€’ Causal effects of Ad targeting? *numbers are Illustrative only Final Bid! X 𝐢𝑇𝑅 = πΆπ‘–π‘π‘˜π‘  πΌπ‘šπ‘π‘Ÿπ‘’π‘ π‘ π‘–π‘œπ‘›π‘  Bid Modifier
  17. 17. 17 Modeling and Evaluation β€’ Random Targeting to Collect Data β€’ Uplift Modeling β€’ Score (Predicted Uplift) = P(buy | Ad) - P(buy | no Ad) β€’ Uplift = Test CVR - Control CVR Control Group: PSA Test Group: Ads Seen Case Study – Uplift Modeling in Display Remarketing Background β€’ A method for modeling and predicting causal effects β€’ Target most incremental (or persuadable) customers β€’ Obama Camp persuaded millions of voters with Uplift Modeling in 2012 Persuadables Sure Things Lost Causes Sleeping Dogs Will Convert if Not Treated WillConvertifTreated No Yes NoYes NumberofIncrementalCustomers 10 20 30 40 50 60 70 80 90 100 20 30 1040 01020 Number of Customers Targeted & Random Targeting Perfect Uplift Model Good Uplift Model Perfect Conversion Model Good Conversion Model
  18. 18. 18 Jen Wang’s Journey to Data Science Ph.D. in (Biophysical) Chemistry Postdoc in Drug Design Health Data Science Fellow Marketing Data Scientist
  19. 19. 19 Wayfair: We Are Hiring! β€’ Wayfair DS Career: https://www.wayfaircareers.com/ β€’ Wayfair DS Blog: http://tech.wayfair.com/category/data-science/ If you are interested in Wayfair data science... Happy to answer any question you have… β€’ LinkedIn: https://www.linkedin.com/in/jenzhenwang/

Γ—