Guest lecture given in introductory data science course in Aalto University on 2017-11-30. About how data science is used in online marketing by Facebook and Smartly.io

1. Data Science in
Online Marketing
Markus Ojala
PhD, Chief Data Scientist @ Smartly.io
Aalto University - 2017-11-30

2. Data Scientist (n.):
Person who is better at statistics than
any software engineer and better at
software engineering than any
statistician.
@josh_wills

3. My
story
2004-2008: Aalto, tech. physics
2008-2011: PhD in inf. & comp. science
2011-2015: Xtract & Comptel
2015- : Smartly.io
Randomization Algorithms for
Assessing the Significance of
Data Mining Results

4. 1 billion spend
Managed yearly
Smartly.io
Facebook &
Instagram Partner

6. 1. Online Marketing
2. Tracking & Attribution
3. Testing
4. Optimizing
5. Optimizing more
6. Summary

7. 1. Online Marketing

8. Dimensions in online marketing
Campaign objective
(sales, branding)
Target audience,
budget, bid
Ad content
Targeting
Ad type
and
content
Bidding
Budget

9. Example: e-commerce with product feed
- XML, CSV, TSV, JSON
- No limits on the number of products
- Refresh up to once an hour
!

10. Retarget audience with dynamic creative

11. Look-a-like

12. Bid value for the
desired outcome
Estimated
action rate
Relevance &
quality
X +Bid =
Maximizes advertiser value Optimizes consumer
experience
Conversion
Optimized
Bidding
It’s simple.
You can bid directly
for your end goal.

13. Bid value for the
desired outcome
Estimated
action rate
Relevance &
quality
X +Bid =
Maximizes advertiser value Optimizes consumer
experience
Conversion
Optimized
Bidding
It’s simple.
You can bid directly
for your end goal.
$50 0.5%$0.25
Predicting estimated action rate per user:
● Target: 1 user converted, 0 user did not convert
● Predictors: age, gender, location, interests,
behaviour…
● Simple model: e.g., logistic regression
● Use historical data to train the model

14. Budget pacing
maximizes
the profit
Facebook finds the
cheapest conversions
throughout the day
if budget limits
https://www.smartly.io/blog/bid-optimization-pacing-on-facebook-explained

15. 2. Tracking & Attribution

16. Customer’s journey to the final conversion is usually not that straightforward

17. It’s a combination of multiple touch points in many channels

18. Facebook attributes conversions to the last ad user clicked or saw in their advertising
ecosystem

19. While Google Analytics does the same within their ecosystem, using last click attribution
as default method

20. Attribution modelling
1) Basic attribution
• First click
• Last click
2) Advanced attribution
• U-model
• Linear
• Time decay
2) Statistical attribution
• Survival model
• Marketing mix model

21. 3. Testing

22. Split audience
into two groups
Show each group
a different campaign
Measure performance
in each
138
conversions
172
conversions
Split test

23. $100 / day$100 / day

24. $100 / day$100 / day
7 2

25. $100 / day$100 / day
7
14
20
31
40
53
63
71
75
80
2
15
24
30
43
56
67
77
86
97

26. 6 108 12 14 16 18 20
B
A
$12.50 ≈ $1000 / 80$1000 / 97 ≈ $10.31
Cost per conversion

27. Statistical significance
Why you should care:
1. There is always random noise.
2. You do not want to make decisions
based on random noise.

28. Lift study
670 conversions 475 conversions
Split audience
in two
Show ads only
to one group
Measure performance
in each
Lift study

29. 4. Optimizing

30. Allocating Budget

31. Modeling results per spend

32. Multi-Armed
Bandit

33. Bayesian Bandits / Thompson sampling
The number of pulls for a given lever should match
its actual probability of being the optimal lever
Sample from the posterior for the mean of each lever

34. Adset Conversions Impressions Conversion rate Budget proportion
1 848 1 417 764 0.060% 92.7%
2 723 1 349 349 0.054% 1.4%
3 60 121 784 0.049% 5.7%
4 107 270 757 0.040% 0.0%
5 22 65 200 0.034% 0.2%
6 45 154 330 0.029% 0.0%

35. 5. Optimizing more

36. Moving from conversions to revenue
ROAS = revenue / cost
= revenue / conversions * conversions / cost
= revenue / conversions * 1 / CPA
ROAS = return on ad spend
CPA = cost per action
Existing model
● Lot of data
● Varies fast
● Big differences
New model
● Little data
● Varies slowly
● Small real differences
● Lot of random variation

37. Modeling the revenue per purchase
Revenue follows usually a long-tailed distribution, use log-normal

38. Multilevel
Model
Account
Campaign
Ad set

39. Multilevel model in Stan

40. Posterior predictive checks

41. Results 1
Common case:
ad sets don’t
really differ

42. Results 2
Campaigns
differ, ad sets
not

43. Results 3
Lot of data
Ad sets differ

44. 6. Summary

45. 1. Turn customer case into analytical problem
2. Data processing
3. Modelling
Data science in practice

46. All models are wrong,
but some are useful.
George Box

47. You do not really understand
something unless you can explain
it to your grandmother.
Einstein / Feynman / ...

48. https://www.smartly.io/blog/
● Experiences in Using R and Python in Production
● Insights to Predictive Budget Allocation
● Automated Statistical Significance Calculator for A/B Testing
● Tutorial: How We Productized Bayesian Revenue Estimation with Stan
Read more

49. WORK SMARTLY
We’re looking for new Humble Hungry Hunters
to build a world-class product company.
https://www.smartly.io/careers
Markus.Ojala@smartly.io