Sri Sri: "When I worked with Skyscanner, before Spotify, I worked on a popular marketing problem called Multi-touch Attribution". In this presentation, she’ll explain Multi-touch attribution problem space and how a Data Science solution was designed for this problem.I focus on a key challenge of designing a Multi-touch attribution solution – Identifying which methodology is best for this problem. This is a thorny issue as the variable we are trying to study isn’t directly observable. She’ll discuss a novel approach to tackle this problem through simulated marketing environment.

1. Multi-touch Attribution
Sri Sri Perangur
Key challenge around designing a solution

2. ABOUT SRI²
Sr. Data Scientist at Spotify
Sr. Data Scientist at Skyscanner
Data Scientist at State
Developer/Tech analyst at RBC Capital Markets
Research Associate at BAE Systems Applied Intelligence Unit

3. SKYSCANNER

4. Multi-touch
Attribution
The challenge is to identify the
distribution of credit amongst
multiple players.
The problem is seen in marketing
and product space amongst
others.
Training Data
The players’ credit is not easy to
observe directly.
Makes it challenging to define
the label/training data.
Need labeled data
to enable comparison of the
many solution options available
these days.
Simulated
Training data
The expert knowledge &
known data trails are captured to
define the Simulated Marketing
Environment .
This enables scoring and stress
testing with a range of training
data sets.
SUMMARY

5. Multi-Touch Attribution
Problem space
1.

7. ATTRIBUTION EXPLAINED
Thanks to SegmentIO

8. WHERE DO WE SEE THIS OCCUR IN PRODUCTS ?

9. Social Networks
Search Engine
Marketing
Google Display
Network
Organic
Direct
Book
through
Time
£
£
WHERE DO WE SEE THIS OCCUR IN MARKETING ?
Day 0 Day 1 Day 7 Day 15 Day 16
www.skyscanner.net

10. Social Networks
Search Engine
Marketing
Google Display
Network
Organic
Direct
Book
through
Time
£
£
WHERE DO WE SEE THIS OCCUR IN MARKETING ?
Day 0 Day 1 Day 7 Day 15 Day 16
www.skyscanner.net
30%
20%
10%
45%
0%
Hypothetically

11. Social Networks
Search Engine
Marketing
Google Display
Network
Organic
Direct
Book
through
Time
WHERE DO WE SEE THIS OCCUR IN MARKETING ?
Day 0 Day 1 Day 7 Day 15 Day 16
30%
20%
10%
45%
0%
Hypothetically

12. Designing a solution
...a Data Driven solution
2.

13. MOVING AWAY FROM THE PARTIAL VIEW
Even Time decay Custom rules

14. 70,000,000 users
and counting

15. Data Driven Attribution

16. DATA DRIVEN SOLUTION

17. WHICH IS THE BEST METHOD ?

18. Methods are ready to train ...Where is the training data ?
Training data3.
1010101010100101010101
0101001010100101010101
0101010100101010100101
1010101010100101010101
0101001010100101010101
0101010100101010100101
1010101011010101010100
1010101010101001010100
1010101010101010100101
0101001011010101010100
1010101010101001010100
1010101010101010100101
0101001011010101010100
1010101010101001010100
1010101010101010100101
0101001010100101010101
0101001010100101010101
0101010100101010100101

19. Which is the best method for MTA
Identifying the best3.

20. TYPICAL MACHINE LEARNING PROBLEM
● Elevation
● Price per
sqft
● Price
● Square
feet
● Year built
● Bathrooms

21. MULTI-TOUCH ATTRIBUTION PROBLEM
● No of
touchpoints
user sees
● Marketing channel
distribution traits
● Sequence of
occurrence
● And so on..
Channel Value
Social Network ?
Search Engine Marketing ?
Google Display Network ?
Organic ?
Direct ?
X

22. aka Latent variables
Hidden in the real world

23. Concepts we can’t
observe directly …
LATENT VARIABLES
but we can infer
with byproducts...

24. Social Networks
Search Engine
Marketing
Google Display
Network
Organic
Direct
Book
through
Time
EXPERT OPINION MATTERS
Day 0 Day 1 Day 7 Day 15 Day 16
30%
20%
10%
45%
0%
Hypothetically

25. MULTI-TOUCH ATTRIBUTION PROBLEM
● No of touchpoints
user sees
● Marketing channel
distribution traits
● Sequence of
occurrence
● And so on..
?X

26. MULTI-TOUCH ATTRIBUTION PROBLEM
X
● No of
touchpoints
user sees
● Marketing channel
distribution traits
● Sequence of
occurrence
● And so on..

27. BUT CAN AN EXPERT GAUGE MARGINS EASILY ?

28. Simulated
training data
to enable classifier training

29. SIMULATED MARKETING ENVIRONMENT

30. CAPTURING EXPERT OPINION
Aim is to capture the educated expectations of experts
in the field, regarding the key perspectives .
To name a few:
● Marketing channel value
● Marketing channel after effect
In a way that we can fluctuate the key variables

31. CAPTURING EXPERT OPINION
Aim is to capture the educated expectations of experts
in the field, regarding the key perspectives .
To name a few:
● Marketing channel value
● Marketing channel after effect
In a way that we can fluctuate the key variables
Channel Value
(hypo
thetic
ally)
Social Network 30%
Search Engine Marketing 20%
Google Display Network 10%
Organic 45%
Direct 0%

32. CAPTURING EXPERT OPINION
Aim is to capture the educated expectations of experts
in the field, regarding the key perspectives .
To name a few:
● Marketing channel value
● Marketing channel after effect
In a way that we can fluctuate the key variables
Impact over weeks for
channel X

33. CAPTURING KNOWN DATA TRAITS
Capturing known traits of channel behaviour seen in company data
● Distribution of users over marketing channels
● Distribution of various marketing touch points over all touchpoint
● Distribution of the marketing touch points over our users’ journey
● Distribution of sessions/conversions/conversion rate per channel

34. CAPTURING KNOWN DATA TRAITS
Capturing known traits of channel behaviour seen in company data
● Distribution of users over marketing channels
● Distribution of various marketing touch points over all touchpoint
● Distribution of the marketing touch points over our users’ journey
● Distribution of sessions/conversions/conversion rate per channel

35. CAPTURING KNOWN DATA TRAITS
Capturing known traits of channel behaviour seen in company data
● Distribution of users over marketing channels
● Distribution of various marketing touch points over all touchpoint
● Distribution of the marketing touch points over our users’ journey
● Distribution of sessions/conversions/conversion rate per channel

36. CAPTURING KNOWN DATA TRAITS
Capturing known traits of channel behaviour seen in company data
● Distribution of users over marketing channels
● Distribution of various marketing touch points over all touchpoint
● Distribution of the marketing touch points over our users’ journey
● Distribution of sessions/conversions/conversion rate per channel
Distribution of seeing channel X
our user touchpoint position

37. CAPTURING KNOWN DATA TRAITS
Capturing known traits of channel behaviour seen in company data
● Distribution of users over marketing channels
● Distribution of various marketing touch points over all touchpoint
● Distribution of the marketing touch points over our users’ journey
● Distribution of sessions/conversions/conversion rate per channel

38. SIMULATED MARKETING ENVIRONMENT

39. Results
Best method for Multi-touch attribution
For Skyscanner

40. MULTI-TOUCH ATTRIBUTION PROBLEM
X
90%

41. SCORING AGAINST SIMULATED GOAL
2.69%
Current
Industries
popular
method
Baseline

42. Stress testing
models
How durable are these solution in other
scenarios ?

43. WHAT IF...
The Channel estimates are wrong ?
Estimated channel value as £ X but should be …
• £ X*100
• £ X * 0.01
• ….

44. STRESS TESTING
1. Fluctuate probability of channel exposure
2. Fluctuate probability of purchase
3. Fluctuate relative touchpoint weights ( ex:
marketing channel after effect)
90%

45. STRESS TESTING RESULTS
The Method Z (i.e. the 90% accurate
one) is the one in Green.
Method Z is the best even at
adapting to change when tested in
20 scenarios.
Underestimate Overestimate

46. APPLYING IN PRODUCTION

47. SIMULATION STACK
Channel Value
(hypotheticall
y)
Social Network 30%
Search Engine Marketing 20%
Google Display Network 10%
Organic 45%
Direct 0%

48. PRODUCTION APPLICATION STACK
Channel Value
Social Network a%
Search Engine Marketing b%
Google Display Network c%
Organic d%
Direct e%

49. CONCLUSION
● Understand accuracy of a approach
● Comparison of different approaches
● Tuning of different solutions
● Stress testing solutions

50. Thank you
Cheers to you all
& my team and community that have made this possible
Segment IO
Growth images
Content: Pretty images:
MC Attribution Blog
Shapley value Attribution blog
Photos from Unsplash
ML Gif
Pretty Icons from Flaticons
Food for thought -
Prof. Tamara Broderick
from MIT has been working
o
n enabling estimates
of latent variables through
Bayesian sampling
methods.
Skyscanner MTA
research poster
Research icons
Movie image
Oscar pic

51. Thank you
Cheers to you all
& my team and community that have made this possible

52. Any questions?
You can find me at:
» @perangur
» www.perangur.com
» Slides: http://bit.ly/MTASummerConf