A real-time bidding platform responds to incoming ad-opportunities (“bid requests”) by deciding whether or not to submit a bid and how much to bid. If the submitted bid wins, the user is shown an ad. Advertisers hope that ad-exposure leads to an increased likelihood of a desired action, such as a click or conversion (purchase, etc). So an important quantity that advertisers want to measure is the causal effect of advertising, namely, what is the response probability of an exposed user, compared with the counterfactual (un-observable) response-rate of the user if they were not exposed to the ad. In an ideal randomized test, the user is randomly assigned to test or control AFTER the submitted bid is won, and test users are served the ad in the normal way, while control users are not. While this is ideal from a statistical perspective, in practice this approach has the drawback that money spent by advertisers is wasted when a user is assigned to control. At MediaMath we have developed a methodology for causal effect measurement where users are assigned to test or control BEFORE bid submission. One challenge here is that not all test-group users are exposed to an ad; only a winning bid results in ad exposure, and the winning population can have a significant bias. This talk will describe our approach to handle this and other challenges to ad impact measurement in this setting, and how we use MCMC Gibbs sampling to arrive at confidence intervals for ad-impact.

1. Estimating Causal Effect of Ads in a
Real-Time-Bidding Platform
Prasad Chalasani (SVP Data Science, MediaMath)
Sep 24, 2016

2. Project Placebo
Or,
How to Measure Causal Effect of Ads in an RTB Platform
Placebo Team (alphabetical):
Ari Buchalter (President, Technology; co-founder)
Prasad Chalasani
Himanish Kushary
Jason Lei
Jonathan Marshall
Michael Neiss
Tristan Piron
Sara Skrmetti
Jawad Stouli
Jaynth Thiagarajan
Ezra Winston

4. listen to ~ 100 Bln ad opportunities daily

5. listen to ~ 100 Bln ad opportunities daily
respond with optimal bids within milliseconds

6. listen to ~ 100 Bln ad opportunities daily
respond with optimal bids within milliseconds
petabytes of data (ad impressions, visits, clicks, conversions)

7. Key Conceptual Take-aways
Definition of causal effect

8. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning

9. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform

10. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful

11. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)

12. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)
MediaMath’s solution

13. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)
MediaMath’s solution
Bayesian Methods for Ad Lift Confidence Bounds

14. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)
MediaMath’s solution
Bayesian Methods for Ad Lift Confidence Bounds
Gibbs Sampling (MCMC – Markov Chain Monte Carlo)

15. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)
MediaMath’s solution
Bayesian Methods for Ad Lift Confidence Bounds
Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
Complications unique to our setting:

16. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)
MediaMath’s solution
Bayesian Methods for Ad Lift Confidence Bounds
Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
Complications unique to our setting:
Long-running experiments

17. Key Conceptual Take-aways
Definition of causal effect
Context: relationship to Machine Learning
Causal effect in a Real-Time Bidding Platform
Simplest approach is wasteful
Less wasteful approach: bias (non-compliance)
MediaMath’s solution
Bayesian Methods for Ad Lift Confidence Bounds
Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
Complications unique to our setting:
Long-running experiments
Multiple cookies per user

18. Ad impact measurement
Advertisers want to know the impact of showing ads to people.

19. Measuring Ad Impact: Two Approaches
Observational studies:

20. Measuring Ad Impact: Two Approaches
Observational studies:
Compare people who happen to be exposed vs not exposed

21. Measuring Ad Impact: Two Approaches
Observational studies:
Compare people who happen to be exposed vs not exposed
Bias a big issue

22. Measuring Ad Impact: Two Approaches
Observational studies:
Compare people who happen to be exposed vs not exposed
Bias a big issue
Randomized tests:

23. Measuring Ad Impact: Two Approaches
Observational studies:
Compare people who happen to be exposed vs not exposed
Bias a big issue
Randomized tests:
Randomly assign people to test (exposed), control (un-exposed)

24. Causal Effect: the questions to ask
When a set of people U is exposed to ads,
what is the avg response-rate R1 of the people in U?

25. Causal Effect: the questions to ask
When a set of people U is exposed to ads,
what is the avg response-rate R1 of the people in U?
what would have been the response rate R0 of U, if they
had not seen the ad?

26. Causal Effect: the questions to ask
When a set of people U is exposed to ads,
what is the avg response-rate R1 of the people in U?
what would have been the response rate R0 of U, if they
had not seen the ad?
relative causal effect, or causal lift = R1/R0 − 1

27. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )

28. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.

29. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:

30. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad

31. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad

32. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.

33. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.
Observed response: Y obs
i = Yi (Wi )

34. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.
Observed response: Y obs
i = Yi (Wi )
if Wi = 1, only Yi (1) is observed, Yi (0) is a counterfactual

35. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.
Observed response: Y obs
i = Yi (Wi )
if Wi = 1, only Yi (1) is observed, Yi (0) is a counterfactual
if Wi = 0, only Yi (0) is observed, Yi (1) is a counterfactual

36. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.
Observed response: Y obs
i = Yi (Wi )
if Wi = 1, only Yi (1) is observed, Yi (0) is a counterfactual
if Wi = 0, only Yi (0) is observed, Yi (1) is a counterfactual
Xi = k-dimensional vector of features

37. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.
Observed response: Y obs
i = Yi (Wi )
if Wi = 1, only Yi (1) is observed, Yi (0) is a counterfactual
if Wi = 0, only Yi (0) is observed, Yi (1) is a counterfactual
Xi = k-dimensional vector of features
e.g. (dayOfWeek, age, location, web-domain)

38. Causal Effect: Notation
“units” i = 1, 2, . . . , n (“users”, “user_context”, . . . )
Yi = 1 if unit i responds (buys, subscribes, . . . ), else 0.
Each unit i has 2 potential responses:
Yi (0) = response when not exposed to an ad
Yi (1) = response when exposed to an ad
Wi = 1 if unit i exposed to ad, else 0.
Observed response: Y obs
i = Yi (Wi )
if Wi = 1, only Yi (1) is observed, Yi (0) is a counterfactual
if Wi = 0, only Yi (0) is observed, Yi (1) is a counterfactual
Xi = k-dimensional vector of features
e.g. (dayOfWeek, age, location, web-domain)
Unit level causal effect is impossible to measure:
τi = Yi (1) − Yi (0)

39. Average Causal/Treatment Effects

40. Average Causal/Treatment Effects
Average Treatment Effect (ATE)
ATE = E[Yi (1) − Yi (0)]

41. Average Causal/Treatment Effects
Average Treatment Effect (ATE)
ATE = E[Yi (1) − Yi (0)]
Average Treatment Effect on the Treated (ATET)
ATET = E[Yi (1) − Yi (0) | Wi = 1]

42. Average Causal/Treatment Effects
Average Treatment Effect (ATE)
ATE = E[Yi (1) − Yi (0)]
Average Treatment Effect on the Treated (ATET)
ATET = E[Yi (1) − Yi (0) | Wi = 1]
Causal Lift (L) (this talk)
L =
E[Yi (1) | Wi = 1]
E[Yi (0) | Wi = 1]
− 1

43. Average Causal/Treatment Effects
Average Treatment Effect (ATE)
ATE = E[Yi (1) − Yi (0)]
Average Treatment Effect on the Treated (ATET)
ATET = E[Yi (1) − Yi (0) | Wi = 1]
Causal Lift (L) (this talk)
L =
E[Yi (1) | Wi = 1]
E[Yi (0) | Wi = 1]
− 1
Conditional Average Treatment Effect: (Athey/Imbens et al)
τ(x) = E[Yi (1) − Yi (0) | Xi = x]

44. Average Causal/Treatment Effects
Average Treatment Effect (ATE)
ATE = E[Yi (1) − Yi (0)]
Average Treatment Effect on the Treated (ATET)
ATET = E[Yi (1) − Yi (0) | Wi = 1]
Causal Lift (L) (this talk)
L =
E[Yi (1) | Wi = 1]
E[Yi (0) | Wi = 1]
− 1
Conditional Average Treatment Effect: (Athey/Imbens et al)
τ(x) = E[Yi (1) − Yi (0) | Xi = x]
Conditional Response Rate (usual Machine Learning problem)
R(x) = E[Yi (1) | Xi = x]

45. Causal Effect Illustration

46. Causal Effect Illustration

47. Causal Effect Illustration

48. Causal Effect Illustration

49. Causal Effect Illustration

50. Causal Effect Illustration: Counterfactuals

51. Causal Effect Illustration: Counterfactuals

52. Causal Effect Illustration: Counterfactuals

53. Causal Effect with Counterfactuals
Counterfactuals are unobservable!

54. Causal Effect with Counterfactuals
Counterfactuals are unobservable!
Instead of comparing:
Resp-rate of exposed users U vs
Counterfactual un-exposed response-rate of same users U,

55. Causal Effect with Counterfactuals
Counterfactuals are unobservable!
Instead of comparing:
Resp-rate of exposed users U vs
Counterfactual un-exposed response-rate of same users U,
We compare:
Resp-rate of exposed users U vs
Resp-rate of un-exposed users statistically equivalent to U.

56. Causal Effect with Counterfactuals
Counterfactuals are unobservable!
Instead of comparing:
Resp-rate of exposed users U vs
Counterfactual un-exposed response-rate of same users U,
We compare:
Resp-rate of exposed users U vs
Resp-rate of un-exposed users statistically equivalent to U.
=⇒ using randomization

57. Ideal Randomized Test:
Randomize after winning bid

58. Ideal Randomized Test

59. Ideal Randomized Test

60. Ideal Randomized Test

61. Ideal Randomized Test: Ad lift

62. Ideal Randomized Test: Ad lift

63. But is this practical?

64. Ideal Randomized Test

65. Ideal Randomized Test

66. Ideal Randomized Test

67. Ideal Randomized Test: Wasted spend

69. MediaMath’s approach:
Randomize before bidding

70. A Less Wasteful Randomized Test

71. A Less Wasteful Randomized Test

72. A Less Wasteful Randomized Test

73. Compare RC vs RT ?

74. Compare RC vs RT ?

75. Compare RC vs RT ?

76. Compare RC vs RTW ?

77. Compare RC vs RTW ? Win-bias

78. Ad Lift: Proper Definition

79. Ad Lift: Proper Definition

80. Ad Lift: Proper Definition

81. Ad Lift: Proper Definition

82. Estimating the Counterfactual RCW

83. Estimating the Counterfactual RCW

84. Estimating the Counterfactual RCW

85. Estimating the Counterfactual RCW

86. Estimating the Counterfactual RCW

87. Estimating the Counterfactual RCW

88. Estimating the Counterfactual RCW

89. Ad Lift Estimation
Main steps:
observe response rates RC , RTW , RTL

90. Ad Lift Estimation
Main steps:
observe response rates RC , RTW , RTL
observe test win-rate w

91. Ad Lift Estimation
Main steps:
observe response rates RC , RTW , RTL
observe test win-rate w
estimate the control counterfactual winner response-rate
RCW =
RC − (1 − w)RTL
w

92. Ad Lift Estimation
Main steps:
observe response rates RC , RTW , RTL
observe test win-rate w
estimate the control counterfactual winner response-rate
RCW =
RC − (1 − w)RTL
w
compute lift L = RTW /RCW − 1

93. Ad Lift Estimation
Main steps:
observe response rates RC , RTW , RTL
observe test win-rate w
estimate the control counterfactual winner response-rate
RCW =
RC − (1 − w)RTL
w
compute lift L = RTW /RCW − 1
similar to Treatment Effect Under Non-compliance in clinicial
trials.

94. Ad Lift Estimation
How to compute the 90% confidence interval for L?

96. Ad Lift: Confidence Intervals with Gibbs sampler

97. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach
Assume a random parameter vector θ consisting of:

98. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach
Assume a random parameter vector θ consisting of:
(RTW , RL, RCW , w, ...)

99. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach
Assume a random parameter vector θ consisting of:
(RTW , RL, RCW , w, ...)
Set up prior distribution on θ ∼ p(θ)

100. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach
Assume a random parameter vector θ consisting of:
(RTW , RL, RCW , w, ...)
Set up prior distribution on θ ∼ p(θ)
Sample M values of unknown θ from posterior: Gibbs Sampler
P(θ |Data) ∝ P(Data | θ) · p(θ)

101. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach
Assume a random parameter vector θ consisting of:
(RTW , RL, RCW , w, ...)
Set up prior distribution on θ ∼ p(θ)
Sample M values of unknown θ from posterior: Gibbs Sampler
P(θ |Data) ∝ P(Data | θ) · p(θ)
For each sampled θ compute lift L = RTW /RCW − 1

102. Ad Lift: Confidence Intervals with Gibbs sampler
Bayesian approach
Assume a random parameter vector θ consisting of:
(RTW , RL, RCW , w, ...)
Set up prior distribution on θ ∼ p(θ)
Sample M values of unknown θ from posterior: Gibbs Sampler
P(θ |Data) ∝ P(Data | θ) · p(θ)
For each sampled θ compute lift L = RTW /RCW − 1
Compute (0.05, 0.95) quantiles of sampled L values

103. Ad Lift: Gibbs Sampling

104. Ad Lift: Gibbs Sampling

105. Ad Lift: Gibbs Sampling

106. Ad Lift: Gibbs Sampling

107. Ad Lift: Gibbs Sampling

108. Ad Lift: Gibbs Sampling

109. Ad Lift: Gibbs Sampling

110. Ad Lift: Gibbs Sampling

111. Ad Lift: Gibbs Sampling

112. Ad Lift Gibbs Sampling: Random variables
Probabilities: w, RTW , RCW , RL
Counts: CW 0, CW 1, CL0, CL1

113. Ad Lift Gibbs Sampling: Random variables
Probabilities: w, RTW , RCW , RL
Counts: CW 0, CW 1, CL0, CL1
Beta(1, 1) priors on probabilities, e.g.:
w ∼ Beta(1, 1) ∼ Uniform(0, 1), . . .

114. Ad Lift Gibbs Sampling: Posterior Probabilities
Likelihood of observed
k = CL1 + TL1 conversions out of
n = CL1 + TL1 + CL0 + TL0 trials,
given loser reponse-rate RL:

115. Ad Lift Gibbs Sampling: Posterior Probabilities
Likelihood of observed
k = CL1 + TL1 conversions out of
n = CL1 + TL1 + CL0 + TL0 trials,
given loser reponse-rate RL:
Binom(k, n; RL) ∝ Rk
L (1 − RL)n−k
,

116. Ad Lift Gibbs Sampling: Posterior Probabilities
Likelihood of observed
k = CL1 + TL1 conversions out of
n = CL1 + TL1 + CL0 + TL0 trials,
given loser reponse-rate RL:
Binom(k, n; RL) ∝ Rk
L (1 − RL)n−k
,
so posterior of RL
P(RL | k, n) ∝ P(k, n | RL) · p(RL)

117. Ad Lift Gibbs Sampling: Posterior Probabilities
Likelihood of observed
k = CL1 + TL1 conversions out of
n = CL1 + TL1 + CL0 + TL0 trials,
given loser reponse-rate RL:
Binom(k, n; RL) ∝ Rk
L (1 − RL)n−k
,
so posterior of RL
P(RL | k, n) ∝ P(k, n | RL) · p(RL)
∝ Rk
L (1 − RL)n−k
· Beta(1, 1)

118. Ad Lift Gibbs Sampling: Posterior Probabilities
Likelihood of observed
k = CL1 + TL1 conversions out of
n = CL1 + TL1 + CL0 + TL0 trials,
given loser reponse-rate RL:
Binom(k, n; RL) ∝ Rk
L (1 − RL)n−k
,
so posterior of RL
P(RL | k, n) ∝ P(k, n | RL) · p(RL)
∝ Rk
L (1 − RL)n−k
· Beta(1, 1)
∝ Rk+1
L (1 − RL)n−k+1

119. Ad Lift Gibbs Sampling: Posterior Probabilities
Likelihood of observed
k = CL1 + TL1 conversions out of
n = CL1 + TL1 + CL0 + TL0 trials,
given loser reponse-rate RL:
Binom(k, n; RL) ∝ Rk
L (1 − RL)n−k
,
so posterior of RL
P(RL | k, n) ∝ P(k, n | RL) · p(RL)
∝ Rk
L (1 − RL)n−k
· Beta(1, 1)
∝ Rk+1
L (1 − RL)n−k+1
∝ Beta(k + 1, n − k + 1)

120. Ad Lift Gibbs Sampling: Posterior Counts
We observe C1 = CL1 + CW 1 (total control conversions).
Need to sample CL1, CW 1

121. Ad Lift Gibbs Sampling: Posterior Counts
We observe C1 = CL1 + CW 1 (total control conversions).
Need to sample CL1, CW 1
CW 1 is a Binomial draw from n = C1, with probability:
P(ctl winner | ctl conversion) =
w · RCW
w · RCW + (1 − w) · RL

122. Ad Lift Gibbs Sampling: Posterior Counts
We observe C1 = CL1 + CW 1 (total control conversions).
Need to sample CL1, CW 1
CW 1 is a Binomial draw from n = C1, with probability:
P(ctl winner | ctl conversion) =
w · RCW
w · RCW + (1 − w) · RL
CL1 = C1 − CW 1

123. Complication 1: We only observe cookies, not users;
A user’s cookies may be in both test and control
(Contamination)

125. Control Contamination due to Multiple Cookies

126. Control Contamination due to Multiple Cookies

127. Control Contamination due to Multiple Cookies

128. Control Contamination due to Multiple Cookies

129. Control Contamination due to Multiple Cookies

130. Cookie-Contamination Questions
How does cookie contamination affect measured lift?

131. Cookie-Contamination Questions
How does cookie contamination affect measured lift?
Does the cookie-distribution matter?

132. Cookie-Contamination Questions
How does cookie contamination affect measured lift?
Does the cookie-distribution matter?
everyone has k cookies vs an average of k cookies

133. Cookie-Contamination Questions
How does cookie contamination affect measured lift?
Does the cookie-distribution matter?
everyone has k cookies vs an average of k cookies
What is the influence of the control percentage?

134. Cookie-Contamination Questions
How does cookie contamination affect measured lift?
Does the cookie-distribution matter?
everyone has k cookies vs an average of k cookies
What is the influence of the control percentage?
Simulations best way to understand this

135. Cookie-Contamination Questions
How does cookie contamination affect measured lift?
Does the cookie-distribution matter?
everyone has k cookies vs an average of k cookies
What is the influence of the control percentage?
Simulations best way to understand this
Monte carlo simulations using Spark

136. Simulations for cookie-contamination
A scenario is a combination of parameters:
M = # trials for this scenario, usually 10K-1M
n = # users, typically 10K - 10M
p = # control percentage (usually 10-50%)
k = cookie-distribution, expressed as 1 : 100, or 1 : 70, 3 : 30
r = (un-contaminated) control user response rate
a = true lift, i.e. exposed user response rate = r ∗ (1 + a).
A scenario file specifies a scenario in each row.
could be thousands of scenarios

138. Complication 2:
Long-running experiments

139. Long-Running Experiments
Ideal randomized test is instantaneous.

140. Long-Running Experiments
Ideal randomized test is instantaneous.
When a test is run for weeks/months,
A test user may sometimes be a winner, sometimes loser.
How to define who is a “winner” and “loser”?
Crucial because lift L = RTW /RCW − 1.

141. Long-Running Experiments
Ideal randomized test is instantaneous.
When a test is run for weeks/months,
A test user may sometimes be a winner, sometimes loser.
How to define who is a “winner” and “loser”?
Crucial because lift L = RTW /RCW − 1.
Our approach (details omitted):
Ad influence period is limited
“refresh” a user after suitable time-period elapses.
Count “user time-spans” rather than “users”
Identify “experiments” within user’s time-line

142. MediaMath’s Placebo App
Currently in production for ∼ 10 advertisers
Advertisers can specify which campaigns to measure
Lift estimation, Gibbs Sampling runs on AWS using Spark
Multiple runs of Gibbs Sampler in parallel (with different priors)

143. Thank you!
pchalasani@mediamath.com