Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Spark Summit East Talk

License: CC Attribution-ShareAlike License

No Downloads

Total views

2,621

On SlideShare

0

From Embeds

0

Number of Embeds

542

Shares

0

Downloads

138

Comments

0

Likes

4

No notes for slide

- 1. I listen to ~ 100 Bln ad opportunities daily
- 2. I listen to ~ 100 Bln ad opportunities daily I respond with optimal bids within milliseconds
- 3. I listen to ~ 100 Bln ad opportunities daily I respond with optimal bids within milliseconds I petabytes of data (ad impressions, visits, clicks, conversions)
- 4. Predicting user response to ads is a Machine-Learning problem.
- 5. Predicting user response to ads is a Machine-Learning problem. but quantifying impact of ad-exposure is a Measurement probem.
- 6. Spark: existing vs simulated data Most Spark applications process existing big data-sets.
- 7. Spark: existing vs simulated data Most Spark applications process existing big data-sets. Today we’re talking about analyzing simulated big data
- 8. Key Conceptual Take-aways I Issues in Ad lift measurement
- 9. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition
- 10. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition I Conﬁdence bounds
- 11. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition I Conﬁdence bounds I Bayesian Methods for Ad Lift Conﬁdence Bounds
- 12. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition I Conﬁdence bounds I Bayesian Methods for Ad Lift Conﬁdence Bounds I Gibbs Sampling (MCMC – Markov Chain Monte Carlo)
- 13. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition I Conﬁdence bounds I Bayesian Methods for Ad Lift Conﬁdence Bounds I Gibbs Sampling (MCMC – Markov Chain Monte Carlo) I Using Spark for:
- 14. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition I Conﬁdence bounds I Bayesian Methods for Ad Lift Conﬁdence Bounds I Gibbs Sampling (MCMC – Markov Chain Monte Carlo) I Using Spark for: I Monte Carlo sampling for conﬁdence-bounds
- 15. Key Conceptual Take-aways I Issues in Ad lift measurement I Proper deﬁnition I Conﬁdence bounds I Bayesian Methods for Ad Lift Conﬁdence Bounds I Gibbs Sampling (MCMC – Markov Chain Monte Carlo) I Using Spark for: I Monte Carlo sampling for conﬁdence-bounds I Monte Carlo simulations
- 16. Application context: ad impact measurement I Advertisers want to know the impact of showing ads to users.
- 17. Measuring Ad Impact: Two Approaches I Observational studies:
- 18. Measuring Ad Impact: Two Approaches I Observational studies: I Compare uses who happen to be exposed vs not exposed
- 19. Measuring Ad Impact: Two Approaches I Observational studies: I Compare uses who happen to be exposed vs not exposed I Bias a big issue
- 20. Measuring Ad Impact: Two Approaches I Observational studies: I Compare uses who happen to be exposed vs not exposed I Bias a big issue I Randomized tests:
- 21. Measuring Ad Impact: Two Approaches I Observational studies: I Compare uses who happen to be exposed vs not exposed I Bias a big issue I Randomized tests: I Randomly expose to test, compare with control (un-exposed)
- 22. Ideal Randomized Test
- 23. Ideal Randomized Test
- 24. Ideal Randomized Test
- 25. Ideal Randomized Test: Ad lift
- 26. Ideal Randomized Test: Ad lift
- 27. Ad Lift: Response Rates If we see k = 200 conversions out of N = 10, 000 users, what is a good estimate for the response-rate?
- 28. Ad Lift: Response Rates If we see k = 200 conversions out of N = 10, 000 users, what is a good estimate for the response-rate? Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
- 29. Ad Lift: Response Rates If we see k = 200 conversions out of N = 10, 000 users, what is a good estimate for the response-rate? Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . . But how conﬁdent are we?
- 30. Response Rate 90% Conﬁdence Bounds
- 31. Response Rate 90% Conﬁdence Bounds P(R > ˆR | r = q5) = 5%
- 32. Response Rate 90% Conﬁdence Bounds P(R > ˆR | r = q5) = 5% P(R < ˆR | r = q95) = 5%
- 33. Response-Rate Conﬁdence Bounds
- 34. Response-Rate Conﬁdence Bounds
- 35. Response-Rate Conﬁdence Bounds
- 36. Response-Rate Conﬁdence Bounds How to ﬁnd (q5, q95) ?
- 37. Response-Rate: Bayesian Conﬁdence Bounds Randomly generate response rates that are consistent with the data.
- 38. Response-Rate: Bayesian Conﬁdence Bounds Randomly generate response rates that are consistent with the data. (Sample rates from posterior distribution given data.)
- 39. Response-Rate: Bayesian Conﬁdence Bounds Randomly generate response rates that are consistent with the data. (Sample rates from posterior distribution given data.) Find the (0.05, 0.95) quantiles of these rates.
- 40. Response-Rate: Bayesian Conﬁdence Bounds I Assume an unknown true rate r, with a prior distrib. p(r) I assume p(r) = Beta(1, 1) = Unif (0, 1)
- 41. Response-Rate: Bayesian Conﬁdence Bounds I Assume an unknown true rate r, with a prior distrib. p(r) I assume p(r) = Beta(1, 1) = Unif (0, 1) I Sample from the posterior distribution of the rate r I conditional on the observed data (k conversions out of N) P(r | k) Ã P(k | r) · p(r)
- 42. Response-Rate: Bayesian Conﬁdence Bounds I Assume an unknown true rate r, with a prior distrib. p(r) I assume p(r) = Beta(1, 1) = Unif (0, 1) I Sample from the posterior distribution of the rate r I conditional on the observed data (k conversions out of N) P(r | k) Ã P(k | r) · p(r) Ã rk (1 ≠ r)N≠k · Beta(1, 1)
- 43. Response-Rate: Bayesian Conﬁdence Bounds I Assume an unknown true rate r, with a prior distrib. p(r) I assume p(r) = Beta(1, 1) = Unif (0, 1) I Sample from the posterior distribution of the rate r I conditional on the observed data (k conversions out of N) P(r | k) Ã P(k | r) · p(r) Ã rk (1 ≠ r)N≠k · Beta(1, 1) Ã rk+1 (1 ≠ r)N≠k+1
- 44. Response-Rate: Bayesian Conﬁdence Bounds I Assume an unknown true rate r, with a prior distrib. p(r) I assume p(r) = Beta(1, 1) = Unif (0, 1) I Sample from the posterior distribution of the rate r I conditional on the observed data (k conversions out of N) P(r | k) Ã P(k | r) · p(r) Ã rk (1 ≠ r)N≠k · Beta(1, 1) Ã rk+1 (1 ≠ r)N≠k+1 Ã Beta(k + 1, N ≠ k + 1)
- 45. Response-Rate: Bayesian Conﬁdence Bounds I Assume an unknown true rate r, with a prior distrib. p(r) I assume p(r) = Beta(1, 1) = Unif (0, 1) I Sample from the posterior distribution of the rate r I conditional on the observed data (k conversions out of N) P(r | k) Ã P(k | r) · p(r) Ã rk (1 ≠ r)N≠k · Beta(1, 1) Ã rk+1 (1 ≠ r)N≠k+1 Ã Beta(k + 1, N ≠ k + 1) I Compute (0.05, 0.95) quantiles from the generated rates.
- 46. Response-Rate: Bayesian Conﬁdence Bounds A simple form of Gibbs Sampling (more later): I sample M values of r from posterior P(r | k) ≥ Beta(k + 1, N ≠ k + 1). I compute (0.05, 0.95) quantiles
- 47. Response-Rate: Bayesian Conﬁdence Bounds A simple form of Gibbs Sampling (more later): I sample M values of r from posterior P(r | k) ≥ Beta(k + 1, N ≠ k + 1). I compute (0.05, 0.95) quantiles from numpy.random import beta from scipy.stats.mstats import mquantiles def conf(N, k, samples = 500): rates = beta(k+1, N-k+1, samples) return mquantiles(rates, prob = [0.05, 0.95])
- 48. Response-Rate: Bayesian Conﬁdence Bounds
- 49. Response-Rate: Bayesian Conﬁdence Bounds
- 50. Response-Rate: Bayesian Conﬁdence Bounds
- 51. Response-Rate: Bayesian Conﬁdence Bounds
- 52. Response-Rate: Bayesian Conﬁdence Bounds
- 53. Response Rates: Example If we see k = 200 conversions out of N = 10, 000 users, what is a good estimate for the response-rate? Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . .
- 54. Response Rates: Example If we see k = 200 conversions out of N = 10, 000 users, what is a good estimate for the response-rate? Estimated response-rate ˆR = k/N = 200/10, 000 = 2%. . . =∆ 90% conﬁdence region (1.8%, 2.2%)
- 55. We’ve talked about Response Rates. . . now let’s consider Ad Lift
- 56. Ad Lift: Simple Example I control: 10,000 users, 200 conversions I test: 100,000 users, 2200 conversions Observed response-rates: I control: ˆRc = 200/10, 000 = 2% I test: ˆRt = 2200/100, 000 = 2.2% Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
- 57. Ad Lift: Simple Example I control: 10,000 users, 200 conversions I test: 100,000 users, 2200 conversions Observed response-rates: I control: ˆRc = 200/10, 000 = 2% I test: ˆRt = 2200/100, 000 = 2.2% Estimated Lift ˆL = 2.2/2 ≠ 1 = 10% This is a great lift !
- 58. Ad Lift: Simple Example I control: 10,000 users, 200 conversions I test: 100,000 users, 2200 conversions Observed response-rates: I control: ˆRc = 200/10, 000 = 2% I test: ˆRt = 2200/100, 000 = 2.2% Estimated Lift ˆL = 2.2/2 ≠ 1 = 10% This is a great lift ! Not so fast! Is this a reliable estimate?
- 59. Ad Lift: Simple Example I control: 10,000 users, 200 conversions I test: 100,000 users, 2200 conversions Observed response-rates: I control: ˆRc = 200/10, 000 = 2% I test: ˆRt = 2200/100, 000 = 2.2% Estimated Lift ˆL = 2.2/2 ≠ 1 = 10% This is a great lift ! Not so fast! Is this a reliable estimate? Could true lift ¸ be 0%, or even negative ?
- 60. Ad Lift: Bayesian Conﬁdence Bounds Sampling approach: Observed data: control: (kc, Nc), test: (kt, Nt) 1. Repeat M times:
- 61. Ad Lift: Bayesian Conﬁdence Bounds Sampling approach: Observed data: control: (kc, Nc), test: (kt, Nt) 1. Repeat M times: I draw control response rate rc from posterior P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1).
- 62. Ad Lift: Bayesian Conﬁdence Bounds Sampling approach: Observed data: control: (kc, Nc), test: (kt, Nt) 1. Repeat M times: I draw control response rate rc from posterior P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1). I draw test response rate rt from posterior P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1).
- 63. Ad Lift: Bayesian Conﬁdence Bounds Sampling approach: Observed data: control: (kc, Nc), test: (kt, Nt) 1. Repeat M times: I draw control response rate rc from posterior P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1). I draw test response rate rt from posterior P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1). I compute lift L = rt/rc ≠ 1
- 64. Ad Lift: Bayesian Conﬁdence Bounds Sampling approach: Observed data: control: (kc, Nc), test: (kt, Nt) 1. Repeat M times: I draw control response rate rc from posterior P(rc | kc) ≥ Beta(kc + 1, Nc ≠ kc + 1). I draw test response rate rt from posterior P(rt | kt) ≥ Beta(kt + 1, Nt ≠ kt + 1). I compute lift L = rt/rc ≠ 1 2. Compute (0.05, 0.95) quantiles of set of M lifts {L}.
- 65. Ad Lift: Bayesian Conﬁdence Intervals I control: nc = 10, 000 users, kc = 200 conversions I test: nt = 100, 000 users, kt = 2, 200 conversions Observed response-rates: I control: ˆRc = 200/10, 000 = 2% I test: ˆRt = 2200/100, 000 = 2.2% Estimated Lift ˆL = 2.2/2 ≠ 1 = 10%
- 66. Ad Lift: Bayesian Conﬁdence Intervals I control: nc = 10, 000 users, kc = 200 conversions I test: nt = 100, 000 users, kt = 2, 200 conversions Observed response-rates: I control: ˆRc = 200/10, 000 = 2% I test: ˆRt = 2200/100, 000 = 2.2% Estimated Lift ˆL = 2.2/2 ≠ 1 = 10% 90% conﬁdence interval: (≠2.7%, 23.6%)
- 67. Complication 1: Auction win-bias
- 68. Ideal Randomized Test
- 69. Ideal Randomized Test
- 70. Ideal Randomized Test
- 71. Ideal Randomized Test Bids on control users are wasted!
- 72. A Less Wasteful Randomized Test
- 73. A Less Wasteful Randomized Test: Win-bias Cannot simply compare Test Winners (tw) and Control (c): I test-winners selection bias: “win bias”
- 74. Ad Lift: Proper Deﬁnition
- 75. Ad Lift: Proper Deﬁnition
- 76. Ad Lift: Proper Deﬁnition
- 77. Ad Lift: Proper Deﬁnition
- 78. Ad Lift: Proper Deﬁnition
- 79. Ad Lift: Proper Deﬁnition
- 80. Ad Lift Estimation Main ideas: I observe test-losers response rate RtL
- 81. Ad Lift Estimation Main ideas: I observe test-losers response rate RtL I observe test win-rate w
- 82. Ad Lift Estimation Main ideas: I observe test-losers response rate RtL I observe test win-rate w I we show one can estimate R0 tw = Rc ≠ (1 ≠ w)RtL w
- 83. Ad Lift Estimation Main ideas: I observe test-losers response rate RtL I observe test win-rate w I we show one can estimate R0 tw = Rc ≠ (1 ≠ w)RtL w I compute lift L = R1 tw /R0 tw ≠ 1
- 84. Ad Lift Estimation Main ideas: I observe test-losers response rate RtL I observe test win-rate w I we show one can estimate R0 tw = Rc ≠ (1 ≠ w)RtL w I compute lift L = R1 tw /R0 tw ≠ 1 I similar to Treatment E ect Under Non-compliance in clinicial trials.
- 85. Ad Lift Estimation How to compute the 90% conﬁdence interval for L?
- 86. Ad Lift: Conﬁdence Intervals with Gibbs sampler
- 87. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of:
- 88. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of: I user latent (potential) behaviors
- 89. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of: I user latent (potential) behaviors I their probabilities
- 90. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of: I user latent (potential) behaviors I their probabilities I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet)
- 91. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of: I user latent (potential) behaviors I their probabilities I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet) I Sample M values of unknown ◊ from posterior: Gibbs Sampler P(◊ |Data) Ã P(Data | ◊) · p(◊)
- 92. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of: I user latent (potential) behaviors I their probabilities I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet) I Sample M values of unknown ◊ from posterior: Gibbs Sampler P(◊ |Data) Ã P(Data | ◊) · p(◊) I For each sampled ◊ compute lift L using above
- 93. Ad Lift: Conﬁdence Intervals with Gibbs sampler Bayesian approach (details omitted, see Chickering/Pearl 1997): I Assume a random parameter vector ◊ consisting of: I user latent (potential) behaviors I their probabilities I Set up prior distribution on ◊ ≥ p(◊) (Dirichlet) I Sample M values of unknown ◊ from posterior: Gibbs Sampler P(◊ |Data) Ã P(Data | ◊) · p(◊) I For each sampled ◊ compute lift L using above I Compute (0.05, 0.95) quantiles of sampled L values
- 94. Ad Lift: Conﬁdence Intervals
- 95. Ad Lift: Conﬁdence Intervals Gibbs sampler convergence may depend on prior distribution: I start with multiple (say 100) priors I run them all in parallel using Spark.
- 96. Uses of Monte Carlo Simulations I conﬁdence intervals
- 97. Uses of Monte Carlo Simulations I conﬁdence intervals I determine “su cient” population sizes for reliably estimating
- 98. Uses of Monte Carlo Simulations I conﬁdence intervals I determine “su cient” population sizes for reliably estimating I response rates
- 99. Uses of Monte Carlo Simulations I conﬁdence intervals I determine “su cient” population sizes for reliably estimating I response rates I lift
- 100. Uses of Monte Carlo Simulations I conﬁdence intervals I determine “su cient” population sizes for reliably estimating I response rates I lift I understand e ect of complex phenomena
- 101. Uses of Monte Carlo Simulations I conﬁdence intervals I determine “su cient” population sizes for reliably estimating I response rates I lift I understand e ect of complex phenomena I validate/verify analytical formulas
- 102. Complication 2: Control contamination due to users with multiple cookies
- 103. Control Contamination due to Multiple Cookies
- 104. Control Contamination due to Multiple Cookies
- 105. Control Contamination due to Multiple Cookies
- 106. Control Contamination due to Multiple Cookies
- 107. Control Contamination due to Multiple Cookies
- 108. Cookie-Contamination Questions I How does cookie contamination a ect measured lift?
- 109. Cookie-Contamination Questions I How does cookie contamination a ect measured lift? I Does the cookie-distribution matter?
- 110. Cookie-Contamination Questions I How does cookie contamination a ect measured lift? I Does the cookie-distribution matter? I everyone has k cookies vs an average of k cookies
- 111. Cookie-Contamination Questions I How does cookie contamination a ect measured lift? I Does the cookie-distribution matter? I everyone has k cookies vs an average of k cookies I What is the inﬂuence of the control percentage?
- 112. Cookie-Contamination Questions I How does cookie contamination a ect measured lift? I Does the cookie-distribution matter? I everyone has k cookies vs an average of k cookies I What is the inﬂuence of the control percentage? I Simulations best way to understand this
- 113. Simulations for cookie-contamination I A scenario is a combination of parameters: I M = # trials for this scenario, usually 10K-1M I n = # users, typically 10K - 10M I p = # control percentage (usually 10-50%) I k = cookie-distribution, expressed as 1 : 100, or 1 : 70, 3 : 30 I r = (un-contaminated) control user response rate I a = true lift, i.e. exposed user response rate = r ú (1 + a). I A scenario ﬁle speciﬁes a scenario in each row. I could be thousands of scenarios
- 114. Scenario Simulations in Spark
- 115. Scenario Simulations in Spark
- 116. Scenario Simulations in Spark
- 117. Scenario Simulations in Spark
- 118. Scenario Simulations in Spark

No public clipboards found for this slide

Be the first to comment