Slides from Budapest BI Forum 2016-10-26

1. BEYOND BASIC
A/B TESTING
János Divényi
Emarsys
@divenyijanos
divenyijanos.github.io
tabularinsights.com

2. @divenyijanosBudapest BI Forum

3. @divenyijanosBudapest BI Forum 20161026
http://dilbert.com/strip/2001-10-25

4. @divenyijanosBudapest BI Forum 20161026
main
90%
control
10%
1 week
comparison period

5. @divenyijanosBudapest BI Forum 20161026
main
90%
control
10%
1 week
comparison period
for 2 months
~450k people

6. variation average sales improvement lift chance to beat baseline
main 0.0919 -0.0037 -3.87% 10.0%
control 0.0956
@divenyijanosBudapest BI Forum 20161026

7. @divenyijanosBudapest BI Forum 20161026
groupAA <- function(sales_dt, proportion = c(0.9, 0.1)) {
sales_dt[, .(
sales_amount,
simulated_AA_program = sample(
c('main', 'control'),
size = .N,
replace = TRUE,
prob = proportion
)
)]
}

8. @divenyijanosBudapest BI Forum 20161026
compareAA <- function(sales_dt, proportion = c(0.9, 0.1)) {
groupAA(sales_dt, proportion) %>%
.[
,
.(avg_sales = mean(sales_amount)),
by = simulated_AA_program
]
}

9. @divenyijanosBudapest BI Forum 20161026
compareAA <- function(sales_dt, proportion = c(0.9, 0.1)) {
groupAA(sales_dt, proportion) %>%
.[
,
.(avg_sales = mean(sales_amount)),
by = simulated_AA_program
]
}
AA_results <- lapply(
1:10000,
function(i) compareAA(sales_dt)
) %>% rbindlist()

10. @divenyijanosBudapest BI Forum 20161026
sales
main
control

11. @divenyijanosBudapest BI Forum 20161026
95%
sales

12. With a 90/10 split of ~450k customers
we can detect
an effect of at least 50%
in average sales.
@divenyijanosBudapest BI Forum 20161026

13. number of observations
variation of outcome
size of expected effect
@divenyijanosBudapest BI Forum 20161026

14. @divenyijanosBudapest BI Forum 20161026
scales by square root n
for half the effect 4 times as
many observations needed

15. number of observations
variation of outcome
size of expected effect
@divenyijanosBudapest BI Forum 20161026

16. @divenyijanosBudapest BI Forum 20161026
95%
log sales

17. @divenyijanosBudapest BI Forum 20161026
95%
conversion rate

18. avg log sales
main 0.0091 0.0023 34.40% 99.7%
control 0.0068
variation average sales improvement lift chance to beat baseline
main 0.0919 -0.0037 -3.87% 10.0%
control 0.0956
@divenyijanosBudapest BI Forum 20161026

19. avg log sales
main 0.0091 0.0023 34.40% 99.7%
control 0.0068
variation average sales improvement lift chance to beat baseline
main 0.0919 -0.0037 -3.87% 10.0%
control 0.0956
@divenyijanosBudapest BI Forum 20161026
conversion
main 0.32% 0.09%p 41.26% 100.0%
control 0.23%

20. Randomized Block Design
(aka Stratified Sampling)
ensures that less variable groups
are being compared
@divenyijanosBudapest BI Forum 20161026

21. number of observations
variation of outcome
size of expected effect
@divenyijanosBudapest BI Forum 20161026

22. CAUSAL TREE
to look for effect within segments
while avoiding multiple testing
@divenyijanosBudapest BI Forum 20161026

23. CAUSAL TREE

24. @divenyijanosBudapest BI Forum 20161026
BAYESIAN A/B TESTING
P(data|versions are the same)
P(versions are the same|data)

25. @divenyijanosBudapest BI Forum 20161026
BAYESIAN A/B TESTING
P(data|versions are the same)
P(versions are the same|data)