- A/B testing involves randomized controlled experiments comparing a treatment group to a control group. However, there are various sources of variability beyond just the treatment that must be accounted for. - Good experiment design aims to minimize bias and convert it to random noise through randomization. The role of statistics is to quantify the magnitude of the treatment effect compared to the noise. - Classical hypothesis testing approaches the problem as "assuming no difference and seeing if the data contradicts that". However, concerns with this approach include overreliance on p-values and not addressing multiple testing. - Bayesian approaches consider the probability of there being a difference given the data, but require specifying a prior probability which is challenging. Alternatives like multi-

1. Crash Course in
A/B testing
A statistical perspective
Wayne Tai Lee

2. Roadmap
• What is A/B testing?
•
•
•
•
•
•
Good experiments and the role of statistics
Similar to proof by contradiction
“Tests”
Big data meets classic asymptotics
Complaints with classical hypothesis testing
Alternatives?

3. What is A/B Testing
• An industry term for controlled and randomized experiment
between treatment/control groups.
• Age old problem….especially with humans

4. What most people know:
Gather samples
Apply treatments
Compare
Measure Outcome
Assign treatments
A
?
B

5. What most people know:
Only difference is in the treatment!
A
?
B

6. Reality:
Variability from
Samples/Inputs
Variability from
Treatment/function
Variability from
Measurement
A
??????
B
How do we account
for all that?

7. Confounding:
• If there are variabilities in addition to the treatment effect,
how can we identify/isolate the effect from the treatment?

8. 3 Types of Variability:
• Controlled variability
• Systematic and desired
• i.e. our treatment
• Bias
• Systematic but not desired
• Anything that can confound our study
• Noise
• Random error but not desired
• Won’t confound the study but makes it hard to
make a decision.

9. How do we categorize each?
Variability from
Samples/Inputs
Variability from
Treatment/function
Variability from
Measurement
A
??????
B

10. Reality:
Good instrumentation!
A
??????
B

11. Reality:
Randomize assignment!
Convert bias to noise
A
??????
B

12. Reality:
Randomize assignment!
Convert bias to noise
A
??????
B
Your population can be skewed or biased….but
that only restricts the generalizability of the results

13. Reality:
Think about what you want to measure and how!
Minimize the noise level/variability in the metric.
A
?
B

14. A good experiment in general:
- Good design and implementation should be used to avoid bias.
- For unavoidable biases, use randomization to turn it into noise.
- Good planning to minimize noise in data.

15. How do we deal with noise?
- Bread and butter of statisticians!
- Quantify the magnitude of the treatment
- Quantify the magnitude of the noise
- Just compare…..most of the time

16. Formalizing the Comparison
Similar to proof by contradiction
- You assume the difference is by chance (noise)

17. Formalizing the Comparison
Similar to proof by contradiction
- You assume the difference is by chance (noise)
- See how the data contradicts the assumption

18. Formalizing the Comparison
Similar to proof by contradiction
- You assume the difference is by chance (noise)
- See how the data contradicts the assumption
- If the surprise surpasses a threshold, we reject
the assumption.
- ….nothing is “100%”

19. Difference due to chance?
Red -> treatment; Black -> control
ID
Person 1
Person 2
Person 3
Person 4
Person 5
Person 6
PV
39
209
31
98
9
151

20. Difference due to chance?
Red -> treatment; Black -> control
ID
Person 1
Person 2
Person 3
Person 4
Person 5
Person 6
PV
39
209
31
98
9
151
|
|
|
|
|
|
|
Let’s measure the difference in means!
mean
72
mean
124.5
Diff = -52.5
….so what?

21. Difference due to chance?
Red -> treatment; Black -> control
ID
PV
ID
PV
Person 1
Person 2
Person 3
Person 4
Person 5
Person 6
39
209
31
98
9
151
1
2
3
4
5
6
39
209
31
98
9
151
If there was no difference from the treatment, shuffling the treatment status
can emulate the randomization of the samples.

22. Difference due to chance?
Red -> treatment; Black -> control
ID
PV
ID
PV
Person 1
Person 2
Person 3
Person 4
Person 5
Person 6
39
209
31
98
9
151
1
2
3
4
5
6
39
209
31
98
9
151
Diff = 122.25 – 24 = 98.25

23. Difference due to chance?
Red -> treatment; Black -> control
ID
PV
ID
PV
Person 1
Person 2
Person 3
Person 4
Person 5
Person 6
39
209
31
98
9
151
1
2
3
4
5
6
39
209
31
98
9
151
Diff = 107. 5 – 53.5 = 54

24. Difference due to chance?
50000 repeats later…..
Our original -52.5

25. Difference due to chance?
Our original -52.5
46.5% of the permutations yielded a larger if not the same
difference as our original sample (in magnitude).
Are you surprised by the initial results?

26. “Tests”
Congratulations!
- You just learned the permutation test!
- The 46.5% is the p-value under the permutation test.

27. “Tests”
Congratulations!
- You just learned the permutation test!
- The 46.5% is the p-value under the permutation test.
Problems:
- Permuting the labels can be computationally costly.
- Not possible before computers!
- Statistical theory says there are many tests out there.

28. Standard t-test:
1) Calculate delta:
= mean_treatment – mean_control
2) Assumes follows a Normal distribution then calculate
the p-value.
p-value = sum of red areas
-
0
3) If p-value < 0.05 then we reject the assumption that there is no
difference between treatment and control.
28
“Tests”

29. Big data meets classic Stats
29
Wait, our metrics may not be Normal!

30. Big Data meets Classic Stat
We care about the “mean of
the metric” and not the actual
metric distribution.
30
Wait, our metrics may not be Normal!

31. Big Data meets Classic Stat
We care about the “mean of
the metric” and not the actual
metric distribution.
31
Wait, our metrics may not be Normal!
Central Limit Theorem:
The “mean of the metric” will be
Normal if the sample size is LARGE!

32. Assumptions with t-test
- Normality of %delta
- Guaranteed with large sample sizes
- Independent Samples
- Not too many 0’s
That’s IT!!!
- Easy to automate.
- Simple and general.
32
Big Data meets Classic Stat

33. What are “Tests”?
33
• Statistical tests are just procedures that depend on data
to make a decision.
• Engineerify: Statistical tests are functions that take
in data, treatments, and return a boolean.

34. • Statistical tests are just procedures that depend on data
to make a decision.
• Engineerify: Statistical tests are functions that take
in data, treatments, and return a boolean.
Guarantees:
• By setting the p-value to compare to a 5% threshold, we control
P( Test says difference exists | In reality NO difference) <= 5%
34
What are “Tests”?

35. • Statistical tests are just procedures that depend on data
to make a decision.
• Engineerify: Statistical tests are functions that take
in data, treatments, and return a boolean.
Guarantees:
• By setting the p-value to compare to a 5% threshold, we control
P( Test says difference exists | In reality NO difference) <= 5%
• By setting the power of the test to be 80%, we control
P( Test says difference exists | In reality difference exists) >= 80%
35
What are “Tests”?

36. • Statistical tests are just procedures that depend on data
to make a decision.
• Engineerify: Statistical tests are functions that take
in data, treatments, and return a boolean.
Guarantees:
• By setting the p-value to compare to a 5% threshold, we control
P( Test says difference exists | In reality NO difference) <= 5%
• By setting the power of the test to be 80%, we control
P( Test says difference exists | In reality difference exists) >= 80%
• Increasing this often requires more data
36
What are “Tests”?

37. Meaning:
All treatments
No difference
Difference exist
37
Reality
Useless treatments
Impactful treatments

38. Meaning:
All treatments
No difference
Difference exist
38
Reality
Useless treatments
Test Decision
No difference
Difference Exists
Impactful treatments
No difference Difference Exists

39. Meaning:
All treatments
No difference
Difference exist
39
Reality
Useless treatments
Test Decision
Guarantees
through
conventional
thresholds
No difference
>95%
Difference Exists
<=5%
Impactful treatments
No difference Difference Exists
<20%
>=80%

40. Meaning:
All treatments
No difference
Difference exist
40
Reality
Useless treatments
Test Decision
Guarantees
through
conventional
thresholds
Jargon
No difference
>95%
Difference Exists
<=5%
Significance level
Impactful treatments
No difference Difference Exists
<20%
>=80%
Power

41. Meaning:
41
- Most appropriate over repeated decision making
- E.g. spammer or not

42. - Most appropriate over repeated decision making
- E.g. spammer or not
- Not seeing a difference could mean
- There is no difference
- Not enough power
42
Meaning:

43. - Most appropriate over repeated decision making
- E.g. spammer or not
- Not seeing a difference could mean
- There is no difference
- Not enough power
- Seeing a difference could mean
- There is a difference
- Got unlucky/lucky
43
Meaning:

44. - Most appropriate over repeated decision making
- E.g. spammer or not
- Not seeing a difference could mean
- There is no difference
- Not enough power
- Seeing a difference could mean
- There is a difference
- Got unlucky/lucky
- Your specific test is either impactful or not. (100% or 0%)
Not what most people want to hear….
44
Meaning:

45. Complaints with Hypth Testing
45
• People get really stuck on p-values and tests.
• Confusing, boring, and formulaic.

46. Complaints with Hypth Testing
46
• People get really stuck on p-values and tests.
• Confusing, boring, and formulaic.
• Statistical significance != Scientific significance
• You could detect a .000001 difference, so what?

47. • People get really stuck on p-values and tests.
• Confusing, boring, and formulaic.
• Statistical significance != Scientific significance
• You could detect a .000001 difference, so what?
• Multiple Hypothesis testing
• 5% false positive is 1 out of 20. Quite high!
• http://xkcd.com/882/
• Most published results are false still (Ioannidis 2005)
47
Complaints with Hypth Testing

48. • People get really stuck on p-values and tests.
• Confusing, boring, and formulaic.
• Statistical significance != Scientific significance
• You could detect a .000001 difference, so what?
• Multiple Hypothesis testing
• 5% false positive is 1 out of 20. Quite high!
• http://xkcd.com/882/
• Most published results are false still (Ioannidis 2005)
• What is it answering?
• Nothing specific about your test…. probabilities are
over repeated trials.
48
Complaints with Hypth Testing

49. Both children of a British mother died within a short period of
time. Mother was convicted of murder because p-value was low.
If she was innocent, the chance of both children dying is low
p-value = P( two deaths | innocent )
49
Abuse: Prosecutor Fallacy

50. Both children of a British mother died within a short period of
time. Mother was convicted of murder because p-value was low.
If she was innocent, the chance of both children dying is low
p-value = P( two deaths | innocent )
In fact, we should be looking at P( innocent | two deaths )
This is the prosecutor’s fallacy.
50
Abuse: Prosecutor Fallacy

51. Example:
51
All Mothers
Guilty Mothers
Two deaths
Innocent Mothers
Two deaths

52. Example: base line matters!
52
All Mothers
Guilty Mothers
Two deaths
Innocent Mothers
Two deaths
P-value can be small.
But base line can be huge.

53. Any Alternatives?
53
P( innocent | two deaths ) is what we want……
but does it make sense?
Bayesian methodology:
P( difference exists | data )
This requires knowing P(difference exists), i.e. the prior
- Philosophical debate, “What is a probability?”
- Easy to cheat the numbers

54. - How to deal with multiple hypothesis testing?
- What are we doing in the company?
- Rumor has it that “Multi-armed bandit > A/B testing”?
54
Questions?