Sara Vafi and Shana Rusonis presented on statistical models for analyzing A/B test results. They discussed the differences between Bayesian and frequentist statistics, and between average error control and all error control. Bayesian A/B testing uses average error control and models results as distributions, while frequentist A/B testing aims for all error control by controlling the false positive rate for each experiment. Small improvements are hardest to detect, and realistic A/B tests may require more data than average error control can provide on its own. Optimizely's Stats Engine takes a blended Bayesian-frequentist approach for more accurate and timely results.

1. Speakers
Statistical Models,
Explored and Explained
Sara Vafi, Stats Expert, Optimizely
Shana Rusonis, Product Marketing, Optimizely

2. Today’s Speakers
Sara Vafi Shana Rusonis

3. Housekeeping
• We’re recording!
• Slides and recording will be
emailed to you tomorrow
• Time for questions at the end

4. Agenda
• Bayesian & Frequentist Statistics
• Error Control - Average vs. All Error Control
• Bayes Rule
• Benefits & Risks
• Optimizely Stats Engine
• Q&A

5. Why Do We Experiment?
● Experimentation is essential for learning
● Try new ideas without fear of failure
● Give your business a signal to act on
in a sea of noisy data

6. What’s most Important to You?
● Running experiments quickly
● But also reporting on results accurately
● When not all statistical solutions are created equal

7. Types of Statistical Methods
Bayesian
OR
Frequentist

8. Bayesian Statistics
● Bayesian statistics take a more bottom-up approach to data analysis
● Our parameters are unknown
● The data is fixed
● There is a prior probability
● “Opinion-based”

9. “A Bayesian is one who, vaguely
expecting a horse, and catching a
glimpse of a donkey, strongly believes
he has seen a mule.”
Source

10. Frequentist Statistics
● Frequentist arguments are more counter-factual in nature
● Parameters remain constant during the repeatable sampling process
● Resemble the type of logic that lawyers use in court
● ‘Is this variation different from the control?’ is a basic building block of this
approach.

11. Example
Dan & Pete Rolling a 6-Sided Die
Scenario:
● Pete will roll a die and the outcome can either be 1, 2, 3, 4, 5, or 6
● If Pete rolls a 4, he will give Dan $1 million
If Dan was a Bayesian statistician, how would he react?
If Dan was a Frequentist statistician, how would he react?

12. Example
Probability of the sun exploding
Source

13. Error Control

14. Error Control Explained
● The likelihood that the observed result of an experiment happened by chance,
rather than a change that you introduced
● When we set the statistical significance on an experiment to 90%, that means
there's a 10% chance of a statistical error, or a 1 in 10 chance that the result
happened by chance

15. Average Error Control
● Corresponds to Bayesian A/B Testing
● Less useful for iterating on test results
● Harder to learn from individual experiments with confidence

16. All Error Control
● Corresponds to Frequentist A/B Testing
● Any experiment will have less than a 10% chance of a mistake
● Rate of errors is 1 in 10

17. Average Error Control vs. All Error Control
● Average error control leads to lower accuracy for small improvements
● All error control is accurate for all users
● There are certain cases where average error control is an appropriate
alternative

18. Error Rates for Experiments

19. Bayes Rule

20. Average Error Control & Bayesian A/B
Testing
● Requires two sources of randomness
○ Randomness or “noise” in the data
○ The makeup of the “typical” experiment group
● Distribution over experiment improvements

21. Different Beliefs in Composition of ‘Typical’ Experiments

22. Bayes Rule

23. Bayes Rule & Bayesian A/B Testing

24. Bayes Rule & Average Error Value

25. Recap Average Error Control
Bayesian A/B Testing
Prior Distributions
Bayes Rule

26. All Error Control is Frequentist A/B Testing
● All error control corresponds to Frequentist AB testing
● We want to aim to control the false positive rate
● Chance an experiment is either called a winner or loser

27. Benefits & Risks

28. Benefits of Bayesian A/B Testing
● Average error control can be very
attractive
● Helps solve the “peeking” problem
● Average error control is fast

29. Risks of Bayesian A/B Testing
● It’s more appealing but it’s risky in practice
● Smaller improvement experiments with fast results = high risk
● Higher error rate than the method actually suggests

30. Benefits of Frequentist A/B Testing
● This type of test will make fewer mistakes on experiments with
non-zero improvements
● The rate of errors will be less than 1 in 10
● Option to speed up experimentation by using a prior

31. Learning from A/B Tests

32. Learning from A/B Tests

33. Risk Involved with Typical Realistic Experiments

34. Realistic Bayesian A/B Tests vs. Stats Engine

35. ● The hardest experiments to call correctly are those with small
improvements
● A/B testing in the wild is not easy
● We need more and more data in order to achieve average error control
on realistic experiments
So what does this mean?

36. Stats Engine

37. Stats EngineTM
Results are valid whenever you
check
Avoid costly statistics errors
Measure real-time results
with confidence

38. Key Takeaways
● Bayesian vs. Frequentist methods
● All error control vs. average error control
● Blended approach leads to greater confidence

39. QUESTIONS?

40. THANK YOU!