Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Statistical Models Explored and Explained

1,237 views

Published on

Statistical models used by A/B testing solutions vary greatly. To interpret your test results with accuracy, you need to be well-versed in the approach your testing solution uses to calculate significance. In this presentation Optimizely stats experts will provide a hard-nosed look at a range of statistical models, the risks and tradeoffs associated with each and explain how not all models are created equal.

Check out these slides to learn:
- How testing solutions use Frequentist and Bayesian models to compute significance
- A refresh on core statistical concepts including significance, error, and more
- How Optimizely’s Stats Engine mitigates risk while allowing experimenters to make decisions quickly


Published in: Technology
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Statistical Models Explored and Explained

  1. 1. Speakers Statistical Models, Explored and Explained Sara Vafi, Stats Expert, Optimizely Shana Rusonis, Product Marketing, Optimizely
  2. 2. Today’s Speakers Sara Vafi Shana Rusonis
  3. 3. Housekeeping • We’re recording! • Slides and recording will be emailed to you tomorrow • Time for questions at the end
  4. 4. Agenda • Bayesian & Frequentist Statistics • Error Control - Average vs. All Error Control • Bayes Rule • Benefits & Risks • Optimizely Stats Engine • Q&A
  5. 5. Why Do We Experiment? ● Experimentation is essential for learning ● Try new ideas without fear of failure ● Give your business a signal to act on in a sea of noisy data
  6. 6. What’s most Important to You? ● Running experiments quickly ● But also reporting on results accurately ● When not all statistical solutions are created equal
  7. 7. Types of Statistical Methods Bayesian OR Frequentist
  8. 8. Bayesian Statistics ● Bayesian statistics take a more bottom-up approach to data analysis ● Our parameters are unknown ● The data is fixed ● There is a prior probability ● “Opinion-based”
  9. 9. “A Bayesian is one who, vaguely expecting a horse, and catching a glimpse of a donkey, strongly believes he has seen a mule.” Source
  10. 10. Frequentist Statistics ● Frequentist arguments are more counter-factual in nature ● Parameters remain constant during the repeatable sampling process ● Resemble the type of logic that lawyers use in court ● ‘Is this variation different from the control?’ is a basic building block of this approach.
  11. 11. Example Dan & Pete Rolling a 6-Sided Die Scenario: ● Pete will roll a die and the outcome can either be 1, 2, 3, 4, 5, or 6 ● If Pete rolls a 4, he will give Dan $1 million If Dan was a Bayesian statistician, how would he react? If Dan was a Frequentist statistician, how would he react?
  12. 12. Example Probability of the sun exploding Source
  13. 13. Error Control
  14. 14. Error Control Explained ● The likelihood that the observed result of an experiment happened by chance, rather than a change that you introduced ● When we set the statistical significance on an experiment to 90%, that means there's a 10% chance of a statistical error, or a 1 in 10 chance that the result happened by chance
  15. 15. Average Error Control ● Corresponds to Bayesian A/B Testing ● Less useful for iterating on test results ● Harder to learn from individual experiments with confidence
  16. 16. All Error Control ● Corresponds to Frequentist A/B Testing ● Any experiment will have less than a 10% chance of a mistake ● Rate of errors is 1 in 10
  17. 17. Average Error Control vs. All Error Control ● Average error control leads to lower accuracy for small improvements ● All error control is accurate for all users ● There are certain cases where average error control is an appropriate alternative
  18. 18. Error Rates for Experiments
  19. 19. Bayes Rule
  20. 20. Average Error Control & Bayesian A/B Testing ● Requires two sources of randomness ○ Randomness or “noise” in the data ○ The makeup of the “typical” experiment group ● Distribution over experiment improvements
  21. 21. Different Beliefs in Composition of ‘Typical’ Experiments
  22. 22. Bayes Rule
  23. 23. Bayes Rule & Bayesian A/B Testing
  24. 24. Bayes Rule & Average Error Value
  25. 25. Recap Average Error Control Bayesian A/B Testing Prior Distributions Bayes Rule
  26. 26. All Error Control is Frequentist A/B Testing ● All error control corresponds to Frequentist AB testing ● We want to aim to control the false positive rate ● Chance an experiment is either called a winner or loser
  27. 27. Benefits & Risks
  28. 28. Benefits of Bayesian A/B Testing ● Average error control can be very attractive ● Helps solve the “peeking” problem ● Average error control is fast
  29. 29. Risks of Bayesian A/B Testing ● It’s more appealing but it’s risky in practice ● Smaller improvement experiments with fast results = high risk ● Higher error rate than the method actually suggests
  30. 30. Benefits of Frequentist A/B Testing ● This type of test will make fewer mistakes on experiments with non-zero improvements ● The rate of errors will be less than 1 in 10 ● Option to speed up experimentation by using a prior
  31. 31. Learning from A/B Tests
  32. 32. Learning from A/B Tests
  33. 33. Risk Involved with Typical Realistic Experiments
  34. 34. Realistic Bayesian A/B Tests vs. Stats Engine
  35. 35. ● The hardest experiments to call correctly are those with small improvements ● A/B testing in the wild is not easy ● We need more and more data in order to achieve average error control on realistic experiments So what does this mean?
  36. 36. Stats Engine
  37. 37. Stats EngineTM Results are valid whenever you check Avoid costly statistics errors Measure real-time results with confidence
  38. 38. Key Takeaways ● Bayesian vs. Frequentist methods ● All error control vs. average error control ● Blended approach leads to greater confidence
  39. 39. QUESTIONS?
  40. 40. THANK YOU!

×