Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
KAISER FUNG
Aug. 26, 2014
New York City
kaiserdatamatter@gmail
Q4
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
Testing has played a big part in my career
(c) Kaiser Fung
Q4
~ Testing
U wanted 2
But Didn’t
1. Why worry about
statistical significance?
2. Can I use a simple
sample size formula?
...
Varying Copy on a Conversion Page
(c) Kaiser Fung
Not Statistically Significant
(c) Kaiser Fung
Q*Q3
~ Testing
U wanted 2
But Didn’t
1
Why should
we care about
statistical
significance?
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
Statistical Noise
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
(c) Kaiser Fung
Statistical Noise
(c) Kaiser Fung
Icon by Laura Begg
40,000
(c) Kaiser Fung
20,000
(c) Kaiser Fung
30,000
(c) Kaiser Fung
50:50
(c) Kaiser Fung
15k 15k (c) Kaiser Fung
125
(c) Kaiser Fung
2
Can we use a
simple sample
sizing formula?
Q*Q*Q2
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
10%90%
(c) Kaiser Fung
More Samples
More Precision
(c) Kaiser Fung
(c) Kaiser Fung
3
Is 90% confidence
too risk averse?
Q2*Q*Q
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
OMG
(c) Kaiser Fung
90% Confidence means:
If the analyst tells me the test is
significant, there is a 90% chance
this effect is really there
(c)...
90% Confidence means:
If the analyst tells me the test is
significant, there is a 90% chance
this effect is really there
Goo...
90% Confidence means:
If the effect is really there, there is a
90% chance the analyst will tell me
the test is significant
...
90% Confidence means:
If the effect is really there, there is a
90% chance the analyst will tell me
the test is significant
...
Positive Predictive Value:
If the analyst tells me the test is
significant, what is the probability
this effect is really t...
90% Confidence =
63% PPV
(c) Kaiser Fung
PPV = 45-60%
(c) Kaiser Fung
Chapter 4:
Timid Testers /
Magic Lassos
(c) Kaiser Fung
4
What’s the value
of testing?
Q3*Q
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung
BEFORE AFTER
xkcd (c) Kaiser Fung
B.T.
30%
70% 63%
37%
A.T.
(c) Kaiser Fung
1. Why statistical
significance
2. How sample
sizing
3. What
confidence level
4. Why test
Statistical Noise
Safe Landing
Mea...
KAISER FUNG
Strategic data advisory services,
including test planning & analysis
Speaking on business analytics & dataviz
...
Thank you
Kaiser Fung
kaiserdatamatter@gmail.com
Twitter: @junkcharts
LinkedIn
(c) Kaiser Fung
Upcoming SlideShare
Loading in …5
×

The Optimizely Experience - Kaiser Fung

This presentation was given at the Optimizely Experience New York City by data scientist, Kaiser Fung. The title of his presentation is 4 questions about testing you wanted to ask but didn't.

  • Be the first to comment

The Optimizely Experience - Kaiser Fung

  1. 1. KAISER FUNG Aug. 26, 2014 New York City kaiserdatamatter@gmail Q4 ~ Testing U wanted 2 But Didn’t (c) Kaiser Fung
  2. 2. Testing has played a big part in my career (c) Kaiser Fung
  3. 3. Q4 ~ Testing U wanted 2 But Didn’t 1. Why worry about statistical significance? 2. Can I use a simple sample size formula? 3. Is 90% confidence too risk-averse? 4. What is the value of testing? (c) Kaiser Fung
  4. 4. Varying Copy on a Conversion Page (c) Kaiser Fung
  5. 5. Not Statistically Significant (c) Kaiser Fung
  6. 6. Q*Q3 ~ Testing U wanted 2 But Didn’t 1 Why should we care about statistical significance? (c) Kaiser Fung
  7. 7. (c) Kaiser Fung
  8. 8. (c) Kaiser Fung
  9. 9. Statistical Noise (c) Kaiser Fung
  10. 10. (c) Kaiser Fung
  11. 11. (c) Kaiser Fung
  12. 12. (c) Kaiser Fung
  13. 13. (c) Kaiser Fung
  14. 14. Statistical Noise (c) Kaiser Fung
  15. 15. Icon by Laura Begg 40,000 (c) Kaiser Fung
  16. 16. 20,000 (c) Kaiser Fung
  17. 17. 30,000 (c) Kaiser Fung
  18. 18. 50:50 (c) Kaiser Fung
  19. 19. 15k 15k (c) Kaiser Fung
  20. 20. 125 (c) Kaiser Fung
  21. 21. 2 Can we use a simple sample sizing formula? Q*Q*Q2 ~ Testing U wanted 2 But Didn’t (c) Kaiser Fung
  22. 22. 10%90% (c) Kaiser Fung
  23. 23. More Samples More Precision (c) Kaiser Fung
  24. 24. (c) Kaiser Fung
  25. 25. 3 Is 90% confidence too risk averse? Q2*Q*Q ~ Testing U wanted 2 But Didn’t (c) Kaiser Fung
  26. 26. OMG (c) Kaiser Fung
  27. 27. 90% Confidence means: If the analyst tells me the test is significant, there is a 90% chance this effect is really there (c) Kaiser Fung
  28. 28. 90% Confidence means: If the analyst tells me the test is significant, there is a 90% chance this effect is really there Good Question – Wrong Definition (c) Kaiser Fung
  29. 29. 90% Confidence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is significant (c) Kaiser Fung
  30. 30. 90% Confidence means: If the effect is really there, there is a 90% chance the analyst will tell me the test is significant Correct Definition – Wrong Question (c) Kaiser Fung
  31. 31. Positive Predictive Value: If the analyst tells me the test is significant, what is the probability this effect is really there? (c) Kaiser Fung
  32. 32. 90% Confidence = 63% PPV (c) Kaiser Fung
  33. 33. PPV = 45-60% (c) Kaiser Fung
  34. 34. Chapter 4: Timid Testers / Magic Lassos (c) Kaiser Fung
  35. 35. 4 What’s the value of testing? Q3*Q ~ Testing U wanted 2 But Didn’t (c) Kaiser Fung
  36. 36. BEFORE AFTER xkcd (c) Kaiser Fung
  37. 37. B.T. 30% 70% 63% 37% A.T. (c) Kaiser Fung
  38. 38. 1. Why statistical significance 2. How sample sizing 3. What confidence level 4. Why test Statistical Noise Safe Landing Measure of Risk PPV Lift A/A Test Sq-root Law PPV Bayes Q4 ~ Testing U wanted 2 But Didn’t (c) Kaiser Fung
  39. 39. KAISER FUNG Strategic data advisory services, including test planning & analysis Speaking on business analytics & dataviz Training in statistical thinking (c) Kaiser Fung
  40. 40. Thank you Kaiser Fung kaiserdatamatter@gmail.com Twitter: @junkcharts LinkedIn (c) Kaiser Fung

×