Smart marketers know that A/B testing helps take the guesswork out of website optimizations. Often times though, A/B testing doesn't give you a true view of how your site or web page is performing. In this presentation, WP Engine Founder & CTO Jason Cohen covers why tools that show “statistical significance” are often wrong, and how you can correct it, how to use insights from Google Analytics to drive A/B tests and what elements of your marketing campaigns should be A/B tested.

1. SIX PUZZLES
How A/B tests lie to us, and
how to drive genuine improvement
Jason Cohen, founder & CTO
http://wpengine.com

2. The first puzzle

3. 654 out of 24,483 visits
85% chance-to-beat

4. What do stats
really tell us?

6. 654 out of 24,483 visits
85% chance-to-beat

7. Finding the Cheaters
105 athletes.
100 are clean.
5 are juicing.
Test is 95% accurate.

8. Finding the Cheaters
IN REALITY
Clean (100) Juicing (5)
TEST
RESULT
(95%
accuracy)
Clean
Juicing 5
095
5

9. Insight:
Rare events
are indistinguishable from
random noise

10. 758 out of 24,483 visits
85% chance-to-beat

11. Easy significance testing
with hamsters
Trust in the Hamster.
bit.ly/abhamster

12. Easy significance testing
N = A+B = total # of conversions.
D = (A–B)/2 = half the difference.
Significant if D2
> N.
bit.ly/abhamster
N = 758 + 685 = 1443
D = ( 758 — 685 ) / 2 = 37
1369 > 1443?
No! Not significant.

13. Final Result: As predicted
Wasted another month

14. 30
40
50
60
70
12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11
OtherInbox Splash Page

16. Lesson #1
Beware the Rare

17. Lesson #2
Trust the Hamster

18. When
statistically-significant
rodentia still aren’t
enough...

19. 41 Shades of Blue

20. 41 Shades of Blue
Does this...
...get more clicks than this?

21. 2 Shades of Blue
with 1 winner
at a 95% confidence level,
chance of a false-positive: 5%

22. 3 Shades of Blue
with 1 winner
at a 95% confidence level,
chance of a false-positive: 9%

23. 10 Shades of Blue
with 1 winner
at a 95% confidence level,
chance of a false-positive: 40%

24. 41 Shades of Blue
with 1 winner
at a 95% confidence level,
chance of a false-positive: 88%

25. The False-Positive Guarantee

26. Oops, you’re doing this.

27. Insight:
Real experiments
test theories.

28. Lesson #3
Test theories,
not headlines

29. Poking around
your analytics

30. Bible Code!

31. Bible Code!

32. Edwards v. Aguillard

33. War and Peace

35. GA reveals ideas, not facts

36. Lesson #4
Exploration
yields new theories,
not new truth.

37. A Unifying Concept

38. Headwinds
to improvement
significance → huge N → N/A
high failure rate → many tests → false-positives
rare events → worse on both!

39. Summary:
Subtly is the Enemy

40. Insight:
You don’t want
“Subtle” anyway!

41. Lesson #5
Seek Huge Effects

42. Can you A/B test
your way to big results?

43. 1,116
250
4,100
1,151
Individuals:
67% off
Group:
3% off

44. Wisdom of the Crowd
“I’m as clueless as you,
but in the equal and opposite direction.”

45. What’s the funniest joke?

46. And the winner is…

47. Jokes, Pizza,
Design, Features, Voice,
Values, Culture

48. Lesson #6
Crowds find objective truth
but kill creativity and greatness

49. Think for yourself.
Test theories.
Seek big effects.
Jason Cohen, founder & CTO
http://wpengine.com

50. Register now for our next webinar!
“Your Workflow, Your Way,
with WP Engine”
Register now:
http://wpeng.in/ywyw

51. Q & A
Jason Cohen, founder & CTO
http://wpengine.com