A/B-testing is awesome, the holy grail of online visitor knowledge and the base of revenue growth for many companies. Ton Wesseling, Testing Guru, frequently asked CRO speaker and CEO of Testing.Agency has 15 years of online optimization experience and will share the following insights in his talk: How & when to apply testing in your company 10 inspirational A/B-test results 20 pitfalls you will make in A/B-testing if you don't attend this talk

1. A/B-testing
H OW, WH E N , E X A M P L E S & P I T T FA L L S
L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r
To n We s s e l i n g – C E O – Te s t i n g . A g e n c y

2. Manchester
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

3. Should I go for Ci ty?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

4. Should I go for Uni ted?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

5. Manchester
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

6. A/B- test ing
A/B-testing != CRO
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

7. A/B- test ing
CRO != user knowledge
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

8. A/B- test ing
A/B-testing
==
user knowledge
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

9. A/B- test ing
HOW & WHEN
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

10. A/B- test ing
EXAMPLES
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

11. A/B- test ing
PITTFALLS
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

12. Ton Wessel ing | @ TonW
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
 More testing
 Quality Assurance
 Lower pricing

13. Onl ine Exper iments – A/B- test ing
Default
Variation
B
Variation
Winner!
A
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

14. Hyper l ink junky
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

15. Digging through log f i les
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

16. Trying to improve
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

17. Over and over again
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

18. What most star t -ups do
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

19. That is:
0 / 1 Testing
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

20. Why can this lead to fai lures?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

21. 0/1 Test ing: weekly change?
26 out of 44 weeks: over 5% change!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

22. Test ing should be l ike:
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

23. Scient i f ic approach
A / B Testing
Live Online Research
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

24. Exper iments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon – 2004!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

25. Growth l ike Amazon
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
German Ecommerce market, thank you André Morys

26. Exper iments per year
1000+
(but it did not start like that)
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

27. Growth curve
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

28. Phase 1: test solut ions
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

29. Fake adwords test ing
Do they
click
on your idea?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

30. Fake facebook test ing
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
Do they
like
your idea?

31. Fake landing page test ing
Do they
subscribe
to your idea?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

32. Make sure i t ’s not
0 / 1 Testing
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

33. Once you br idged the gap
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

34. The growth phase!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

35. Dupl icate?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

36. I t ’s t h e w e b , i t ’s scalable!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

37. You need to opt imize!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

38. Over and over again
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

39. I t takes many exper iments to win
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

40. Maximize your test capaci ty
Time
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
Conversion%

41. Speed up – more people
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

42. Growth l ike Amazon
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
German Ecommerce market, thank you André Morys

43. Real growth
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

44. I t takes heavy exper iment ing
250+ in a year
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

45. Not every exper iment is successful
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

46. I t takes a ful l stadium a week for 250+
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

47. Or roughly
1000+
conversion points
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

48. Too slow? Energy wi l l run out !
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

49. And you wi l l never get this
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

50. You need the real fast paced cul ture
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

51. How does Bi l l win?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

52. Emot ion vs Rat io
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

53. T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

54. Why is your user here?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

55. How can you
help your user ’s brain
to fullfil it’s needs?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

56. I t ’s not about A/B- test ing
Growth is about learning
how to serve brains!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

57. Test ing costs t ime
Use it to LEARN
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

58. FA C T & A C T – 7 s t e p s t o l e a r n
Find Analyze Create Test
&
Analyze Conclude Tell
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

59. N o w : f o c u s o n C TA
Create
Test
Analyze
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

60. Create = Data & Psychology
Analyze
&
Experiment
Digital Data Persuasion Psychology
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

61. Wheel -of-Persuasion.com
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

62. Persuasion on forms
+25% sales
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

63. Same technique
Signi f icant more
submi ts, but . .
Less registrat ions
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

64. A smal l sentence
+18%
sales
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

65. But also
+10% appl icat ions
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

66. Mot ivat ion
Winner
(sales)
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

67. Win your own form A/B- test
http://testing.agency/manchester
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

68. And social proof?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
Many went before you
In november 1.675
people applied for
a loan at MoneYou.
Winner
(sales)

69. Chunk your U S P ’s
+14%
sales
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

70. Wi th learning: Phase 3 becomes easier
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

71. Test ing becomes user knowledge
Conversion Op miza on Market Research
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

72. Pi t t fal ls
How NOT to test
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

73. A/B- test ing can be
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

74. D o n ’ t t rust your A/B- test analyt ics
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

75. It keeps you away f rom this fai lure
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

76. Enough t raff ic & conversions?
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

77. Low signi f icance levels
 Big chance the winner is not a real winner
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

78. Low power levels
 Big chance a real winner is not recognized
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

79. Tested percentage is a spread!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

80. Example test -setup
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

81. Average power : 40%
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

82. So:
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

83. Another er ror warning:
 You just go and dig unitil you find something
(you will always find something that seems true BUT is
not!)
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

84. OK, resul ts are in
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

85. Not signi f icant
 It’s not a winner, stop
 But, it’s does not mean it’s a loser!
 Average: 30% of your tests will have winners
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

86. More?
http://ondi.me/contentverve
http://ondi.me/conversionxl
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

87. Conclude
 Do nothing?
or
 Re-test?
or
 Implement?
or
 Scale?
and
 Continue or change test roadmap
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

88. Knowledge l ibrary: ABtestguide.com
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

89. I t ’s not about A/B- test ing
Growth is about learning
how to serve brains!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

90. So you wi l l get this curve
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

91. Growth l ike Amazon
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4
German Ecommerce market, thank you André Morys

92. A/B- test ing
HOW & WHEN
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

93. A/B- test ing
EXAMPLES
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

94. A/B- test ing
PITTFALLS
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

95. Sl ides & more
http://testing.agency/manchester
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

96. Be l ike Bi l l , and win!
T o n We s s e l i n g | @ T o nW # l e a n c o n f Ma n c h e s t e r 2 0 1 4

97. ht tp: / / testing.agency/manchester
H OW, WH E N , E X A M P L E S & P I T T FA L L S
L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r
To n We s s e l i n g – C E O – Te s t i n g . A g e n c y