The basics you should take a look at before starting A/B testing.

1. Best Practices and
Lessons Learned from
100,000 A/B Tests
Martijn Geerlings
Thursday, November 28, 13

2. Is de (klant) reis...
belangrijker dan de bestemming?
Thursday, November 28, 13

3. 2009!
Thursday, November 28, 13
2010!
2011!
2012!
2013!

4. Lessons Learned
Thursday, November 28, 13

5. Lesson #1:
Define success .
Thursday, November 28, 13

6. Thursday, November 28, 13

7. Lesson #1:
Define
Thursday, November 28, 13
success
.

8. Lesson #1:
Define quantifiable success metrics.
Thursday, November 28, 13

9. Thursday, November 28, 13

10. visitors
per day
Thursday, November 28, 13
Visit

11. visitors
per day
Thursday, November 28, 13
Visit
sign up
rate
Email

12. visitors
per day
Thursday, November 28, 13
Visit
sign up
rate
Email
$ per recipient
$

13. visitors
per day
Thursday, November 28, 13
Visit
sign up
rate
Email
$ per recipient
$

14. Media
Button
Thursday, November 28, 13

15. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Thursday, November 28, 13

16. Button: “Sign Up”
Thursday, November 28, 13

17. Button: “Learn More”
Thursday, November 28, 13

18. Button: “Join Us Now”
Thursday, November 28, 13

19. Button: “Sign Up Now”
Thursday, November 28, 13

20. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Thursday, November 28, 13

21. Media: “Get Involved”
Thursday, November 28, 13

22. Media: “Family”
Thursday, November 28, 13

23. Media: “Change”
Thursday, November 28, 13

24. Media: “Barack’s Video”
Thursday, November 28, 13

25. Media: “Springfield Video”
Thursday, November 28, 13

26. Media: “Sam’s Video”
Thursday, November 28, 13

27. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Thursday, November 28, 13

28. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Thursday, November 28, 13

29. Splash page experiment
Variations:
Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Thursday, November 28, 13

30. Splash page experiment
Thursday, November 28, 13

31. Splash page experiment
Thursday, November 28, 13

32. Splash page experiment
Thursday, November 28, 13

33. Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:
7,120,000
+40.6%
+2,880,000
+288,000 +$57,000,000
New:
10,000,000
1,000,000 $200,000,000
Thursday, November 28, 13
712,000
$143,000,000

34. Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:
7,120,000
+40.6%
+2,880,000
+288,000 +$57,000,000
New:
10,000,000
1,000,000 $200,000,000
Thursday, November 28, 13
712,000
$143,000,000

35. Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:
7,120,000
+40.6%
+2,880,000
+288,000 +$57,000,000
New:
10,000,000
1,000,000 $200,000,000
Thursday, November 28, 13
712,000
$143,000,000

36. Lesson #1:
Define quantifiable success metrics.
Thursday, November 28, 13

37. Lesson #2:
Less is more. Reduce choices.
Thursday, November 28, 13

38. vs.
Thursday, November 28, 13

39. vs.
+8% engagement
Thursday, November 28, 13

40. +43.8%
Purchases
Thursday, November 28, 13

41. vs.
Thursday, November 28, 13

42. vs.
+14.8% conversion rate
Thursday, November 28, 13

43. Lesson #2:
Less is more. Reduce choices.
Thursday, November 28, 13

44. Lesson #3:
Words matter. Focus on your call to action.
Thursday, November 28, 13

45. Donation button experiment
Variations
Thursday, November 28, 13

46. Donation button experiment
Previously
Donated
0.0%
0.0%
+2.3%
+27.8%
+16.3%
-27.8%
N/A
N/A
+15.2%
-24.6%
+11.9%
+8.5%
Thursday, November 28, 13
Signed Up,
Never Donated
0.0%
Variations
Never
Signed Up
+2.9%
+18.4%

47. vs.
Thursday, November 28, 13

48. vs.
Thursday, November 28, 13

49. vs.
+14.6% clicks
Thursday, November 28, 13

50. Lesson #3:
Words matter. Focus on your call to action.
Thursday, November 28, 13

51. Lesson #4:
Fail fast.
Thursday, November 28, 13

52. vs.
Thursday, November 28, 13

53. vs.
-10% conversions
Thursday, November 28, 13

54. vs.
Thursday, November 28, 13

55. vs.
-10% conversions
Thursday, November 28, 13

56. Lesson #4:
Fail fast.
Thursday, November 28, 13

57. Lesson #5:
Seek the Global Maximum.
Thursday, November 28, 13

58. Thursday, November 28, 13

59. Thursday, November 28, 13

60. Thursday, November 28, 13

61. Thursday, November 28, 13

62. Thursday, November 28, 13

63. +0.5% Retention
+1.4% Engagement
Thursday, November 28, 13

64. Lesson #5:
Seek the Global Maximum.
Thursday, November 28, 13

65. Lessons Learned
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call to action.
4. Fail fast.
5. Seek the Global Maximum.
Thursday, November 28, 13

66. Thursday, November 28, 13

67. One more thing...
Thursday, November 28, 13

68. 6000+ happy testers
Start Today!
Thursday, November 28, 13