The document summarizes lessons learned from over 313,649 A/B and multivariate tests: 1) Define quantifiable success metrics to evaluate experiments objectively. 2) Less choices are better - reducing options increases conversion rates. 3) Word choice matters - focus on clear calls to action. 4) Fail experiments fast if they are not working to save time and resources. 5) Challenge assumptions and test anything to find unexpected improvements.

1. Best Practices and Lessons Learned
From Over 100,000 A/B and MVT Tests

2. Best Practices and Lessons Learned
From Over 313,649 A/B and MVT Tests

3. 2009
2010
2011
2012
2013

5. Lesson #1:
Define Success

7. Lesson #1:
Define Success

8. Lesson #1:
Define Quantifiable Success Metrics

9. Visitors Per
Day
Visit

10. Visitors Per
Day
Sign-Up
Rate
Visit
Email

11. Visitors Per
Day
Sign-Up
Rate
Visit
$ Per
Recipient
Email
$$

12. Visitors Per
Day
Sign-Up
Rate
Visit
$ Per
Recipient
Email
$$

13. Media
Button

14. Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. “Get Involved” Image
2. “Family” Image
3. “Change” Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

15. Button: “Sign Up”

16. Button: “Learn More”

17. Button: “Join Us Now”

18. Button: “Sign Up Now”

19. Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. “Get Involved” Image
2. “Family” Image
3. “Change” Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

20. Media: “Get Involved”

21. Media: “Family”

22. Media: “Change”

23. QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Media: “Barack’s Video”

24. QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Media: “Springfield Video”

25. QuickTime™ and a
H.264 decompressor
are needed to see this picture.
Media: “Sam’s Video”

26. Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. “Get Involved” Image
2. “Family” Image
3. “Change” Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

27. Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. “Get Involved” Image
2. “Family” Image
3. “Change” Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

28. Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media:
1. “Get Involved” Image
2. “Family” Image
3. “Change” Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video

29. Splash Page Experiment

30. Splash Page Experiment

31. Splash Page Experiment

32. Splash Page Experiment
Email
Subscriptions
Volunteers
Amount Raised
Original
7,120,000
712,000
$143,000,000
+40.6%
+2,880,000
+288,000
+$57,000,000
New
10,000,000
1,000,000
$200,000,000

33. Splash Page Experiment
Email
Subscriptions
Volunteers
Amount Raised
Original
7,120,000
712,000
$143,000,000
+40.6%
+2,880,000
+280,000
+$57,000,000
New
10,000,000
1,000,000
$200,000,000

34. Splash Page Experiment
Email
Subscriptions
Volunteers
Amount Raised
Original
7,120,000
712,000
$143,000,000
+40.6%
+2,880,000
+280,000
+$57,000,000
New
10,000,000
1,000,000
$200,000,000

35. Lesson #1:
Define Quantifiable Success Metrics

36. Lesson #2:
Less Is More. Reduce Choices.

37. VS.

38. VS.
+8%
Sign-Ups

39. +49%
Conversion Rate
(240 A/B Tests)

40. +15% Conversion Rate
+10% $ Per Visitor

41. Lesson #2:
Less Is More. Reduce Choices.

42. Lesson #3:
Word Matter,
Focus On Your Call To Action.

43. Donation Button Experiment

44. Donation Button Experiment
Never
Signed-Up
Signed-Up
No Donate
Previously
Donated
0.0%
0.0%
0.0%
2.3%
27.8%
16.3%
-27.8
N/A
N/A
15.2%
-24.6%
11.9%
8.5%
2.9%
18.4%

45. +14.6%
Clicks

46. +15.7%
$$ Per Visitor

47. Lesson #3:
Word Matter,
Focus On Your Call To Action.

48. Lesson #4:
Fail Fast.

49. VS.
-92.3 % CTR

50. VS.
-10 % AOV

51. Lesson #4:
Fail Fast.

52. Lesson #5:
Nothing Is Sacred.
Challenge Assumptions.

54. VS.
+600%
Engagement

55. Lesson #5:
Nothing Is Sacred.
Challenge Assumptions.

56. The Future of Optimization
One to Many
What’s the best experience
for the average user?

57. The Future of Optimization
One to Many
One to Few
What’s the best experience
for the average user?
What’s the best experience for
segments of users?

58. The Future of Optimization
One to Many
What’s the best experience
for the average user?
One to Few
One to One
What’s the best experience for What’s the best experience
segments of users?
for each user?

59. Live Demo!

60. @optimizely