Best Practices and Lessons Learned
From Over 100,000 A/B and MVT Tests
Best Practices and Lessons Learned
From Over 313,649 A/B and MVT Tests
2009

2010

2011

2012

2013
Lesson #1:
Define Success
Lesson #1:
Define Success
Lesson #1:
Define Quantifiable Success Metrics
Visitors Per
Day
Visit
Visitors Per
Day

Sign-Up
Rate
Visit

Email
Visitors Per
Day

Sign-Up
Rate
Visit

$ Per
Recipient
Email

$$
Visitors Per
Day

Sign-Up
Rate
Visit

$ Per
Recipient
Email

$$
Media

Button
Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now

Media:
1. “Get Involved”...
Button: “Sign Up”
Button: “Learn More”
Button: “Join Us Now”
Button: “Sign Up Now”
Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now

Media:
1. “Get Involved”...
Media: “Get Involved”
Media: “Family”
Media: “Change”
QuickTime™ and a
H.264 decompressor
are needed to see this picture.

Media: “Barack’s Video”
QuickTime™ and a
H.264 decompressor
are needed to see this picture.

Media: “Springfield Video”
QuickTime™ and a
H.264 decompressor
are needed to see this picture.

Media: “Sam’s Video”
Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now

Media:
1. “Get Involved”...
Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now

Media:
1. “Get Involved”...
Splash Page Experiment
Variations
Button:
1. Sign Up
2. Learn More
3. Join Us Now
4. Sign Up Now

Media:
1. “Get Involved”...
Splash Page Experiment
Splash Page Experiment
Splash Page Experiment
Splash Page Experiment
Email
Subscriptions

Volunteers

Amount Raised

Original

7,120,000

712,000

$143,000,000

+40.6%
...
Splash Page Experiment
Email
Subscriptions

Volunteers

Amount Raised

Original

7,120,000

712,000

$143,000,000

+40.6%
...
Splash Page Experiment
Email
Subscriptions

Volunteers

Amount Raised

Original

7,120,000

712,000

$143,000,000

+40.6%
...
Lesson #1:
Define Quantifiable Success Metrics
Lesson #2:
Less Is More. Reduce Choices.
VS.
VS.

+8%
Sign-Ups
+49%
Conversion Rate
(240 A/B Tests)
+15% Conversion Rate
+10% $ Per Visitor
Lesson #2:
Less Is More. Reduce Choices.
Lesson #3:
Word Matter,
Focus On Your Call To Action.
Donation Button Experiment
Donation Button Experiment
Never
Signed-Up

Signed-Up
No Donate

Previously
Donated

0.0%

0.0%

0.0%

2.3%

27.8%

16.3%
...
+14.6%
Clicks
+15.7%
$$ Per Visitor
Lesson #3:
Word Matter,
Focus On Your Call To Action.
Lesson #4:
Fail Fast.
VS.
-92.3 % CTR
VS.

-10 % AOV
Lesson #4:
Fail Fast.
Lesson #5:
Nothing Is Sacred.
Challenge Assumptions.
VS.
+600%
Engagement
Lesson #5:
Nothing Is Sacred.
Challenge Assumptions.
The Future of Optimization

One to Many
What’s the best experience
for the average user?
The Future of Optimization

One to Many

One to Few

What’s the best experience
for the average user?

What’s the best exp...
The Future of Optimization

One to Many
What’s the best experience
for the average user?

One to Few

One to One

What’s t...
Live Demo!
@optimizely
Optimizely - Conversion Optimization Masterclass
Optimizely - Conversion Optimization Masterclass
Optimizely - Conversion Optimization Masterclass
Upcoming SlideShare
Loading in …5
×

Optimizely - Conversion Optimization Masterclass

921 views

Published on

Published in: Marketing, Business, Technology

Optimizely - Conversion Optimization Masterclass

  1. 1. Best Practices and Lessons Learned From Over 100,000 A/B and MVT Tests
  2. 2. Best Practices and Lessons Learned From Over 313,649 A/B and MVT Tests
  3. 3. 2009 2010 2011 2012 2013
  4. 4. Lesson #1: Define Success
  5. 5. Lesson #1: Define Success
  6. 6. Lesson #1: Define Quantifiable Success Metrics
  7. 7. Visitors Per Day Visit
  8. 8. Visitors Per Day Sign-Up Rate Visit Email
  9. 9. Visitors Per Day Sign-Up Rate Visit $ Per Recipient Email $$
  10. 10. Visitors Per Day Sign-Up Rate Visit $ Per Recipient Email $$
  11. 11. Media Button
  12. 12. Splash Page Experiment Variations Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. “Get Involved” Image 2. “Family” Image 3. “Change” Image 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  13. 13. Button: “Sign Up”
  14. 14. Button: “Learn More”
  15. 15. Button: “Join Us Now”
  16. 16. Button: “Sign Up Now”
  17. 17. Splash Page Experiment Variations Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. “Get Involved” Image 2. “Family” Image 3. “Change” Image 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  18. 18. Media: “Get Involved”
  19. 19. Media: “Family”
  20. 20. Media: “Change”
  21. 21. QuickTime™ and a H.264 decompressor are needed to see this picture. Media: “Barack’s Video”
  22. 22. QuickTime™ and a H.264 decompressor are needed to see this picture. Media: “Springfield Video”
  23. 23. QuickTime™ and a H.264 decompressor are needed to see this picture. Media: “Sam’s Video”
  24. 24. Splash Page Experiment Variations Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. “Get Involved” Image 2. “Family” Image 3. “Change” Image 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  25. 25. Splash Page Experiment Variations Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. “Get Involved” Image 2. “Family” Image 3. “Change” Image 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  26. 26. Splash Page Experiment Variations Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. “Get Involved” Image 2. “Family” Image 3. “Change” Image 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  27. 27. Splash Page Experiment
  28. 28. Splash Page Experiment
  29. 29. Splash Page Experiment
  30. 30. Splash Page Experiment Email Subscriptions Volunteers Amount Raised Original 7,120,000 712,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New 10,000,000 1,000,000 $200,000,000
  31. 31. Splash Page Experiment Email Subscriptions Volunteers Amount Raised Original 7,120,000 712,000 $143,000,000 +40.6% +2,880,000 +280,000 +$57,000,000 New 10,000,000 1,000,000 $200,000,000
  32. 32. Splash Page Experiment Email Subscriptions Volunteers Amount Raised Original 7,120,000 712,000 $143,000,000 +40.6% +2,880,000 +280,000 +$57,000,000 New 10,000,000 1,000,000 $200,000,000
  33. 33. Lesson #1: Define Quantifiable Success Metrics
  34. 34. Lesson #2: Less Is More. Reduce Choices.
  35. 35. VS.
  36. 36. VS. +8% Sign-Ups
  37. 37. +49% Conversion Rate (240 A/B Tests)
  38. 38. +15% Conversion Rate +10% $ Per Visitor
  39. 39. Lesson #2: Less Is More. Reduce Choices.
  40. 40. Lesson #3: Word Matter, Focus On Your Call To Action.
  41. 41. Donation Button Experiment
  42. 42. Donation Button Experiment Never Signed-Up Signed-Up No Donate Previously Donated 0.0% 0.0% 0.0% 2.3% 27.8% 16.3% -27.8 N/A N/A 15.2% -24.6% 11.9% 8.5% 2.9% 18.4%
  43. 43. +14.6% Clicks
  44. 44. +15.7% $$ Per Visitor
  45. 45. Lesson #3: Word Matter, Focus On Your Call To Action.
  46. 46. Lesson #4: Fail Fast.
  47. 47. VS. -92.3 % CTR
  48. 48. VS. -10 % AOV
  49. 49. Lesson #4: Fail Fast.
  50. 50. Lesson #5: Nothing Is Sacred. Challenge Assumptions.
  51. 51. VS. +600% Engagement
  52. 52. Lesson #5: Nothing Is Sacred. Challenge Assumptions.
  53. 53. The Future of Optimization One to Many What’s the best experience for the average user?
  54. 54. The Future of Optimization One to Many One to Few What’s the best experience for the average user? What’s the best experience for segments of users?
  55. 55. The Future of Optimization One to Many What’s the best experience for the average user? One to Few One to One What’s the best experience for What’s the best experience segments of users? for each user?
  56. 56. Live Demo!
  57. 57. @optimizely

×