Follow @ShameOnUX
Split Testing 101     KISS     If you can’t measure it, you can’t test it     For clean data you can only test ONE     ...
Types of Conversions                    •   Purchase                    •   Contact                    •   Phone Call     ...
A Quick Note… When testing for specific conversions, focus on specific signals that drive the specific conversion.        ...
Split Test 1              Goal: Increase Content Acquisition   Conversion: Form Submittal   Type: user Generated Review   ...
Split Test 1 ControlFollow @ShameOnUX
Split Test 1 Variation 1: Remove Unnecessary and potentially distracting page elements Element: Main NavigationFollow @Sha...
Follow @ShameOnUX
Split Test 1, Variation 1 Test Results                   Conversion Increase: 63%                         Visitors        ...
Split Test 1    Variation 2: Add elements of social proof    Element: Most recent reviews from other usersFollow @ShameOnUX
Follow @ShameOnUX
Variation 2 Test Results                     Conversion Increase: 105%                       Visitors           Conversion...
Split Test 2              Goal: User Sign-Up   Conversion: Form Submittal   Type: Personal Information   Submission   Moti...
Split Test 2 ControlFollow @ShameOnUX
Split Test 2    Variation 1: Change Call-to-Action text    Element: ButtonFollow @ShameOnUX
Follow @ShameOnUX
Split Test 2, Variation 1 Test Results                    Conversion Increase: 25%                     Visitors           ...
Split Test 2    Variation 2: Prominent Display of Price    Element: Sub-title / TextFollow @ShameOnUX
Follow @ShameOnUX
Split Test 2, Variation 1 Test Results                     Conversion Increase: 42%                     Visitors          ...
Split Test 3              Goal: Increase click-through rate   Conversion: Interest-based   Type: Click   Motivation: Inter...
Split Test 2 ControlFollow @ShameOnUX
Split Test 3    Scenario: Data supports that visitors who first visit the    product detail page prior to entering the che...
Follow @ShameOnUX
Split Test 3, Variation 1 Test Results                    Conversion Decrease: 18%                       Visitors         ...
Split Testing Key Takeaways     • Don’t Guess. Test.     • If at First you don’t succeed, test again. If       at first yo...
Follow @ShameOnUX
Follow @ShameOnUXVisit ShameOnUX.com
Upcoming SlideShare
Loading in...5
×

Split Testing Using Real Data

2,904

Published on

This is my presentation from June's Shame On UX, where we focused on split testing using real conversion data and confidence intervals to evaluate test results.

Published in: Design, Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,904
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
10
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Split Testing Using Real Data

  1. 1. Follow @ShameOnUX
  2. 2. Split Testing 101  KISS  If you can’t measure it, you can’t test it  For clean data you can only test ONE element at a time  Attribution is everythingFollow @ShameOnUX
  3. 3. Types of Conversions • Purchase • Contact • Phone Call • Sign-up • Subscription (on-going purchase) • Content • Share • Time on site / page • Pages per visit • DownloadFollow @ShameOnUX
  4. 4. A Quick Note… When testing for specific conversions, focus on specific signals that drive the specific conversion. • Purchase vs. Sign up • Submit content vs. Get the click • Social Share vs. Link-baitFollow @ShameOnUX
  5. 5. Split Test 1 Goal: Increase Content Acquisition Conversion: Form Submittal Type: user Generated Review Motivation: Share frustration, publicize experience, receive sentiment, receive compensationFollow @ShameOnUX
  6. 6. Split Test 1 ControlFollow @ShameOnUX
  7. 7. Split Test 1 Variation 1: Remove Unnecessary and potentially distracting page elements Element: Main NavigationFollow @ShameOnUX
  8. 8. Follow @ShameOnUX
  9. 9. Split Test 1, Variation 1 Test Results Conversion Increase: 63% Visitors Conversions Conversion Rate Standard Error Control 11643 961 8.25% 0.26% Variation 12564 1689 13.44% 0.30% 95% Conversion Rate Limits From To Significant At 7.83% 8.67% 95% confidence: YES 12.94% 13.95% 99% confidence: YES 99% Conversion Rate Limits Z-score -13.069389 From To 7.75% 8.75% P-value 2.5E-39 12.85% 14.04%Follow @ShameOnUX
  10. 10. Split Test 1 Variation 2: Add elements of social proof Element: Most recent reviews from other usersFollow @ShameOnUX
  11. 11. Follow @ShameOnUX
  12. 12. Variation 2 Test Results Conversion Increase: 105% Visitors Conversions Conversion Rate Standard Error Control 15263 2106 13.80% 0.28% Variation 14007 3957 28.25% 0.38% 95% Conversion Rate Limits From To Significant At 13.34% 14.26% 95% confidence: YES 27.62% 28.88% 99% confidence: YES 99% Conversion Rate Limits Z-score -30.628883 From To 13.25% 14.35% P-value 3E-206 27.50% 29.00%Follow @ShameOnUX
  13. 13. Split Test 2 Goal: User Sign-Up Conversion: Form Submittal Type: Personal Information Submission Motivation: Interested in starting free trialFollow @ShameOnUX
  14. 14. Split Test 2 ControlFollow @ShameOnUX
  15. 15. Split Test 2 Variation 1: Change Call-to-Action text Element: ButtonFollow @ShameOnUX
  16. 16. Follow @ShameOnUX
  17. 17. Split Test 2, Variation 1 Test Results Conversion Increase: 25% Visitors Conversions Conversion Rate Standard Error Control 1254 112 8.93% 0.81% Variation 1304 145 11.12% 0.87% 95% Conversion Rate Limits From To Significant At 7.60% 10.26% 95% confidence: YES 9.68% 12.56% 99% confidence: NO 99% Conversion Rate Limits Z-score -1.8450767 From To 7.35% 10.51% P-value 0.03251 9.41% 12.83%Follow @ShameOnUX
  18. 18. Split Test 2 Variation 2: Prominent Display of Price Element: Sub-title / TextFollow @ShameOnUX
  19. 19. Follow @ShameOnUX
  20. 20. Split Test 2, Variation 1 Test Results Conversion Increase: 42% Visitors Conversions Conversion Rate Standard Error Control 1366 151 11.05% 0.85% Variation 1284 201 15.65% 1.01% 95% Conversion Rate Limits From To Significant At 9.65% 12.45% 95% confidence: YES 13.98% 17.33% 99% confidence: YES 99% Conversion Rate Limits Z-score -3.4791822 From To 9.39% 12.72% P-value 0.00025 13.67% 17.64%Follow @ShameOnUX
  21. 21. Split Test 3 Goal: Increase click-through rate Conversion: Interest-based Type: Click Motivation: Interested in learning more about offerFollow @ShameOnUX
  22. 22. Split Test 2 ControlFollow @ShameOnUX
  23. 23. Split Test 3 Scenario: Data supports that visitors who first visit the product detail page prior to entering the checkout are 50% more likely to complete their purchase Variation 1: Remove Buy Now Option Element: ButtonFollow @ShameOnUX
  24. 24. Follow @ShameOnUX
  25. 25. Split Test 3, Variation 1 Test Results Conversion Decrease: 18% Visitors Conversions Conversion Rate Standard Error Control 10112 543 5.37% 0.22% Variation 11213 511 4.56% 0.20% 95% Conversion Rate Limits From To Significant At 5.00% 5.74% 95% confidence: YES 4.23% 4.88% 99% confidence: YES 99% Conversion Rate Limits Z-score 2.72335304 From To 4.93% 5.81% P-value 0.99677 4.17% 4.94%Follow @ShameOnUX
  26. 26. Split Testing Key Takeaways • Don’t Guess. Test. • If at First you don’t succeed, test again. If at first you do succeed, test again. • Remove emotion from testing • Tests will fail • Tests will succeed • Put yourself in the shoes of your users; feel their pain, now design a testFollow @ShameOnUX
  27. 27. Follow @ShameOnUX
  28. 28. Follow @ShameOnUXVisit ShameOnUX.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×