Split Testing Using Real Data

Follow @ShameOnUX

Split Testing 101  KISS  If you can’t measure it, you can’t test it  For clean data you can only test ONE element at a time  Attribution is everythingFollow @ShameOnUX

Types of Conversions • Purchase • Contact • Phone Call • Sign-up • Subscription (on-going purchase) • Content • Share • Time on site / page • Pages per visit • DownloadFollow @ShameOnUX

A Quick Note… When testing for specific conversions, focus on specific signals that drive the specific conversion. • Purchase vs. Sign up • Submit content vs. Get the click • Social Share vs. Link-baitFollow @ShameOnUX

Split Test 1 Goal: Increase Content Acquisition Conversion: Form Submittal Type: user Generated Review Motivation: Share frustration, publicize experience, receive sentiment, receive compensationFollow @ShameOnUX

Split Test 1 ControlFollow @ShameOnUX

Split Test 1 Variation 1: Remove Unnecessary and potentially distracting page elements Element: Main NavigationFollow @ShameOnUX

Follow @ShameOnUX

Split Test 1, Variation 1 Test Results Conversion Increase: 63% Visitors Conversions Conversion Rate Standard Error Control 11643 961 8.25% 0.26% Variation 12564 1689 13.44% 0.30% 95% Conversion Rate Limits From To Significant At 7.83% 8.67% 95% confidence: YES 12.94% 13.95% 99% confidence: YES 99% Conversion Rate Limits Z-score -13.069389 From To 7.75% 8.75% P-value 2.5E-39 12.85% 14.04%Follow @ShameOnUX

Split Test 1 Variation 2: Add elements of social proof Element: Most recent reviews from other usersFollow @ShameOnUX

Follow @ShameOnUX

Variation 2 Test Results Conversion Increase: 105% Visitors Conversions Conversion Rate Standard Error Control 15263 2106 13.80% 0.28% Variation 14007 3957 28.25% 0.38% 95% Conversion Rate Limits From To Significant At 13.34% 14.26% 95% confidence: YES 27.62% 28.88% 99% confidence: YES 99% Conversion Rate Limits Z-score -30.628883 From To 13.25% 14.35% P-value 3E-206 27.50% 29.00%Follow @ShameOnUX

Split Test 2 Goal: User Sign-Up Conversion: Form Submittal Type: Personal Information Submission Motivation: Interested in starting free trialFollow @ShameOnUX

Split Test 2 Variation 1: Change Call-to-Action text Element: ButtonFollow @ShameOnUX

Follow @ShameOnUX

Split Test 2, Variation 1 Test Results Conversion Increase: 25% Visitors Conversions Conversion Rate Standard Error Control 1254 112 8.93% 0.81% Variation 1304 145 11.12% 0.87% 95% Conversion Rate Limits From To Significant At 7.60% 10.26% 95% confidence: YES 9.68% 12.56% 99% confidence: NO 99% Conversion Rate Limits Z-score -1.8450767 From To 7.35% 10.51% P-value 0.03251 9.41% 12.83%Follow @ShameOnUX

Split Test 2 Variation 2: Prominent Display of Price Element: Sub-title / TextFollow @ShameOnUX

Follow @ShameOnUX

Split Test 2, Variation 1 Test Results Conversion Increase: 42% Visitors Conversions Conversion Rate Standard Error Control 1366 151 11.05% 0.85% Variation 1284 201 15.65% 1.01% 95% Conversion Rate Limits From To Significant At 9.65% 12.45% 95% confidence: YES 13.98% 17.33% 99% confidence: YES 99% Conversion Rate Limits Z-score -3.4791822 From To 9.39% 12.72% P-value 0.00025 13.67% 17.64%Follow @ShameOnUX

Split Test 3 Goal: Increase click-through rate Conversion: Interest-based Type: Click Motivation: Interested in learning more about offerFollow @ShameOnUX

Split Test 3 Scenario: Data supports that visitors who first visit the product detail page prior to entering the checkout are 50% more likely to complete their purchase Variation 1: Remove Buy Now Option Element: ButtonFollow @ShameOnUX

Follow @ShameOnUX

Split Test 3, Variation 1 Test Results Conversion Decrease: 18% Visitors Conversions Conversion Rate Standard Error Control 10112 543 5.37% 0.22% Variation 11213 511 4.56% 0.20% 95% Conversion Rate Limits From To Significant At 5.00% 5.74% 95% confidence: YES 4.23% 4.88% 99% confidence: YES 99% Conversion Rate Limits Z-score 2.72335304 From To 4.93% 5.81% P-value 0.99677 4.17% 4.94%Follow @ShameOnUX

Split Testing Key Takeaways • Don’t Guess. Test. • If at First you don’t succeed, test again. If at first you do succeed, test again. • Remove emotion from testing • Tests will fail • Tests will succeed • Put yourself in the shoes of your users; feel their pain, now design a testFollow @ShameOnUX

Follow @ShameOnUX

Follow @ShameOnUXVisit ShameOnUX.com

http://sistemasumadcup.blogspot.mx	15
http://sistemasumadcup.blogspot.com	3
http://127.0.0.1	2
http://www.onlydoo.com	1
http://us-w1.rockmelt.com	1

Split Testing Using Real Data

by Nick Eubanks on Jun 30, 2012

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 22

Statistics

Split Testing Using Real Data Presentation Transcript