SlideShare a Scribd company logo
1 of 28
Follow @ShameOnUX
Split Testing 101
     KISS
     If you can’t measure it, you can’t test it
     For clean data you can only test ONE
      element at a time
     Attribution is everything




Follow @ShameOnUX
Types of Conversions

                    •   Purchase
                    •   Contact
                    •   Phone Call
                    •   Sign-up
                    •   Subscription (on-going purchase)
                    •   Content
                    •   Share
                    •   Time on site / page
                    •   Pages per visit
                    •   Download


Follow @ShameOnUX
A Quick Note…


 When testing for specific conversions, focus
 on specific signals that drive the specific
 conversion.

               • Purchase vs. Sign up
               • Submit content vs. Get the click
               • Social Share vs. Link-bait




Follow @ShameOnUX
Split Test 1
              Goal: Increase Content Acquisition

   Conversion: Form Submittal
   Type: user Generated Review
   Motivation: Share frustration, publicize
   experience, receive sentiment, receive
   compensation



Follow @ShameOnUX
Split Test 1 Control




Follow @ShameOnUX
Split Test 1

 Variation 1: Remove Unnecessary and potentially
 distracting page elements

 Element: Main Navigation




Follow @ShameOnUX
Follow @ShameOnUX
Split Test 1, Variation 1 Test Results
                   Conversion Increase: 63%

                         Visitors          Conversions   Conversion Rate           Standard Error
       Control               11643              961          8.25%                     0.26%

       Variation             12564              1689        13.44%                     0.30%


                                                                 95% Conversion Rate Limits
                                                                      From            To

                 Significant At                                            7.83%       8.67%

                 95% confidence:                YES                     12.94%        13.95%

                 99% confidence:                YES
                                                                 99% Conversion Rate Limits
       Z-score                -13.069389                              From            To

                                                                           7.75%       8.75%

       P-value                 2.5E-39                                  12.85%        14.04%




Follow @ShameOnUX
Split Test 1

    Variation 2: Add elements of social proof

    Element: Most recent reviews from other users




Follow @ShameOnUX
Follow @ShameOnUX
Variation 2 Test Results
                     Conversion Increase: 105%

                       Visitors           Conversions   Conversion Rate            Standard Error
     Control                15263              2106         13.80%                     0.28%

     Variation              14007              3957         28.25%                     0.38%


                                                                 95% Conversion Rate Limits
                                                                      From            To

               Significant At                                             13.34%      14.26%

               95% confidence:                 YES                        27.62%      28.88%

               99% confidence:                 YES
                                                                 99% Conversion Rate Limits
     Z-score                -30.628883                                From            To

                                                                          13.25%      14.35%

     P-value                     3E-206                                   27.50%      29.00%




Follow @ShameOnUX
Split Test 2
              Goal: User Sign-Up

   Conversion: Form Submittal
   Type: Personal Information
   Submission
   Motivation: Interested in starting free
   trial



Follow @ShameOnUX
Split Test 2 Control




Follow @ShameOnUX
Split Test 2

    Variation 1: Change Call-to-Action text

    Element: Button




Follow @ShameOnUX
Follow @ShameOnUX
Split Test 2, Variation 1 Test Results
                    Conversion Increase: 25%

                     Visitors            Conversions   Conversion Rate           Standard Error

   Control                 1254                112          8.93%                     0.81%

   Variation               1304                145         11.12%                     0.87%


                                                                95% Conversion Rate Limits
                                                                     From            To

             Significant At                                              7.60%       10.26%

             95% confidence:                  YES                        9.68%       12.56%

             99% confidence:                   NO
                                                                99% Conversion Rate Limits

   Z-score                 -1.8450767                                From            To

                                                                         7.35%       10.51%

   P-value                     0.03251                                   9.41%       12.83%


Follow @ShameOnUX
Split Test 2

    Variation 2: Prominent Display of Price

    Element: Sub-title / Text




Follow @ShameOnUX
Follow @ShameOnUX
Split Test 2, Variation 1 Test Results
                     Conversion Increase: 42%

                     Visitors            Conversions   Conversion Rate            Standard Error

   Control                 1366                151         11.05%                      0.85%

   Variation               1284                201         15.65%                      1.01%


                                                                95% Conversion Rate Limits
                                                                     From             To

             Significant At                                              9.65%        12.45%

             95% confidence:                  YES                        13.98%       17.33%

             99% confidence:                  YES
                                                                99% Conversion Rate Limits

   Z-score                 -3.4791822                                From             To

                                                                         9.39%        12.72%

   P-value                     0.00025                                   13.67%       17.64%


Follow @ShameOnUX
Split Test 3
              Goal: Increase click-through rate

   Conversion: Interest-based
   Type: Click
   Motivation: Interested in learning more
   about offer




Follow @ShameOnUX
Split Test 2 Control




Follow @ShameOnUX
Split Test 3

    Scenario: Data supports that visitors who first visit the
    product detail page prior to entering the checkout are
    50% more likely to complete their purchase
    Variation 1: Remove Buy Now Option
    Element: Button




Follow @ShameOnUX
Follow @ShameOnUX
Split Test 3, Variation 1 Test Results
                    Conversion Decrease: 18%

                       Visitors          Conversions   Conversion Rate           Standard Error
     Control                10112             543           5.37%                    0.22%

     Variation              11213             511           4.56%                    0.20%


                                                               95% Conversion Rate Limits
                                                                    From            To

               Significant At                                            5.00%       5.74%

               95% confidence:                YES                        4.23%       4.88%

               99% confidence:                YES
                                                               99% Conversion Rate Limits


     Z-score                2.72335304                              From            To
                                                                         4.93%       5.81%

     P-value                 0.99677                                     4.17%       4.94%




Follow @ShameOnUX
Split Testing Key Takeaways

     • Don’t Guess. Test.
     • If at First you don’t succeed, test again. If
       at first you do succeed, test again.
     • Remove emotion from testing
     • Tests will fail
     • Tests will succeed
     • Put yourself in the shoes of your users; feel
       their pain, now design a test



Follow @ShameOnUX
Follow @ShameOnUX
Follow @ShameOnUX

Visit ShameOnUX.com

More Related Content

Recently uploaded

办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一
办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一
办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一F dds
 
3D Printing And Designing Final Report.pdf
3D Printing And Designing Final Report.pdf3D Printing And Designing Final Report.pdf
3D Printing And Designing Final Report.pdfSwaraliBorhade
 
2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degreeyuu sss
 
NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...
NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...
NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...Amil baba
 
VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130
VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130
VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130Suhani Kapoor
 
8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR
8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR
8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCRdollysharma2066
 
Call Girls Meghani Nagar 7397865700 Independent Call Girls
Call Girls Meghani Nagar 7397865700  Independent Call GirlsCall Girls Meghani Nagar 7397865700  Independent Call Girls
Call Girls Meghani Nagar 7397865700 Independent Call Girlsssuser7cb4ff
 
Call In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCR
Call In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCRCall In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCR
Call In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCRdollysharma2066
 
Architecture case study India Habitat Centre, Delhi.pdf
Architecture case study India Habitat Centre, Delhi.pdfArchitecture case study India Habitat Centre, Delhi.pdf
Architecture case study India Habitat Centre, Delhi.pdfSumit Lathwal
 
办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一
办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一
办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一Fi L
 
西北大学毕业证学位证成绩单-怎么样办伪造
西北大学毕业证学位证成绩单-怎么样办伪造西北大学毕业证学位证成绩单-怎么样办伪造
西北大学毕业证学位证成绩单-怎么样办伪造kbdhl05e
 
MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...
MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...
MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...katerynaivanenko1
 
PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024
PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024
PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024CristobalHeraud
 
Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`
Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`
Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`dajasot375
 
原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degreeyuu sss
 
办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书
办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书
办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书zdzoqco
 
Design Portfolio - 2024 - William Vickery
Design Portfolio - 2024 - William VickeryDesign Portfolio - 2024 - William Vickery
Design Portfolio - 2024 - William VickeryWilliamVickery6
 
(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一
(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一
(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一Fi sss
 
Cosumer Willingness to Pay for Sustainable Bricks
Cosumer Willingness to Pay for Sustainable BricksCosumer Willingness to Pay for Sustainable Bricks
Cosumer Willingness to Pay for Sustainable Bricksabhishekparmar618
 

Recently uploaded (20)

办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一
办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一
办理学位证(SFU证书)西蒙菲莎大学毕业证成绩单原版一比一
 
3D Printing And Designing Final Report.pdf
3D Printing And Designing Final Report.pdf3D Printing And Designing Final Report.pdf
3D Printing And Designing Final Report.pdf
 
2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
2024新版美国旧金山州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
 
Cheap Rate ➥8448380779 ▻Call Girls In Iffco Chowk Gurgaon
Cheap Rate ➥8448380779 ▻Call Girls In Iffco Chowk GurgaonCheap Rate ➥8448380779 ▻Call Girls In Iffco Chowk Gurgaon
Cheap Rate ➥8448380779 ▻Call Girls In Iffco Chowk Gurgaon
 
NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...
NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...
NO1 Famous Amil Baba In Karachi Kala Jadu In Karachi Amil baba In Karachi Add...
 
VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130
VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130
VIP Call Girls Service Bhagyanagar Hyderabad Call +91-8250192130
 
8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR
8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR
8377877756 Full Enjoy @24/7 Call Girls in Nirman Vihar Delhi NCR
 
Call Girls Meghani Nagar 7397865700 Independent Call Girls
Call Girls Meghani Nagar 7397865700  Independent Call GirlsCall Girls Meghani Nagar 7397865700  Independent Call Girls
Call Girls Meghani Nagar 7397865700 Independent Call Girls
 
Call In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCR
Call In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCRCall In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCR
Call In girls Bhikaji Cama Place 🔝 ⇛8377877756 FULL Enjoy Delhi NCR
 
Architecture case study India Habitat Centre, Delhi.pdf
Architecture case study India Habitat Centre, Delhi.pdfArchitecture case study India Habitat Centre, Delhi.pdf
Architecture case study India Habitat Centre, Delhi.pdf
 
办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一
办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一
办理学位证(NUS证书)新加坡国立大学毕业证成绩单原版一比一
 
西北大学毕业证学位证成绩单-怎么样办伪造
西北大学毕业证学位证成绩单-怎么样办伪造西北大学毕业证学位证成绩单-怎么样办伪造
西北大学毕业证学位证成绩单-怎么样办伪造
 
MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...
MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...
MT. Marseille an Archipelago. Strategies for Integrating Residential Communit...
 
PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024
PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024
PORTFOLIO DE ARQUITECTURA CRISTOBAL HERAUD 2024
 
Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`
Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`
Abu Dhabi Call Girls O58993O4O2 Call Girls in Abu Dhabi`
 
原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
原版美国亚利桑那州立大学毕业证成绩单pdf电子版制作修改#毕业文凭制作#回国入职#diploma#degree
 
办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书
办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书
办理卡尔顿大学毕业证成绩单|购买加拿大文凭证书
 
Design Portfolio - 2024 - William Vickery
Design Portfolio - 2024 - William VickeryDesign Portfolio - 2024 - William Vickery
Design Portfolio - 2024 - William Vickery
 
(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一
(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一
(办理学位证)埃迪斯科文大学毕业证成绩单原版一比一
 
Cosumer Willingness to Pay for Sustainable Bricks
Cosumer Willingness to Pay for Sustainable BricksCosumer Willingness to Pay for Sustainable Bricks
Cosumer Willingness to Pay for Sustainable Bricks
 

Featured

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by HubspotMarius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTExpeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsPixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 

Featured (20)

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 

Split Testing Using Real Data

  • 2. Split Testing 101  KISS  If you can’t measure it, you can’t test it  For clean data you can only test ONE element at a time  Attribution is everything Follow @ShameOnUX
  • 3. Types of Conversions • Purchase • Contact • Phone Call • Sign-up • Subscription (on-going purchase) • Content • Share • Time on site / page • Pages per visit • Download Follow @ShameOnUX
  • 4. A Quick Note… When testing for specific conversions, focus on specific signals that drive the specific conversion. • Purchase vs. Sign up • Submit content vs. Get the click • Social Share vs. Link-bait Follow @ShameOnUX
  • 5. Split Test 1 Goal: Increase Content Acquisition Conversion: Form Submittal Type: user Generated Review Motivation: Share frustration, publicize experience, receive sentiment, receive compensation Follow @ShameOnUX
  • 6. Split Test 1 Control Follow @ShameOnUX
  • 7. Split Test 1 Variation 1: Remove Unnecessary and potentially distracting page elements Element: Main Navigation Follow @ShameOnUX
  • 9. Split Test 1, Variation 1 Test Results Conversion Increase: 63% Visitors Conversions Conversion Rate Standard Error Control 11643 961 8.25% 0.26% Variation 12564 1689 13.44% 0.30% 95% Conversion Rate Limits From To Significant At 7.83% 8.67% 95% confidence: YES 12.94% 13.95% 99% confidence: YES 99% Conversion Rate Limits Z-score -13.069389 From To 7.75% 8.75% P-value 2.5E-39 12.85% 14.04% Follow @ShameOnUX
  • 10. Split Test 1 Variation 2: Add elements of social proof Element: Most recent reviews from other users Follow @ShameOnUX
  • 12. Variation 2 Test Results Conversion Increase: 105% Visitors Conversions Conversion Rate Standard Error Control 15263 2106 13.80% 0.28% Variation 14007 3957 28.25% 0.38% 95% Conversion Rate Limits From To Significant At 13.34% 14.26% 95% confidence: YES 27.62% 28.88% 99% confidence: YES 99% Conversion Rate Limits Z-score -30.628883 From To 13.25% 14.35% P-value 3E-206 27.50% 29.00% Follow @ShameOnUX
  • 13. Split Test 2 Goal: User Sign-Up Conversion: Form Submittal Type: Personal Information Submission Motivation: Interested in starting free trial Follow @ShameOnUX
  • 14. Split Test 2 Control Follow @ShameOnUX
  • 15. Split Test 2 Variation 1: Change Call-to-Action text Element: Button Follow @ShameOnUX
  • 17. Split Test 2, Variation 1 Test Results Conversion Increase: 25% Visitors Conversions Conversion Rate Standard Error Control 1254 112 8.93% 0.81% Variation 1304 145 11.12% 0.87% 95% Conversion Rate Limits From To Significant At 7.60% 10.26% 95% confidence: YES 9.68% 12.56% 99% confidence: NO 99% Conversion Rate Limits Z-score -1.8450767 From To 7.35% 10.51% P-value 0.03251 9.41% 12.83% Follow @ShameOnUX
  • 18. Split Test 2 Variation 2: Prominent Display of Price Element: Sub-title / Text Follow @ShameOnUX
  • 20. Split Test 2, Variation 1 Test Results Conversion Increase: 42% Visitors Conversions Conversion Rate Standard Error Control 1366 151 11.05% 0.85% Variation 1284 201 15.65% 1.01% 95% Conversion Rate Limits From To Significant At 9.65% 12.45% 95% confidence: YES 13.98% 17.33% 99% confidence: YES 99% Conversion Rate Limits Z-score -3.4791822 From To 9.39% 12.72% P-value 0.00025 13.67% 17.64% Follow @ShameOnUX
  • 21. Split Test 3 Goal: Increase click-through rate Conversion: Interest-based Type: Click Motivation: Interested in learning more about offer Follow @ShameOnUX
  • 22. Split Test 2 Control Follow @ShameOnUX
  • 23. Split Test 3 Scenario: Data supports that visitors who first visit the product detail page prior to entering the checkout are 50% more likely to complete their purchase Variation 1: Remove Buy Now Option Element: Button Follow @ShameOnUX
  • 25. Split Test 3, Variation 1 Test Results Conversion Decrease: 18% Visitors Conversions Conversion Rate Standard Error Control 10112 543 5.37% 0.22% Variation 11213 511 4.56% 0.20% 95% Conversion Rate Limits From To Significant At 5.00% 5.74% 95% confidence: YES 4.23% 4.88% 99% confidence: YES 99% Conversion Rate Limits Z-score 2.72335304 From To 4.93% 5.81% P-value 0.99677 4.17% 4.94% Follow @ShameOnUX
  • 26. Split Testing Key Takeaways • Don’t Guess. Test. • If at First you don’t succeed, test again. If at first you do succeed, test again. • Remove emotion from testing • Tests will fail • Tests will succeed • Put yourself in the shoes of your users; feel their pain, now design a test Follow @ShameOnUX