Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Is Not CRO But It’s Still Darn Important

3,133 views

Published on

This presentation is from Affiliate Summit East 2015 (August 2-4, 2015 in New York, NY). Session description: Testing has hit the mainstream, but when something is popular - bad habits form. In this session you’ll learn when you should test, what to test, and how to construct a winning optimization campaign.

Published in: Business
  • Be the first to comment

  • Be the first to like this

Testing Is Not CRO But It’s Still Darn Important

  1. 1. TESTING IS NOT CRO BUT IT’S STILL DARN IMPORTANT
  2. 2. FIRST LET’S COVER SOME A/B TESTING BASICS
  3. 3. Types Of Tests Synchronous Asynchronous Sequential Testing A/B/…/N Testing Multivariate Testing Test Types
  4. 4. Synchronous Testing July 1st - July 15th July 16th - July 30th
  5. 5. A/b/…/n Split Testing $$$ $$$$$$ $$$ $$$
  6. 6. A/b/…/n Testing You’ll Get Page Conversion Data (Not Element By Element) Better For Low Traffic Sites
  7. 7. A/b/…/n Testing Some Rules: Isolate Individual Elements If You Don’t Isolate Elements, DO NOT attribute the lift or loss to any single element
  8. 8. Multivariate Testing You’ll Get Element By Element Data Don’t Run If Less Than 100k Pageviews Per Month
  9. 9. NOW LET’S GIVE A/B TESTING SOME LOVE
  10. 10. I BUILT MY CAREER AROUND SPLIT TESTING
  11. 11. IT’S THE ONLY WAY TO GET DEFINITIVE ANSWERS IN A CLEAN ENVIRONMENT
  12. 12. TESTING HELPS GUIDE DECISIONS AND IMPROVE THE USER EXPERIENCE WHILE BOOSTING PROFITS
  13. 13. TESTING GOT THE CONVERSATION STARTED!
  14. 14. A/B TESTS ARE SEXY
  15. 15. 89.97% LIFT IN SALESSource: vwo.com
  16. 16. 12.6% LIFT IN SUBMISSION S Source: Unbounce.com
  17. 17. 18.70% DROP IN SUBMISSION SSource: ContentVerve.com
  18. 18. 19.47% LIFT IN SUBMISSION S Source: ContentVerve.com
  19. 19. AWESOME STUFF RIGHT?
  20. 20. THERE’S ONE BIG PROBLEM
  21. 21. EVERYONE WANTS TO TEST EVEN WHEN THEY CAN’T
  22. 22. SIMPLY PUT: NOT EVERYONE CAN OR SHOULD BE SPLIT TESTING
  23. 23. NOT BEING ABLE TO TEST DOESN’T MEAN…
  24. 24. …NOT BEING ABLE TO OPTIMIZE
  25. 25. THERE ARE PLENTY OF WAYS TO BOOST CONVERSIONS OUTSIDE OF SPLIT TESTING
  26. 26. TODAY IN A NUTSHELL 1. When You Can Test 2. Should You Actually Test 3. What To Do When You Can’t/Shouldn’t Test
  27. 27. Part Can I Actually Run A Test
  28. 28. Testing Eligibility • ~100 Conversion Per Variation • Run for at least a full week • Not run longer than 8 weeks • Test can reach 95% confidence in this time frame
  29. 29. Testing Eligibility • ~100 Conversion Per Variation • Run for at least a full week • Not run longer than 8 weeks • Test can reach 95% confidence in this time frame
  30. 30. Conversion Per Day Chart Minimum Number Of Conversions Per Day Test Period (Days) 2 Variants 3 Variants 4 Variants 5 Variants 7 29 43 58 72 14 15 22 29 36 21 10 15 20 24 28 8 11 15 18 35 6 9 12 15 42 5 8 10 12 49 5 7 9 11 56 4 6 8 9
  31. 31. Problem: Not Enough Conversions Solution: Run Longer Secondarily: If Outside Of 8 Week Testing Window - Don’t Test It
  32. 32. VARIABLE NUMBER ISN’T THE ONLY THING THAT DICTATES TIME!
  33. 33. THE PERCENT LIFT IS A HUGE TIMING FACTOR
  34. 34. In order for this test to get statistical results, it would require a 10% lift
  35. 35. AND SO IS THE CONFIDENCE RATEBut we’ll talk more about that later
  36. 36. Does Your Page Get Enough Daily Conversions? YES No Can’t Test - Sorry! Is There A Big Enough Lift Potential?* YES No This Is A Testable Page Can’t Test - Sorry! *With An Appropriate Confidence Rate/Index
  37. 37. Part Should I Actually Run That Test
  38. 38. THE CONVERSION HIERARCHY
  39. 39. Functional Does the page work and do what it’s supposed to?
  40. 40. Accessible Can people of all skill levels on any device use the site? What are the barriers on the site?
  41. 41. Usable Can people understand your site, i.e., is it user friendly?
  42. 42. Intuitive Does the site have a logical flow? Does the offer match the source?
  43. 43. Persuasive Is your site convincing people to buy or take an action?
  44. 44. “For every test you run, there are an infinite number of other tests that you could have run” - Justindamus 120 AD
  45. 45. Part How To Optimize If I Can’t/Shouldn’t Test
  46. 46. You Have A Few Options • Evaluate Your Offer • On-Site Retargeting • Changes Based On Qualitative Research
  47. 47. EVALUATE YOUR OFFER
  48. 48. It’s not a traffic problem, it’s an offer problem
  49. 49. It’s not a testing problem, it’s an offer problem
  50. 50. You might have to head back to the drawing board completely. Does your product/service provide real value?
  51. 51. You also might be going after the wrong traffic source. Identify if your ideal customer is ‘hanging out’ on the traffic medium
  52. 52. TRY ON SITE RETARGETIN G
  53. 53. What is On Site Retargeting? Evaluating current interactions and providing a relevant offer based on these interactions.
  54. 54. Time On Site Number Of Pageviews Type Of Content Consumed Exit Intent Referring Source
  55. 55. Just Under Half Of All Offer Pageviews Sent By Optimonk
  56. 56. Typical Campaign Conversion: 69% With Exit pop: 80.7%
  57. 57. Using this method we got 2,689 more leads in just 14 days!
  58. 58. “Respondents targeting more than 20% of their visitors with personalized content achieved a 5% conversion rate, which was 2x the average conversion rate of 2.5% in their study” [Adobe 2014 Survey]
  59. 59. PAGE CHANGES BASED ON OPTIMIZATION RESEARCH
  60. 60. Here are a few things to try • Click/Scroll Tracking • Usability Testing • User Surveys • Session Recordings
  61. 61. CLICK/SCROLL TRACKING
  62. 62. People who actually scroll to the bottom click Not enough people are scrolling that far down ~15% of all clicks on this page are on the nav bar
  63. 63. Try a shorter variation to get people to scroll to the CTA. Add multiple CTA buttons at areas where scrolling is highest. Hide the nav bar on this page
  64. 64. USABILITY TESTING
  65. 65. USER SURVEYS
  66. 66. • Was it was easy to find what you were looking for? • Were the words/vocab used to define categories/sub-categories clear to you? • Do you have any suggestions to improve our website navigation? • Does our website look credible to you? • Is our pricing clear? • Is there anything else you’d like to know before signing up with us? • Will you shop with us again? Why/why not? • Do you think the form has any confusing/unnecessary input fields? QUESTION IDEAS
  67. 67. SESSION RECORDINGS
  68. 68. Shows individual recordings of a user interacting with your site. Unlike heatmaps, this isn’t aggregated data
  69. 69. Really blow their mind with a real life user recording with audio
  70. 70. TRICKS TO TEST WHEN YOU DON’T HAVE THE TRAFFIC
  71. 71. USE MICRO CONVERSIONS AS INDICATORS
  72. 72. Test Legnth Minimum Conversions Per Variation Per Day 7 Days 29 14 Days 15 21 Days 10 28 Days 8 35 Days 6 42 Days 5 49 Days 5 56 Days 4
  73. 73. LOWER THE CONFIDENCE THRESHOLD
  74. 74. Lift: 20% Confidence Rate: 95% _________________ Required Sample Size: 7,682
  75. 75. Lift: 20% Confidence Rate: 90% _________________ Required Sample Size: 6,052 21% Decrease
  76. 76. REMEMBER - SPLIT TESTING IS NOT OPTIMIZATION…
  77. 77. IT’S AN OPTIMIZATION TOOL
  78. 78. Split Testing Personalization User Surveys Funnel Analysis Web Analytics Personas Neurotesting Advanced Segments Eye Tracking Scroll Tracking CRO
  79. 79. SPLIT TESTING CAN ACTUALLY HURT A COMPANY
  80. 80. SO PRACTICE SAFE TESTING WITH A PAGE THAT MATTERS
  81. 81. • Email: justin.rondeau@digitalmarketer.com • Twitter: @jtrondeau • Website: digitalmarketer.com Justin Rondeau Optimization Manager, Digital Marketer Let’s Talk More!

×