Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

937 views

Published on

How confident are you of understanding how (and why) customers react to your campaigns?

In his ClickZ presentation, Taylor Kennedy, Senior Manager of Training, shared lessons that will optimize ecommerce campaigns to help you identify principles that improve your own testing strategy, while getting a better understanding of your customer.

Published in: Marketing
  • Be the first to comment

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

  1. 1. November 3–6, 2014 #CZLCHI | @ClickZLive The Global Conference Series Designed by Digital Marketers, for Digital Marketers 3 Common Mistakes in Online Testing (And How to Avoid Them) Taylor Kennedy MECLABS Senior Manager, Optimization & Strategy
  2. 2. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive 30% 16% Do you understand your results? 60% 12% 43% 8% 62% 6%
  3. 3. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive 30% 16% 60% 12% 43% 8% 62% 6%
  4. 4. November 3–6, 2014 | #CZLCHI | @ClickZLive What does it mean?
  5. 5. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive You’re (probably) testing wrong if you: • Are stuck optimizing a part of your funnel that lacks significant impact on the bottom line • Have been blindly following “best practices” • Are not further establishing customer theory with each test
  6. 6. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Our testing process
  7. 7. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Our testing process
  8. 8. November 3–6, 2014 | #CZLCHI | @ClickZLive Case Study Optimizing the wrong part of the funnel
  9. 9. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 1: Control
  10. 10. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 1: Treatment
  11. 11. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 1: Results Results were flat. Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 14.3% NA --- Treatment 13.5% -4.99% 44% ?
  12. 12. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 2: Control
  13. 13. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 2: Treatment
  14. 14. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 2: Results Results were flat, even trending downward. Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 13.7% NA --- Treatment 11.1% -19.26% 88% ?
  15. 15. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Why aren’t tests getting results? 5.7% 41% All visitors Add to cart Payment info Convert 40%
  16. 16. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Through proper planning you: Identifythe highest impact area of your site to test. Findpotential validity threats linked to normal traffic that may impact results. Discover visitor behavior that helps you create long-term test plans.
  17. 17. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Product Page 1: Control
  18. 18. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Product Page 1: Treatment
  19. 19. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Product Page Test 1: Results • “Add to Cart” increased by over 15% • Conversion increased by 40% Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 1.07% NA --- Treatment 1.50% 40.05% 98% !
  20. 20. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive What motivated the cart test? A. Needed improvements were obvious B. Our client or boss told us to C. Testing close to the conversion event has the biggest ability to increase revenue D. The data told us to
  21. 21. November 3–6, 2014 | #CZLCHI | @ClickZLive You’ve been following “best practices” Case study examples
  22. 22. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive How are successful and unsuccessful ecommerce companies approaching testing? MarketingSherpa asked marketers:
  23. 23. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  24. 24. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  25. 25. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  26. 26. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  27. 27. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  28. 28. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Your peers are following best practices too.
  29. 29. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive How often do you hear the following? “Keep your call-to-action above the fold” “Video converts” “Use a compelling image” “Mobile-friendly is critical” “Include privacy and security statements to establish trust”
  30. 30. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Control
  31. 31. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Treatment 1
  32. 32. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Treatment 2
  33. 33. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Treatment 3
  34. 34. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Results Shopping Cart ClickthroughRate Rel. Difference Level of Confidence Control 66.14% NA --- Treatment 1 66.95% 1.2% 50% Treatment 2 65.00% -1.7% 57% Treatment 3 64.60% -2.3% 54% Results were flat. ?
  35. 35. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Mobile Test: Control
  36. 36. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Mobile Test: Treatment
  37. 37. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Mobile Redesign Test: Results Results were significantly negative. Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 0.5% NA --- Treatment 0.3% -40.75% 96% !
  38. 38. November 3–6, 2014 | #CZLCHI | @ClickZLive Redefine best practice: • There are no shortcuts to customer wisdom. • Just because a test worked for someone else, doesn’t mean it will work for you. • Rigorous, methodology-driven, hypothesis testing ensures you are generating real customer insights.
  39. 39. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v + 2(i-f) -2a
  40. 40. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C= 4m + 3v + 2(i-f) -2a Probability of conversion
  41. 41. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m+ 3v + 2(i-f) -2a Motivation of user
  42. 42. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v+ 2(i-f) -2a Clarity of value proposition
  43. 43. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v + 2(i-f)-2a Incentive -Friction
  44. 44. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v + 2(i-f) -2a Anxiety
  45. 45. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increase in clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  46. 46. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increase in clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  47. 47. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increase in clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  48. 48. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increasein clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  49. 49. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: CHANGING __________________ INTO__________________ WILL_______________________ What your analysis indicates is the problem What change you think will solve the problem The effect it will have on your KPI
  50. 50. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: “Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page”
  51. 51. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: “Changing the presentation of products from a single column format into a dual column format willincrease the number of product options visible and boost clickthrough to the product details page”
  52. 52. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: “Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page.”
  53. 53. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis is: • A proposed statement • Made on the basis of limited evidence related to a specific problem • Can be proved or disproved • Used as a starting point for further investigation
  54. 54. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Motivation Test: Control
  55. 55. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Motivation Test: Control
  56. 56. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Motivation Test: Control Hypothesis: Changing the right-column form intoa call-to- action requiring no personal information will reduce friction and boost clickthrough to the subsequent page.
  57. 57. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Removed form completely Treatment 1:
  58. 58. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Removed form, added starting price Treatment 2:
  59. 59. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Kept form, added starting price Treatment 3:
  60. 60. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Homepage Motivation Test: Treatments Shopping Cart Rel. CTR Level of Confidence Control -- --- Treatment 1 202% 99% Treatment2 149% 99% Treatment 3 -20% 99%
  61. 61. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Remember: • Proper planning allows you to run tests with clear outcomes predetermined in advance • Formulate test sequences that generate big-picture customer analysis !
  62. 62. November 3–6, 2014 | #CZLCHI | @ClickZLive Key takeaways Follow a “testing process.” Don’t just “test.” 1. Always perform at least a basic site data analysis (funnel analysis) before beginning testing. 2. Analyze each page through the mind of your prospect to identify areas of opportunity for testing. 3. Formulate hypotheses that enable you to learn from each and every test. That will lead you to a more complete picture of your ideal prospect.
  63. 63. November 3–6, 2014 | #CZLCHI | @ClickZLive http://bit.ly/OptimizeTK Presentation Link

×