Successfully reported this slideshow.

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

5

Share

1 of 63
1 of 63

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

5

Share

Download to read offline

How confident are you of understanding how (and why) customers react to your campaigns?

In his ClickZ presentation, Taylor Kennedy, Senior Manager of Training, shared lessons that will optimize ecommerce campaigns to help you identify principles that improve your own testing strategy, while getting a better understanding of your customer.

How confident are you of understanding how (and why) customers react to your campaigns?

In his ClickZ presentation, Taylor Kennedy, Senior Manager of Training, shared lessons that will optimize ecommerce campaigns to help you identify principles that improve your own testing strategy, while getting a better understanding of your customer.

More Related Content

More from MarketingExperiments

Related Books

Free with a 14 day trial from Scribd

See all

ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)

  1. 1. November 3–6, 2014 #CZLCHI | @ClickZLive The Global Conference Series Designed by Digital Marketers, for Digital Marketers 3 Common Mistakes in Online Testing (And How to Avoid Them) Taylor Kennedy MECLABS Senior Manager, Optimization & Strategy
  2. 2. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive 30% 16% Do you understand your results? 60% 12% 43% 8% 62% 6%
  3. 3. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive 30% 16% 60% 12% 43% 8% 62% 6%
  4. 4. November 3–6, 2014 | #CZLCHI | @ClickZLive What does it mean?
  5. 5. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive You’re (probably) testing wrong if you: • Are stuck optimizing a part of your funnel that lacks significant impact on the bottom line • Have been blindly following “best practices” • Are not further establishing customer theory with each test
  6. 6. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Our testing process
  7. 7. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Our testing process
  8. 8. November 3–6, 2014 | #CZLCHI | @ClickZLive Case Study Optimizing the wrong part of the funnel
  9. 9. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 1: Control
  10. 10. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 1: Treatment
  11. 11. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 1: Results Results were flat. Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 14.3% NA --- Treatment 13.5% -4.99% 44% ?
  12. 12. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 2: Control
  13. 13. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 2: Treatment
  14. 14. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Cart Test 2: Results Results were flat, even trending downward. Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 13.7% NA --- Treatment 11.1% -19.26% 88% ?
  15. 15. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Why aren’t tests getting results? 5.7% 41% All visitors Add to cart Payment info Convert 40%
  16. 16. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Through proper planning you: Identifythe highest impact area of your site to test. Findpotential validity threats linked to normal traffic that may impact results. Discover visitor behavior that helps you create long-term test plans.
  17. 17. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Product Page 1: Control
  18. 18. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Product Page 1: Treatment
  19. 19. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Product Page Test 1: Results • “Add to Cart” increased by over 15% • Conversion increased by 40% Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 1.07% NA --- Treatment 1.50% 40.05% 98% !
  20. 20. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive What motivated the cart test? A. Needed improvements were obvious B. Our client or boss told us to C. Testing close to the conversion event has the biggest ability to increase revenue D. The data told us to
  21. 21. November 3–6, 2014 | #CZLCHI | @ClickZLive You’ve been following “best practices” Case study examples
  22. 22. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive How are successful and unsuccessful ecommerce companies approaching testing? MarketingSherpa asked marketers:
  23. 23. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  24. 24. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  25. 25. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  26. 26. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  27. 27. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive
  28. 28. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Your peers are following best practices too.
  29. 29. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive How often do you hear the following? “Keep your call-to-action above the fold” “Video converts” “Use a compelling image” “Mobile-friendly is critical” “Include privacy and security statements to establish trust”
  30. 30. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Control
  31. 31. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Treatment 1
  32. 32. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Treatment 2
  33. 33. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Treatment 3
  34. 34. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Security Seal Test: Results Shopping Cart ClickthroughRate Rel. Difference Level of Confidence Control 66.14% NA --- Treatment 1 66.95% 1.2% 50% Treatment 2 65.00% -1.7% 57% Treatment 3 64.60% -2.3% 54% Results were flat. ?
  35. 35. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Mobile Test: Control
  36. 36. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Mobile Test: Treatment
  37. 37. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Mobile Redesign Test: Results Results were significantly negative. Shopping Cart Conversion Rate Rel. Difference Level of Confidence Control 0.5% NA --- Treatment 0.3% -40.75% 96% !
  38. 38. November 3–6, 2014 | #CZLCHI | @ClickZLive Redefine best practice: • There are no shortcuts to customer wisdom. • Just because a test worked for someone else, doesn’t mean it will work for you. • Rigorous, methodology-driven, hypothesis testing ensures you are generating real customer insights.
  39. 39. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v + 2(i-f) -2a
  40. 40. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C= 4m + 3v + 2(i-f) -2a Probability of conversion
  41. 41. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m+ 3v + 2(i-f) -2a Motivation of user
  42. 42. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v+ 2(i-f) -2a Clarity of value proposition
  43. 43. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v + 2(i-f)-2a Incentive -Friction
  44. 44. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Framework © C = 4m + 3v + 2(i-f) -2a Anxiety
  45. 45. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increase in clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  46. 46. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increase in clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  47. 47. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increase in clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  48. 48. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A bad hypothesis: “By reducing friction on the page associated with product selection, we will see an increasein clickthrough.” • Unclear link to a particular problem • Not related to specific elements on a particular page or process • Not linked to information or data about the page
  49. 49. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: CHANGING __________________ INTO__________________ WILL_______________________ What your analysis indicates is the problem What change you think will solve the problem The effect it will have on your KPI
  50. 50. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: “Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page”
  51. 51. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: “Changing the presentation of products from a single column format into a dual column format willincrease the number of product options visible and boost clickthrough to the product details page”
  52. 52. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis: “Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page.”
  53. 53. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive A good hypothesis is: • A proposed statement • Made on the basis of limited evidence related to a specific problem • Can be proved or disproved • Used as a starting point for further investigation
  54. 54. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Motivation Test: Control
  55. 55. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Motivation Test: Control
  56. 56. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Motivation Test: Control Hypothesis: Changing the right-column form intoa call-to- action requiring no personal information will reduce friction and boost clickthrough to the subsequent page.
  57. 57. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Removed form completely Treatment 1:
  58. 58. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Removed form, added starting price Treatment 2:
  59. 59. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Kept form, added starting price Treatment 3:
  60. 60. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Homepage Motivation Test: Treatments Shopping Cart Rel. CTR Level of Confidence Control -- --- Treatment 1 202% 99% Treatment2 149% 99% Treatment 3 -20% 99%
  61. 61. @OptimizeDude November 3–6, 2014 | #CZLCHI | @ClickZLive Remember: • Proper planning allows you to run tests with clear outcomes predetermined in advance • Formulate test sequences that generate big-picture customer analysis !
  62. 62. November 3–6, 2014 | #CZLCHI | @ClickZLive Key takeaways Follow a “testing process.” Don’t just “test.” 1. Always perform at least a basic site data analysis (funnel analysis) before beginning testing. 2. Analyze each page through the mind of your prospect to identify areas of opportunity for testing. 3. Formulate hypotheses that enable you to learn from each and every test. That will lead you to a more complete picture of your ideal prospect.
  63. 63. November 3–6, 2014 | #CZLCHI | @ClickZLive http://bit.ly/OptimizeTK Presentation Link

Editor's Notes

  • “Preparing online tests the right way” – alternate name for the presentation

    I like “Preparing…” better as a title – KC
  • Have you ever run a test where you don’t understand the results?
    The biggest enemy to your online testing efforts is the inability to explain your own results
    How often as an online tester are you confused about why a test showed a particular result?
    Without being able to properly explain your test results, you will never get the buy-in you need to unlock the true power of testing
  • Have you ever run a test where you don’t understand the results?
    The biggest enemy to your online testing efforts is the inability to explain your own results
    How often as an online tester are you confused about why a test showed a particular result?
    Without being able to properly explain your test results, you will never get the buy-in you need to unlock the true power of testing
  • If you’ve had a result you didn’t understand more than a handful of times, then you’re testing wrong
    While not a big Jeff Foxworthy fan myself, I can’t help think of this slide as a “you might be a redneck if … sort of slide” because I’m going to outline some common mistakes that all marketers have likely made in their experiences running online tests.
    I’ve personally made these mistakes, and I want to ensure that I teach you something that will help you to avoid them.
  • While this may seem like common sense, you would be amazed by how many partners that I have worked with that do not follow a repeatable process
    By focusing on proper discovery and planning, you can ensure that you will understand your test results the vast majority of the time by properly accounting for variables under your control.
  • While this may seem like common sense, you would be amazed by how many partners that I have worked with that do not follow a repeatable process
    By focusing on proper discovery and planning you can ensure that you will understand your test results the vast majority of the time by properly accounting for variables under your control.
  • Total price is extremely unclear, order summary is at the bottom of the page
  • Added clarity
  • While this may seem like common sense, you would be amazed by how many partners that I have worked with that do not follow a repeatable process
    By focusing on proper discovery and planning you can ensure that you will understand your test results the vast majority of the time by properly accounting for variables under your control.
  • We only got to do a few of the things we wanted in our first test, so now were going to do a variable cluster test and change multiple elements to add clarity
  • Brought the cart summary above the fold
    Clarified and streamlined the shopping cart experience
    Added dynamic information about shipping options
  • Flat again
  • It became obvious when we finally got our data analysis back…motivation to convert was so high in the cart, that the changes we made did not produce any significant impact.
    So we had to move up in the funnel…
  • Link to blog post or other resource with more information
  • Once we saw the picture that the data painted, we knew where our efforts would have the most impact. We moved up in the funnel to the product page.
    Long copy, with large blocks of information below the fold
  • Prioritized “customer service” and “product quality” content that we hypothesized would affect the prospects buying decision
  • Flat again
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • Displayed limited options for “default” smartphones
  • Displayed limited options for “default” smartphones
  • Displayed limited options for “default” smartphones
  • Displayed limited options for “default” smartphones
  • Displayed limited options for “default” smartphones
  • Displayed limited options for “default” smartphones
  • Could put common statements on this slide or show a template broken out
  • Expanded smartphone options to include a longer page with additional popular products
  • Expanded smartphone options to include a longer page with additional popular products
  • Expanded smartphone options to include a longer page with additional popular products
  • Expanded smartphone options to include a longer page with additional popular products
  • Flat again
  • Displays numerous product options and categories to select from
  • Displays numerous product options and categories to select from
  • Flat again
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • We didn’t answer D, but whatever situation you are in, you should be looking to answer D
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • Single product ecommerce flow that required visitors to input their delivery information before full price disclosure
    Observed a high bounce rate on the landing page, with relatively high completion rates for the rest of the funnel
    What is the number 1 thing visitors in the ecommerce space are traditionally looking for? – Price!
  • Single product ecommerce flow that required visitors to input their delivery information before full price disclosure
    Observed a high bounce rate on the landing page, with relatively high completion rates for the rest of the funnel
    What is the number 1 thing visitors in the ecommerce space are traditionally looking for? – Price!
  • If we open the gates, we’ll drive a ton more people to the product details page
  • The results were exactly what was expected, and justified the customer theory that visitors weren’t actually interested in the product, but were interested instead in the pricing.
  • http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis.html
  • Follow a testing process, don’t just test
  • ×