Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[CXL Live 16] How to Utilize Your Test Capacity? by Ton Wesseling

4,715 views

Published on

The more tests you run, the more you learn. Every test will deliver extra insights, but winners will pay the direct costs. How many tests should you run how often? Should it be many small tests or a couple of big ones? Ton will explain this to you based on his 15+ years of A/B-testing experience.

Published in: Marketing
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/Q3f58 ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/Q3f58 ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[CXL Live 16] How to Utilize Your Test Capacity? by Ton Wesseling

  1. 1. C o n v e r s i o n X L L i v e , A u s t i n , M a r c h 3 1 s t 2 0 1 6 To n W e s s e l i n g - F o u n d e r - Te s t i n g . A g e n c y Tw i t t e r : @ To n W - To n @ Te s t i n g . A g e n c y Utilize your test capacity
  2. 2. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Amazing job
  3. 3. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive
  4. 4. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive
  5. 5. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Johan Cruijff 1947 - 2016
  6. 6. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Lead the way
  7. 7. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive ”You will only be able to see it once you get it.”
  8. 8. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Online Dialogue
  9. 9. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Testing.Agency
  10. 10. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Test capacity maximum number of a/b-tests per year that you could run on your users
  11. 11. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive What do companies do within CRO?
  12. 12. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Are companies improving conversion?
  13. 13. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Is it because of the ease of testing?
  14. 14. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Most effective to improve conversions
  15. 15. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Regret amount of money you are losing per year by not using your full test capacity
  16. 16. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive the ROAR model
  17. 17. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Optimization phases - ROAR Time span Conversionspermonth Risk
  18. 18. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Optimization phases - ROAR Time span Conversionspermonth Risk + Optimization
  19. 19. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Optimization phases - ROAR Time span Conversionspermonth Risk + Optimization + Automation
  20. 20. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Optimization phases - ROAR Time span Conversionspermonth Risk + Optimization + Automation Re-think
  21. 21. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Optimization phases - ROAR Time span Conversionspermonth Risk + Optimization + Automation Re-think 10.000 conversions per month 1.000 conversions per month
  22. 22. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive How much impact do you need? http://ondi.me/size
  23. 23. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Optimization phases - ROAR Time span Conversionspermonth Risk + Optimization + Automation Re-think 10.000 conversions per month 1.000 conversions per month 15% impact needed 5% impact needed Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%
  24. 24. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Yes, I’ve got those 1000+ conversions Time span Conversionspermonth Risk + Optimization + Automation Re-think 1.000 conversions per month 10.000 conversions per month
  25. 25. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive With 1000+ conversions a month Your test capacity is max. 20+ tests a year with a statistical power of 80% on predicted uplifts of 15%
  26. 26. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests with measurable impact On average 1 out of 3 Simple truth: you’re just not always able to create a winner with enough impact
  27. 27. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive So in this case, you’ve got impact Every 7 or 8 weeks
  28. 28. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Slower? Energy will run out! And your continous optimization program will die
  29. 29. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive What to test? Business cases! Delivery –test with in stock. 4 weeks or 2 days?
  30. 30. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive What to test? Business cases! Service charge, do it or not? Include in price or not?
  31. 31. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive What to test? Big design changes
  32. 32. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive What to test? Big design changes
  33. 33. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive What to test? Big design changes
  34. 34. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive ROAR – Optimization – moving up Time span Conversionspermonth Risk + Optimization + Automation Re-think 1.000 conversions per month 10.000 conversions per month
  35. 35. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Source: Psychology
  36. 36. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Use that persuasion knowledge Wheel-of-Persuasion.com Resource & courses
  37. 37. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  38. 38. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  39. 39. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  40. 40. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  41. 41. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  42. 42. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  43. 43. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Tests that can be done
  44. 44. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive + Optimization + Automation Back to the ROAR model Time span Conversionspermonth Risk Re-think 1.000 conversions per month 10.000 conversions per month
  45. 45. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive The Data Driven DNA border Time span Conversionspermonth Risk + Optimization + Automation Re-think 1.000 conversions per month 10.000 conversions per month
  46. 46. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Test capacity users IT releases Campaigns Behavior
  47. 47. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive IT release testing
  48. 48. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive IT release testing Testing to see if the release is not lowering the conversion rate
  49. 49. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Campaign testing
  50. 50. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Campaigns should be bandits
  51. 51. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Maximize your exploitation
  52. 52. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Behavior learning
  53. 53. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Behavior learning
  54. 54. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive You need many tests to fill this
  55. 55. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive That’s why you need the traffic Time span Conversionspermonth Risk + Optimization + Automation Re-think 1.000 conversions per month 10.000 conversions per month
  56. 56. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Test capacity users, aiming for: IT releases Campaigns Behavior 25% or less 25% or less 50% or more
  57. 57. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Calculating behavioral capacity
  58. 58. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Behavioral knowledge test capacity •  Sum your # tests per element •  Take the behavior share •  divide by 2 (no major overlap!) •  Consider your resources •  Get resources or test with higher Power
  59. 59. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Winner ratio •  Know your % winners bottom limit •  Better testing •  Bolder testing •  Less testing (more data in 1 test)
  60. 60. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive T-test evaluation problem: Winner
  61. 61. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive No Winner?
  62. 62. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Bayesian solution: Winner?
  63. 63. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Winner! – with ROI, which C-level gets
  64. 64. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Bayesian tools & pdf R Calculator: ondi.me/bayes White Paper PDF Free download: http://ondi.me/change
  65. 65. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive ü  Use your full test capacity by great planning ü  Seperate test bandwidth for IT, campaigns and for learnings ü  Think like a manager – work with business cases Be a pro, like Johan was
  66. 66. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive An the whole company will love you
  67. 67. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive One more thing
  68. 68. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Celebrate failures
  69. 69. Ton Wesseling | @ TonW Utilize your test capacity #CXLlive Only celebrating winners will lead to: People that are scared to create non-winning variations – not taking the risk they should!
  70. 70. C o n v e r s i o n X L L i v e , A u s t i n , M a r c h 3 1 s t 2 0 1 6 To n W e s s e l i n g - F o u n d e r - Te s t i n g . A g e n c y Tw i t t e r : @ To n W - To n @ Te s t i n g . A g e n c y Utilize your test capacity

×