Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

#Conversion2015 Amsterdam keynote Ton Wesseling

3,421 views

Published on

April 14th - 09:45 - 12 hours earlier Ton Wesseling was asked to replace Craig Sullivan as morning keynote of Emerce Conversion 2015. Based on the ROAR model he created for CROday 2015 and the analytics tips on A/B-testing slidedeck he created voor DDTT on A/B-testing, he presented these slides for a 350 people audience in Amsterdam.

Any questions?
Email: ton@onlinedialogue.com
Twitter: @tonw

Published in: Business
  • Be the first to comment

#Conversion2015 Amsterdam keynote Ton Wesseling

  1. 1. ONLINE DIALOGU E G u i d i n g y o u t o d i g i t a l m a t u r i t y D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s # C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5 To n W e s s e l i n g A /B-testen
  2. 2. Email: ton@wesseling. net! Linkedin: t on wesseling! Tw it ter: @tonw! ! Ton Wesseling!
  3. 3. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Webanalisten.nl
  4. 4. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Online Dialogue
  5. 5. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Testing.agency
  6. 6. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Emerce magazine: conversion column
  7. 7. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Conversion Conferences worldwide
  8. 8. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Conversion speakers
  9. 9. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Where is Craig Sullivan?
  10. 10. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Replacing Craig…
  11. 11. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 At least he quotes me
  12. 12. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 So..
  13. 13. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 More experiments “If you double the number of experiments you do per year you’re going to double your inventiveness” Jeff Bezos, CEO Amazon – 2004!
  14. 14. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Like Amazon, Like Zalando GermanEcommercemarket,thankyouAndréMorys
  15. 15. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Experiments per year 1000+
  16. 16. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
  17. 17. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Why not? Compare to the past?
  18. 18. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Why can this lead to failures?
  19. 19. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 A big weekly change! 26 out of 44 weeks: over 5% change!
  20. 20. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Methods used to improve conversion
  21. 21. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Methods companies plan to use
  22. 22. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 It’s just HOT
  23. 23. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Lack of resources
  24. 24. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Test more
  25. 25. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Importance of a structured approach
  26. 26. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 More experiments
  27. 27. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 The negative effect trend Time span! Averageeffectofoptimzation! atcompaniesapplyingoptimization! Moneyspendon! Optimizationworldwide!
  28. 28. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 It’s not a solution It’s a method
  29. 29. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 It’s a way of working It’s DNA
  30. 30. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 So it’s a method: When How What
  31. 31. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Growth curve – how it was teached
  32. 32. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Optimization phases - ROAR Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  33. 33. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Phase 1: bridge the gap – Risk
  34. 34. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Fake adwords testing Do they click on your idea?
  35. 35. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Fake facebook testing Do they like your idea?
  36. 36. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Fake landing page testing Do they subscribe to your idea?
  37. 37. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Phase 1: bridge the gap – Risk
  38. 38. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Get out of the office
  39. 39. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Risk
  40. 40. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 ROAR - Optimization Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  41. 41. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Warning: low significance levels ü Big chance the winner is not a real winner
  42. 42. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Warning: low power levels ü Big chance a real winner is not recognized
  43. 43. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Example test set-up ü 90% significance level ü 100 conversions per variation ü Average conversion rate: 2% ü 20 out of 100 ideas is a winner… ü …with an average uplift of 10%
  44. 44. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Average: 75% significance & 40% power
  45. 45. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 So: ü 100 tests will give 10 false positives (90% significance) ü 40% of the 20 winners are recognized(40% power) ü 8 real winners & 10 false positives ü NOT 18 improvements of 10%+
  46. 46. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 You need big numbers per variation Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  47. 47. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 With 1000 conversions + a month 20+ tests a year
  48. 48. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tests with impact, on average 1 out of 3 Simply truth: you’re just not always able to create a winner with enough impact!
  49. 49. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 So in this case, you’ve impact Every 6 weeks
  50. 50. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Slower? Energy will run out!
  51. 51. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 What to test? Business cases! Delivery –test with in stock. 4 weeks or 2 days?!
  52. 52. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 What to test? Business cases! Service charge, do it or not? Include in price or not?!
  53. 53. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 What to test? Big design changes
  54. 54. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Warning: choose what to analyze up front ü You just go and dig unitil you find something (you will always find something that seems true BUT is not!)
  55. 55. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Don’t stop the test once it’s significant!
  56. 56. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Don’t stop the test once it’s significant!
  57. 57. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 You just let it run until it’s significant
  58. 58. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 You just let it run until it’s significant Your test has a fixed lenght! It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!
  59. 59. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 ABTestGuide.com/calc
  60. 60. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 ROAR – Optimization – moving up Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  61. 61. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Finally you can test details… Is Multi Variatie Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!
  62. 62. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 You should grow to Customer Buying Reasons
  63. 63. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Reasons behind the customer journey
  64. 64. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Why is your user here?
  65. 65. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 How can you help your user’s brain to fullfil it’s needs?
  66. 66. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Data & Psychology Digital Data Persuasion Psychology Analyze & Experiment
  67. 67. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Testing takes time Use it to LEARN
  68. 68. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Lack of resources
  69. 69. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Create a test schedule
  70. 70. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 FA C T & A C T – 7 s t e p s t o l e a r n & Tell Conclude Analyze Test Create Analyze Find
  71. 71. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Source: Psychology
  72. 72. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Source: Ask
  73. 73. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Source: Listen
  74. 74. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Many sources out there to find & analyze
  75. 75. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Test which dialogue has impact
  76. 76. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Analytics
  77. 77. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Heatmaps
  78. 78. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Which locations are critical
  79. 79. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 FA C T & A C T – 7 s t e p s t o l e a r n & Tell Conclude Analyze Test Create Analyze Find
  80. 80. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Now you’re testing On your way to 4 tests a week Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!
  81. 81. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Your winners are still spare? Less than 25%? Please higher your significance level to 95% or more if it drops below this 25 number!!
  82. 82. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: Your test takes too long ü Cookie deletion (you will loose 10% in two weeks) They will re-enter the the and pollute your samples!
  83. 83. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: Don’t confuse cross device users ü  Only test 1 device at a time!
  84. 84. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: Maybe you’re testing broken stuff ü  Always check browser compatibility! ü  Dont’ break dynamic stuff!
  85. 85. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: You forgot to leave out some visitors ü Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)
  86. 86. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: You’re not slowing the control down ü  Add the same code to the control
  87. 87. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: You’re not testing on fresh people ü Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)
  88. 88. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Tip: You forgot to give them time to buy ü Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)
  89. 89. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 ROAR – Automation Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  90. 90. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 You’re already thinking of more teams And still have test capacity *test capacity – the number of tests you can start per week based on your audience & conversions!
  91. 91. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Algorithms
  92. 92. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 You tested which locations are critical
  93. 93. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Automation
  94. 94. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Now you’ve moved to Algorithm testing Which internal and external influencers do have an impact on which segment of users?!
  95. 95. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 ROAR – Automation Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  96. 96. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 CRO becomes user knowledge
  97. 97. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 More impact on discover
  98. 98. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 A way of life
  99. 99. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Customer knowledge center Making money from conversion improvements Creating more and more customer insights Which can be monetized on even a larger scale
  100. 100. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Now, that’s HOT
  101. 101. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 So you keep on groing GermanEcommercemarket,thankyouAndréMorys
  102. 102. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 Done testing? No! TESTING NEVER STOPS Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!
  103. 103. onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015 ROAR – when to start with what Time span! Conversionspermonth! Risk! + Optimization! + Automation! Re-think! 1.000 conversions! per month! 10.000 conversions per month!
  104. 104. ONLINE DIALOGU E G u i d i n g y o u t o d i g i t a l m a t u r i t y D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s # C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5 To n W e s s e l i n g A /B-testen
  105. 105. Email: ton@wesseling. net! Linkedin: t on wesseling! Tw it ter: @tonw! ! Ton Wesseling!

×