The Evolution of A/B Testing - NMHC Optech 2016

155 views

Published on

Many marketers are eager to start A/B testing but aren't aware of dangerous misconceptions and prioritize tactics over strategy. This presentation explores the evolution of A/B testing, discusses what it can and cannot do, and covers the basic principles of conversion rate optimization so you can avoid risk, use data effectively and find solutions to YOUR problems.

Originally presented at National Multifamily Housing Council NMHC Optech Nov. 2016 on behalf of G5.

NMHC OPTECH is the premier destination for what’s next in multifamily. Open to NMHC members and non-members alike, OPTECH provides a forum for senior practice leaders from a wide variety of disciplines to collaborate on strategic action, incubate new ideas and elevate the industry across all areas of business operations and technology.

Published in: Marketing
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
155
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

The Evolution of A/B Testing - NMHC Optech 2016

  1. 1. THE EVOLUTION OF A/B TESTING @Nicole_Mintiens Data Analysis & Insights Manager @GetG5
  2. 2. 41 Shades of Blue The Evolution of A/B Testing @Nicole_Mintiens
  3. 3. $200 MILLION The Evolution of A/B Testing @Nicole_Mintiens
  4. 4. The Evolution of A/B Testing @Nicole_Mintiens
  5. 5. Advertising and Marketing • Improved ROI = Good The Evolution of A/B Testing @Nicole_Mintiens
  6. 6. Advertising and Marketing • Improved ROI = Good • Wasted Resources = Bad The Evolution of A/B Testing @Nicole_Mintiens
  7. 7. Advertising and Marketing • Improved ROI = Good • Wasted Resources = Bad • Marketing = Complex The Evolution of A/B Testing @Nicole_Mintiens
  8. 8. Advertising and Marketing • Improved ROI = Good • Wasted Resources = Bad • Marketing = Complex • Facts > Opinion The Evolution of A/B Testing @Nicole_Mintiens
  9. 9. WHY SPEND TIME TESTING? • $92 traffic /$1 optimization Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014 The Evolution of A/B Testing @Nicole_Mintiens
  10. 10. WHY SPEND TIME TESTING? • $92 traffic /$1 optimization • 87% run 1-5 tests/month Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014 The Evolution of A/B Testing @Nicole_Mintiens
  11. 11. WHY SPEND TIME TESTING? • $92 traffic /$1 optimization • 87% run 1-5 tests/month – Best brands run hundreds per month Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014 The Evolution of A/B Testing @Nicole_Mintiens
  12. 12. A/B TESTING FACTS • Improves conversion rate The Evolution of A/B Testing @Nicole_Mintiens
  13. 13. A/B TESTING FACTS • Improves conversion rate • Enhances effectiveness The Evolution of A/B Testing @Nicole_Mintiens
  14. 14. A/B TESTING FACTS • Improves conversion rate • Enhances effectiveness • Optimizes ROI The Evolution of A/B Testing @Nicole_Mintiens
  15. 15. A/B TESTING FACTS • Improves conversion rate • Enhances effectiveness • Optimizes ROI So… YOU should test! The Evolution of A/B Testing @Nicole_Mintiens
  16. 16. The Evolution of A/B Testing @Nicole_Mintiens
  17. 17. MISCONCEPTIONS • Hurt conversion rate The Evolution of A/B Testing @Nicole_Mintiens
  18. 18. MISCONCEPTIONS • Hurt conversion rate • Reduce effectiveness The Evolution of A/B Testing @Nicole_Mintiens
  19. 19. MISCONCEPTIONS • Hurt conversion rate • Reduce effectiveness • Waste resources The Evolution of A/B Testing @Nicole_Mintiens
  20. 20. MISCONCEPTIONS • Hurt conversion rate • Reduce effectiveness • Waste resources • Damage ROI The Evolution of A/B Testing @Nicole_Mintiens
  21. 21. TESTING IS NOT OPTIMIZATION The Evolution of A/B Testing @Nicole_Mintiens
  22. 22. TESTING IS NOT OPTIMIZATION …but it plays a role. The Evolution of A/B Testing @Nicole_Mintiens
  23. 23. DON’T JUST TEST, OPTIMIZE. • Avoid mistakes The Evolution of A/B Testing @Nicole_Mintiens
  24. 24. DON’T JUST TEST, OPTIMIZE. • Avoid mistakes • Increase effectiveness The Evolution of A/B Testing @Nicole_Mintiens
  25. 25. DON’T JUST TEST, OPTIMIZE. • Avoid mistakes • Increase effectiveness • Improve ROI The Evolution of A/B Testing @Nicole_Mintiens
  26. 26. I’LL BE YOUR GUIDE • 7+ years optimizing multifamily – PPC, SMM, SEO & CRO The Evolution of A/B Testing @Nicole_Mintiens
  27. 27. A/B TESTING BASICS • Statistical hypothesis-based experimentation between variations of a control. The Evolution of A/B Testing @Nicole_Mintiens
  28. 28. A/B TESTING BASICS • Statistical hypothesis-based experimentation between variations of a control. • Goal: – Isolate differences that maximize an outcome • i.e. conversion The Evolution of A/B Testing @Nicole_Mintiens
  29. 29. CONVERSION DEFINITION Action YOU want the user to take. The Evolution of A/B Testing @Nicole_Mintiens
  30. 30. A/B/N TESTING The Evolution of A/B Testing @Nicole_Mintiens
  31. 31. A/B/N TESTING The Evolution of A/B Testing @Nicole_Mintiens
  32. 32. A/B/N TESTING The Evolution of A/B Testing @Nicole_Mintiens
  33. 33. • 2000 – Google runs first test to find optimum number of SERP results. A/B TESTING HISTORY The Evolution of A/B Testing @Nicole_Mintiens
  34. 34. • 2000 – Google runs first test to find optimum number of SERP results. • 2008 – A/B tests raise additional $60 million for Obama campaign. A/B TESTING HISTORY The Evolution of A/B Testing @Nicole_Mintiens
  35. 35. • 2000 – Google runs first test to find optimum number of SERP results. • 2008 – A/B tests raise additional $60 million for Obama campaign. • 2010 – Obama campaign’s Director of Analytics founds Optimizely. A/B TESTING HISTORY The Evolution of A/B Testing @Nicole_Mintiens
  36. 36. • 2000 – Google runs first test to find optimum number of SERP results. • 2008 – A/B tests raise additional $60 million for Obama campaign. • 2010 – Obama campaign’s Director of Analytics founds Optimizely. • 2011 – Google has run over 7,000 A/B tests A/B TESTING HISTORY The Evolution of A/B Testing @Nicole_Mintiens
  37. 37. MULTIFAMILY AVG CONVERSION RATE Source: 2016 G5 Multifamily Google Analytics Data The Evolution of A/B Testing @Nicole_Mintiens
  38. 38. MULTIFAMILY AVG CONVERSION RATE Source: 2016 G5 Multifamily Google Analytics Data ~2% The Evolution of A/B Testing @Nicole_Mintiens
  39. 39. ECOMMERCE AVG CONVERSION RATE ~3% The Evolution of A/B Testing @Nicole_Mintiens
  40. 40. AMAZON PRIME’S CONVERSION RATE Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time The Evolution of A/B Testing @Nicole_Mintiens
  41. 41. 2366% Better than average! AMAZON PRIME’S CONVERSION RATE Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time The Evolution of A/B Testing @Nicole_Mintiens
  42. 42. AMAZON’S TECHNOLOGY Source: Builtwith.com The Evolution of A/B Testing @Nicole_Mintiens
  43. 43. AMAZON’S TECHNOLOGY Source: Builtwith.com The Evolution of A/B Testing @Nicole_Mintiens
  44. 44. SOFTWARE ADVANCEMENTS • Marketing automation The Evolution of A/B Testing @Nicole_Mintiens
  45. 45. SOFTWARE ADVANCEMENTS • Marketing automation • Easy to install The Evolution of A/B Testing @Nicole_Mintiens
  46. 46. SOFTWARE ADVANCEMENTS • Marketing automation • Easy to install • WYSIWYG editors The Evolution of A/B Testing @Nicole_Mintiens
  47. 47. WHAT A/B TESTING CAN DO • Validate hypotheses The Evolution of A/B Testing @Nicole_Mintiens
  48. 48. WHAT A/B TESTING CAN DO • Validate hypotheses • Inform choices The Evolution of A/B Testing @Nicole_Mintiens
  49. 49. WHAT A/B TESTING CAN DO • Validate hypotheses • Inform choices • Insight for iteration The Evolution of A/B Testing @Nicole_Mintiens
  50. 50. WHAT A/B TESTING CAN’T DO • Determine YOUR –Problem Thanks, Boss! The Evolution of A/B Testing @Nicole_Mintiens
  51. 51. WHAT A/B TESTING CAN’T DO • Determine YOUR –Problem –User’s needs/wants Thanks, Boss! The Evolution of A/B Testing @Nicole_Mintiens
  52. 52. WHAT A/B TESTING CAN’T DO • Determine YOUR –Problem –User’s needs/wants • Improve traffic –Quality Thanks, Boss! The Evolution of A/B Testing @Nicole_Mintiens
  53. 53. WHAT A/B TESTING CAN’T DO • Determine YOUR –Problem –User’s needs/wants • Improve traffic –Quality –Volume Thanks, Boss! The Evolution of A/B Testing @Nicole_Mintiens
  54. 54. CAN DOES NOT EQUAL SHOULD Testing will waste your time without: • Enough traffic The Evolution of A/B Testing @Nicole_Mintiens
  55. 55. CAN DOES NOT EQUAL SHOULD Testing will waste your time without: • Enough traffic • Enough research The Evolution of A/B Testing @Nicole_Mintiens
  56. 56. CAN DOES NOT EQUAL SHOULD Testing will waste your time without: • Enough traffic • Enough research • Technical resources The Evolution of A/B Testing @Nicole_Mintiens
  57. 57. THEN HOW DO YOU IMPROVE?! The Evolution of A/B Testing @Nicole_Mintiens
  58. 58. A PROCESS to improve performance informed by INSIGHT and defined by GOALS. CONVERSION RATE OPTIMIZATION (CRO) The Evolution of A/B Testing @Nicole_Mintiens
  59. 59. A PROCESS to improve performance informed by INSIGHT and defined by GOALS. GOALS CONVERSION RATE OPTIMIZATION (CRO) The Evolution of A/B Testing @Nicole_Mintiens
  60. 60. ALWAYS ROOM FOR IMPROVEMENT • Customer acquisition cost The Evolution of A/B Testing @Nicole_Mintiens
  61. 61. ALWAYS ROOM FOR IMPROVEMENT • Customer acquisition cost • Return on investment The Evolution of A/B Testing @Nicole_Mintiens
  62. 62. ALWAYS ROOM FOR IMPROVEMENT • Customer acquisition cost • Return on investment • Customer understanding The Evolution of A/B Testing @Nicole_Mintiens
  63. 63. ALWAYS ROOM FOR IMPROVEMENT • Customer acquisition cost • Return on investment • Customer understanding • Google’s perception The Evolution of A/B Testing @Nicole_Mintiens
  64. 64. GOALS & TRACKING Source: http://borsen.dk/opinion/blogs/view/17/4159/den_svaereste_marketingdisciplin.html The Evolution of A/B Testing @Nicole_Mintiens
  65. 65. YOUR IDEAL RESIDENTS Prospects Value I only rent the most affordable apartments. The Evolution of A/B Testing @Nicole_Mintiens
  66. 66. YOUR IDEAL RESIDENTS Prospects Value I only rent the most affordable apartments. I’ll rent near my tech coworkers regardless of price. The Evolution of A/B Testing @Nicole_Mintiens
  67. 67. CUSTOMER RESEARCH Source: https://blog.kissmetrics.com/wp-content/uploads/2015/02/NYU.lowres.ifiwere.jpg The Evolution of A/B Testing @Nicole_Mintiens
  68. 68. “The EFFECTIVENESS of any lead generation campaign comes down to your HOOK. ”Mike King @IPullRank Founder iPullRank Consultant to Fortune/INC500 brands. The Evolution of A/B Testing @Nicole_Mintiens
  69. 69. VOICE OF CUSTOMER RESEARCH • What makes your property unique? The Evolution of A/B Testing @Nicole_Mintiens
  70. 70. VOICE OF CUSTOMER RESEARCH • What makes your property unique? • Why you and not the competition? The Evolution of A/B Testing @Nicole_Mintiens
  71. 71. VOICE OF CUSTOMER RESEARCH • What makes your property unique? • Why you and not the competition? The Evolution of A/B Testing @Nicole_Mintiens
  72. 72. UNIQUE VALUE PROPOSITION (UVP): Above-the-fold EXPERIENCE that communicates WHAT you do and WHY it’s UNIQUE. The Evolution of A/B Testing @Nicole_Mintiens
  73. 73. Original (A) UVP: FIVE SECOND TEST The Evolution of A/B Testing @Nicole_Mintiens
  74. 74. Original (A) Variation (B) UVP: FIVE SECOND TEST The Evolution of A/B Testing @Nicole_Mintiens
  75. 75. Original (A) Variation (B) UVP: FIVE SECOND TEST The Evolution of A/B Testing @Nicole_Mintiens
  76. 76. CUSTOMER SEGMENTATION The Evolution of A/B Testing @Nicole_Mintiens
  77. 77. CUSTOMER SEGMENTATION User Persona Creator: https://xtensio.com/user-persona/ The Evolution of A/B Testing @Nicole_Mintiens
  78. 78. Slide credit to Angie Schottmuller - Check out the full presentation: http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grail The Evolution of A/B Testing @Nicole_Mintiens
  79. 79. LARGEST LEAKS Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg The Evolution of A/B Testing @Nicole_Mintiens
  80. 80. LARGEST LEAKS Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg The Evolution of A/B Testing @Nicole_Mintiens
  81. 81. ANALYZE ENGAGEMENT PATTERNS Heatmaps The Evolution of A/B Testing @Nicole_Mintiens
  82. 82. ANALYZE ENGAGEMENT PATTERNS Heatmaps The Evolution of A/B Testing @Nicole_Mintiens
  83. 83. ANALYZE PERCEPTIONS The Evolution of A/B Testing @Nicole_Mintiens
  84. 84. ANALYSIS FRAMEWORK G5 D-LITE Deliver Relevance - Match then exceed user expectations.D L I T E The Evolution of A/B Testing @Nicole_Mintiens
  85. 85. ANALYSIS FRAMEWORK G5 D-LITE Lower Friction - Reduce user anxiety. Simplify their process. Deliver Relevance - Match then exceed user expectations.D L I T E The Evolution of A/B Testing @Nicole_Mintiens
  86. 86. ANALYSIS FRAMEWORK G5 D-LITE Improve Clarity - Ensure messaging is clear, concise & valued. Lower Friction - Reduce user anxiety. Simplify their process. Deliver Relevance - Match then exceed user expectations.D L I T E The Evolution of A/B Testing @Nicole_Mintiens
  87. 87. ANALYSIS FRAMEWORK G5 D-LITE Improve Clarity - Ensure messaging is clear, concise & valued. Lower Friction - Reduce user anxiety. Simplify their process. Deliver Relevance - Match then exceed user expectations. Trigger Action - Motivate conversion. Remove distraction. D L I T E The Evolution of A/B Testing @Nicole_Mintiens
  88. 88. ANALYSIS FRAMEWORK G5 D-LITE Improve Clarity - Ensure messaging is clear, concise & valued. Lower Friction - Reduce user anxiety. Simplify their process. Deliver Relevance - Match then exceed user expectations. Experiment - Learn, test and optimize for your users. Trigger Action - Motivate conversion. Remove distraction. D L I T E The Evolution of A/B Testing @Nicole_Mintiens
  89. 89. “A solid test hypothesis is an INFORMED SOLUTION to a real problem - not an arbitrary guess. ”Michael Aagaard @ContentVerve International Speaker and Senior CRO @Unbounce The Evolution of A/B Testing @Nicole_Mintiens
  90. 90. HYPOTHESIS TEMPLATE If [cause], then [effect], because [rationale]. Source: https://help.optimizely.com/Ideate_and_Hypothesize/Design_a_hypothesis_that_drives_your_business_goals The Evolution of A/B Testing @Nicole_Mintiens
  91. 91. TESTING The Evolution of A/B Testing @Nicole_Mintiens
  92. 92. TESTING Wait… do you even have enough traffic? The Evolution of A/B Testing @Nicole_Mintiens
  93. 93. Gives you CONFIDENCE that the changes you make to your website ACTUALLY have a positive IMPACT. https://www.optimizely.com/optimization-glossary/statistical-significance/ STATISTICAL SIGNIFICANCE “ ” The Evolution of A/B Testing @Nicole_Mintiens
  94. 94. MULTIFAMILY AVG SESSIONS/DAY Source: 2016 G5 Multifamily Google Analytics Data The Evolution of A/B Testing @Nicole_Mintiens
  95. 95. MULTIFAMILY AVG SESSIONS/DAY Source: 2016 G5 Multifamily Google Analytics Data 66 The Evolution of A/B Testing @Nicole_Mintiens
  96. 96. TEST DURATION CALCULATOR Source: https://www.convert.com/tools/ab-test-duration-calculator/ Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html The Evolution of A/B Testing @Nicole_Mintiens
  97. 97. TEST DURATION CALCULATOR Source: https://www.convert.com/tools/ab-test-duration-calculator/ Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html The Evolution of A/B Testing @Nicole_Mintiens
  98. 98. LOW TRAFFIC OPTIMIZATION Source: http://unbounce.com/landing-pages/i-dont-have-enough-traffic-to-ab-test-my-landing-pages/ The Evolution of A/B Testing @Nicole_Mintiens
  99. 99. A/B TESTING TOOLS The Evolution of A/B Testing @Nicole_Mintiens
  100. 100. A/B TESTING TOOLS The Evolution of A/B Testing @Nicole_Mintiens
  101. 101. BUILD VARIATIONS Use technical resources to ensure proper setup and tracking The Evolution of A/B Testing @Nicole_Mintiens
  102. 102. BUILD VARIATIONS Use technical resources to ensure proper setup and tracking The Evolution of A/B Testing @Nicole_Mintiens
  103. 103. LAUNCH TEST Original (A) Variation (B) The Evolution of A/B Testing @Nicole_Mintiens
  104. 104. LAUNCH TEST Original (A) Variation (B) The Evolution of A/B Testing @Nicole_Mintiens
  105. 105. MEASURE RESULTS Image source: https://www.optimizely.com/ab-testing/ The Evolution of A/B Testing @Nicole_Mintiens
  106. 106. MEASURE RESULTS Image source: https://www.optimizely.com/ab-testing/ The Evolution of A/B Testing @Nicole_Mintiens
  107. 107. DRAW INSIGHTS AND ITERATE The Evolution of A/B Testing @Nicole_Mintiens
  108. 108. Your properties aren't 1-size fits all and your strategy shouldn't be either. DRAW INSIGHTS AND ITERATE The Evolution of A/B Testing @Nicole_Mintiens
  109. 109. MOST IMPORTANTLY Research and user feedback is VITAL. The Evolution of A/B Testing @Nicole_Mintiens
  110. 110. REPEAT GOALS The Evolution of A/B Testing @Nicole_Mintiens
  111. 111. THERE IS NO BEST COLOR 41 Shades of Blue The Evolution of A/B Testing @Nicole_Mintiens
  112. 112. INDUSTRY BENCHMARKS • Contextless data The Evolution of A/B Testing @Nicole_Mintiens
  113. 113. INDUSTRY BENCHMARKS • Contextless data • Weak conclusions The Evolution of A/B Testing @Nicole_Mintiens
  114. 114. INDUSTRY BENCHMARKS • Contextless data • Weak conclusions • Blind duplication The Evolution of A/B Testing @Nicole_Mintiens
  115. 115. INDUSTRY BENCHMARKS • Contextless data • Weak conclusions • Blind duplication • Tactics over strategy The Evolution of A/B Testing @Nicole_Mintiens
  116. 116. USE STRATEGY, NOT JUST TACTICS • Avoid risks The Evolution of A/B Testing @Nicole_Mintiens
  117. 117. USE STRATEGY, NOT JUST TACTICS • Avoid risks • Use data effectively The Evolution of A/B Testing @Nicole_Mintiens
  118. 118. USE STRATEGY, NOT JUST TACTICS • Avoid risks • Use data effectively • Discover solutions to YOUR problems The Evolution of A/B Testing @Nicole_Mintiens
  119. 119. FREEMIUM (OR CLOSE) RESOURCES • Research: – Personas – Tracking – Heatmapping – Test Duration Calculator The Evolution of A/B Testing @Nicole_Mintiens
  120. 120. FREEMIUM (OR CLOSE) RESOURCES • Research: – Personas – Tracking – Heatmapping – Test Duration Calculator • Analysis Tools: G5 D-LITE CRO Framework The Evolution of A/B Testing @Nicole_Mintiens
  121. 121. FREEMIUM (OR CLOSE) RESOURCES • Research: – Personas – Tracking – Heatmapping – Test Duration Calculator • Analysis Tools: G5 D-LITE CRO Framework • Hypotheses: If [cause], then [effect], because [rationale] The Evolution of A/B Testing @Nicole_Mintiens
  122. 122. FREEMIUM (OR CLOSE) RESOURCES • Research: – Personas – Tracking – Heatmapping – Test Duration Calculator • Analysis Tools: G5 D-LITE CRO Framework • Hypotheses: If [cause], then [effect], because [rationale] • Testing: The Evolution of A/B Testing @Nicole_Mintiens
  123. 123. “CRO IS LEARNING WHY YOUR USERS aren’t converting and FIXING IT.” The Evolution of A/B Testing @Nicole_Mintiens
  124. 124. GET THESE SLIDES: goo.gl/TPUOgA THANKS! DON’T JUST TEST, OPTIMIZE. The Evolution of A/B Testing @Nicole_Mintiens

×