Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Would you bet your job on your A/B test results?

984 views

Published on

It is possible that 72% of your successful A/B tests may not be driving any business benefit, or may actually be harming your bottom line. This is because bad methodology is preventing you from detecting the truly successful ideas.

In this webinar, Qubit explains how you can avoid this problem. With special guests from Forrester who explain how A/B testing fits into the digital landscape, and Staples, who give practical advice for how to set up an A/B testing campaign, this webinar will change the way you think about A/B testing forever

Published in: Business
  • Be the first to comment

Would you bet your job on your A/B test results?

  1. 1. Would you bet your job on your A/B test results? @QubitGroup #QubitABtest Friday, 28 February 14
  2. 2. Agenda • 5 digital experience technology trends that will guide your technology investment — Anjali Yakkundi, Forrester Research, Inc. • 5 fundamental A/B testing mistakes and how to avoid them —Will Browne, Qubit • How to run a successful A/B testing program —Melanie Kyrklund, Staples • Q&A @QubitGroup #QubitABtest Friday, 28 February 14
  3. 3. 5 Digital Experience Technology Trends To Guide Your Investment Anjali Yakkundi, Analyst February 27, 2014 Friday, 28 February 14
  4. 4. Burberry brings digital into the store © 2013 Forrester Research, Inc. Reproduction Prohibited 4 Friday, 28 February 14
  5. 5. Room Number Address & directions Starwood’s Preferred Guest Streamlines The Check-In Process © 2013 Forrester Research, Inc. Reproduction Prohibited 5 Friday, 28 February 14
  6. 6. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 6 Friday, 28 February 14
  7. 7. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 7 Friday, 28 February 14
  8. 8. Age of the customer Source: October 10, 2013, “Competitive Strategy In The Age Of The Customer” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 8 Friday, 28 February 14
  9. 9. Your customers are always connected… Source: September 19, 2013, “Build Seamless Experiences Now” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 9 Friday, 28 February 14
  10. 10. …often on multiple devices Source: Google - The New Multi-screen World: Understanding Cross-platform Consumer Behavior, August 2012 © 2013 Forrester Research, Inc. Reproduction Prohibited 10 Friday, 28 February 14
  11. 11. You need to sustain your customer’s attention with engaging experiences. Friday, 28 February 14
  12. 12. Engaging experiences requires strong technology management © 2013 Forrester Research, Inc. Reproduction Prohibited 12 Friday, 28 February 14
  13. 13. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 13 Friday, 28 February 14
  14. 14. Digital experience reference architecture Friday, 28 February 14
  15. 15. Many solutions make up the digital experience ecosystem Testing Site search Commerce platform Product content management Optimization Portals Web content management Digital asset management Analytics (mobile, social, Web) Recommendation engine Online video platform Campaign management Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 15 Friday, 28 February 14
  16. 16. Testing and optimization solutions are key to engagement and measurement Testing Site search Commerce platform Product content management Optimization Portals Web content management Digital asset management Analytics (mobile, social, Web) Recommendation engine Online video platform Campaign management Source: February 12, 2013, “Choose Digital Experience Tools To Engage With Customers In A Cross-Channel World” Forrester report © 2013 Forrester Research, Inc. Reproduction Prohibited 16 Friday, 28 February 14
  17. 17. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 17 Friday, 28 February 14
  18. 18. #1- This is the beginning of the end of the allin-one-suite 9% Only prefer a suite approach Base: 233 Digital Experience Professionals Source: March 2013, “Digital Experience Delivery Online Survey” © 2013 Forrester Research, Inc. Reproduction Prohibited 18 Friday, 28 February 14
  19. 19. #2- Tools must create business value out of data Solutions must effectively harness customer data, product data, social media data, and other information to create and optimize experiences. © 2013 Forrester Research, Inc. Reproduction Prohibited 19 Friday, 28 February 14
  20. 20. #3- Tools must support personalized experiences • • • • • Likely next steps Device / Channel • Historical past • Immediate past Location • Current session Time of day Weather Situation Real-time optimization Prediction © 2013 Forrester Research, Inc. Reproduction Prohibited 20 Friday, 28 February 14
  21. 21. #4- Look for more than software: look for solutions › Look for solutions that help marketers and business users quickly adapt to changing customer needs. This can be achieved by: •Keeping IT out of critical publishing paths •Prioritizing user interface needs © 2013 Forrester Research, Inc. Reproduction Prohibited 21 Friday, 28 February 14
  22. 22. #5- Solutions must prioritize multichannel © 2013 Forrester Research, Inc. Reproduction Prohibited 22 Friday, 28 February 14
  23. 23. Agenda › Why digital experience technology matters › Key digital customer experience technologies › Five trends that will drive your digital experience technology investment › Recommendations © 2013 Forrester Research, Inc. Reproduction Prohibited 23 Friday, 28 February 14
  24. 24. Recommendations › You can’t do it all at once — prioritize key customer touchpoints › Assess current technology gaps– most organizations tend to have gaps around: •Optimization •Personalization •Multichannel delivery › When sourcing new solutions, look to vendors that can help you connect the dots. © 2013 Forrester Research, Inc. Reproduction Prohibited 24 Friday, 28 February 14
  25. 25. Thank you Anjali Yakkundi ayakkundi@forrester.com @Ayakkundi Friday, 28 February 14
  26. 26. 5 fundamental A/B testing mistakes and how to avoid them Will Browne, Qubit will.browne@qubitdigital.com @QubitGroup #QubitABtest Friday, 28 February 14
  27. 27. Why do I have to test? Friday, 28 February 14
  28. 28. A/B testing is an essential tool @TheGrahamCooke Friday, 28 February 14
  29. 29. A/B testing is an essential tool • It is the best, simplest way to prove the business value of your digital technology @QubitGroup @TheGrahamCooke Friday, 28 February 14 #QubitABtest
  30. 30. A/B testing is an essential tool • It is the best, simplest way to prove the business value of your digital technology • It is the only way to get close to proving cause and effect in what you do @QubitGroup @TheGrahamCooke Friday, 28 February 14 #QubitABtest
  31. 31. But there are costs associated with A/B testing Friday, 28 February 14
  32. 32. But there are costs associated with A/B testing • The monthly cost of your testing platform @QubitGroup #QubitABtest Friday, 28 February 14
  33. 33. But there are costs associated with A/B testing • The monthly cost of your testing platform • The salaries and time of your developers and team @QubitGroup #QubitABtest Friday, 28 February 14
  34. 34. But there are costs associated with A/B testing • The monthly cost of your testing platform • The salaries and time of your developers and team • The lost revenue from implementing changes that do nothing for your site (or actively harm it) @QubitGroup #QubitABtest Friday, 28 February 14
  35. 35. We’re here to show you how to reduce these costs, and increase ROI @QubitGroup #QubitABtest Friday, 28 February 14
  36. 36. We’re here to show you how to reduce these costs, and increase ROI • Find the ideas that are likely to have a real positive effect on user behavior (true winners) @QubitGroup #QubitABtest Friday, 28 February 14
  37. 37. We’re here to show you how to reduce these costs, and increase ROI • Find the ideas that are likely to have a real positive effect on user behavior (true winners) • Detect as many true winners as possible @QubitGroup #QubitABtest Friday, 28 February 14
  38. 38. We’re here to show you how to reduce these costs, and increase ROI • Find the ideas that are likely to have a real positive effect on user behavior (true winners) • Detect as many true winners as possible • Minimize the chance of finding tests that look like winners, but have no true effect @QubitGroup #QubitABtest Friday, 28 February 14
  39. 39. Find the ideas that are likely to have a real positive effect on user behavior 39 @QubitGroup #QubitABtest Friday, 28 February 14
  40. 40. Size does matter Friday, 28 February 14
  41. 41. Detect as many true winners as possible We’ve found that tests based on data are 7.7x more effective than tests based on intuition alone (Qubit, 2013) Friday, 28 February 14
  42. 42. Imagine testing for weighted coins We’vemany coin flips would be on datato prove that one found that tests based enough are • How coin was weighted? 7.7x more effective than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  43. 43. Imagine testing for weighted coins We’vemany coin flips would be on datato prove that one found that tests based enough are • How coin was weighted? 7.7x more effective • It is impossible to be 100% certain, but the more data you have, the closer you can get than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  44. 44. Imagine testing for weighted coins We’vemany coin flips would be on datato prove that one found that tests based enough are • How coin was weighted? 7.7x more effective • It is impossible to be 100% certain, but the more data you have, the closer you can get • We follow scientific standards and wait until there is an than tests based on intuition alone 80% chance of detecting a difference (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  45. 45. Minimize the chance of finding tests that look like winners, but have no true effect We’ve found that tests based on data are 7.7x more effective than tests based on intuition alone (Qubit, 2013) Friday, 28 February 14
  46. 46. Minimize the chance of finding tests that look like winners, but have no true effect •We’ve found that tests based on data are there is a 5% We generally assume that with good testing false positive rate 7.7x more effective than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  47. 47. Minimize the chance of finding tests that look like winners, but have no true effect •We’ve found that tests based on data are there is a 5% We generally assume that with good testing false positive rate 7.7x more effective • We can test this using A/A tests (showing both groups the exact thing) than tests based on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  48. 48. Minimize the chance of finding tests that look like winners, but have no true effect •We’ve found that tests based on data are there is a 5% We generally assume that with good testing false positive rate 7.7x more effective • We can test this using A/A tests (showing both groups the exact thing) • So out of 100 A/A tests, 5 would be winners, despite than tests based nothing doing absolutely on intuition alone (Qubit, 2013) @QubitGroup #QubitABtest Friday, 28 February 14
  49. 49. Splitting tests too much can lead to false positives Friday, 28 February 14
  50. 50. Get a second opinion Friday, 28 February 14
  51. 51. Validate your results We recommend a 95/5 split to confirm your winning variation is still winning “Most winning A/B test results are illusory”, Qubit 2014 @QubitGroup #QubitABtest Friday, 28 February 14
  52. 52. A worked example: Imagine Mr Bean’s jelly beans shop “Most winning A/B test results are illusory”, Qubit 2014 @QubitGroup Friday, 28 February 14
  53. 53. A worked example: Imagine Mr Bean’s jelly beans shop • Mr Bean has a 1% conversion rate • He runs 20 different tests in one year • There are 2 good tests with uplifts of 5% • There are also 2 bad tests with uplifts of -5% “Most winning A/B test results are illusory”, Qubit 2014 @QubitGroup #QubitABtest @QubitGroup Friday, 28 February 14
  54. 54. Truth @QubitGroup #QubitABtest Friday, 28 February 14 Correct method Naive method
  55. 55. Truth Correct method Naive method No. of tests 20 20 20 Reported winners 2 3 9 @QubitGroup #QubitABtest Friday, 28 February 14
  56. 56. Truth Correct method Naive method No. of tests 20 20 20 Reported winners 2 3 9 After validation 2 2 9 @QubitGroup #QubitABtest Friday, 28 February 14
  57. 57. Truth Correct method Naive method No. of tests 20 20 20 Reported winners 2 3 9 After validation 2 2 9 2 x ~5% 9x (5-40)% Reported uplift 2 x 5% @QubitGroup #QubitABtest Friday, 28 February 14
  58. 58. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) @QubitGroup #QubitABtest Friday, 28 February 14
  59. 59. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) None 7 unnecessary tests Wasted resource @QubitGroup #QubitABtest Friday, 28 February 14
  60. 60. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) None 7 unnecessary tests Improved Degraded Improved Confused Wasted resource Customer experience Your understanding @QubitGroup #QubitABtest Friday, 28 February 14
  61. 61. Correct method Naive method Reported uplift 2 x 5% 9 x (5-40)% Delivered uplift 2 x 5% 2 x 5% + (? x -5%) None 7 unnecessary tests Improved Degraded Improved Confused Happy Angry Wasted resource Customer experience Your understanding Your manager @QubitGroup #QubitABtest Friday, 28 February 14
  62. 62. So, to conclude, those 5 mistakes are: • Mistake 1: Not A/B testing • Mistake 2: Not prioritizing tests properly • Mistake 3: Not waiting for significance and sample size • Mistake 4: Testing too many things at once • Mistake 5: Not validating the tests after implementation @QubitGroup #QubitABtest Friday, 28 February 14
  63. 63. Questions you should ask your testing provider • How do you prioritize tests? • How do you calculate the sample size required? • How do you account for the multiple testing problem? • How do you validate winning tests? @QubitGroup #QubitABtest Friday, 28 February 14
  64. 64. Thank you If you have any detailed questions please contact research@qubitdigital.com Will Browne, Qubit will.browne@qubitdigital.com @QubitGroup #QubitABtest Friday, 28 February 14
  65. 65. How to run a successful A/B testing program Melanie Kyrklund Senior Ecommerce Analyst Friday, 28 February 14
  66. 66. Testing programs have the potential to drive the online customer experience  Understand your customers and respond to their needs  Optimize the user journey Customer experience  Create personalized experiences  Quickly respond to market pressures Personalization Persuasion Targeted offers User experience, functionality & content  Rapidly align marketing & onsite experience  Validate development items  Every visit represents an opportunity to establish a relationship with a customer 66 Friday, 28 February 14
  67. 67. The reality of running a testing program For many organizations, testing represents a cultural change with many political and operational barriers to be overcome:  Resource & operational factors  Buy-in / Multiple stakeholders  Alignment with other business priorities and development plans  Instilling a data-driven culture 67 Friday, 28 February 14
  68. 68. A lot of testing is required to deliver results  1/5 tests or 1/10 tests will deliver statistically significant wins  A rapid pace of testing is required to build a program that is delivering consistent results month over month 68 Friday, 28 February 14
  69. 69. Two factors are critical in building a successful testing program Approach Speed Methodology Idea Generation Operational Process Prioritization 69 Friday, 28 February 14
  70. 70. The right approach to optimization Be inspired – every visit is an opportunity to establish a relationship with a customer Use data to understand how visitors are using your website, what is working and what isn’t Customer feedback to understand why visitors are not purchasing on your website If working over multiple websites & countries, look for insight at local level first and then identify commonalities at a later stage Persuasion research, understand the psychology of your customers Supplement with quick UX testing Thoroughly research & quantify each opportunity (visitors reached, expected uplift and incremental revenue) UX optimization Personalization Persuasion 70 Friday, 28 February 14
  71. 71. Idea Generation 71 Friday, 28 February 14
  72. 72. Idea Generation 71 Friday, 28 February 14
  73. 73. Idea Generation 71 Friday, 28 February 14
  74. 74. Idea Generation 71 Friday, 28 February 14
  75. 75. Speed – Operational Processes Testing Tool selection : Map out the most efficient implementation process  Operational efficiency  Pick managed services (analysis, implementation) as required Prioritization Build a test pipeline logging:  Idea & hypothesis  Expected uplift & incremental revenue  Implementation estimates  Test duration  Prioritize tests weekly, to keep up momentum of testing and focus on biggest wins 72 Friday, 28 February 14
  76. 76. Conclusion: A rapid, data-driven approach is required  Testing programs have the potential to drive the online customer experience, however a rapid data-driven approach is required to deliver results  Plan the required resources upfront  Consider ease of use, test time to market and managed services in tool selection  Map out the most efficient end to end process to get tests live  Process supported by data-driven idea generation & prioritization 73 Friday, 28 February 14
  77. 77. Thank you Any questions? Friday, 28 February 14
  78. 78. More A/B testing material from Qubit bit.ly/qABresearch 75 @QubitGroup #QubitABtests Friday, 28 February 14 bit.ly/qABinfographic

×