Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

1,010 views

Published on

Qubit's webinar with Econsultancy and Brooks Bell gave an honest insight into A/B testing challenges faced by Topshop, Shop Direct , Gatwick Airport and more!

  • Be the first to comment

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

  1. 1. The Past, Present and Future of Website Optimisation In association with Qubit Linus Gregoriadis Research Director @LinusGreg
  2. 2. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 6 | The Past, Present and Future of Website Optimisation
  3. 3. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 7 | The Past, Present and Future of Website Optimisation
  4. 4. Aims of the report ● To find out how companies are approaching the issue of testing and optimisation. ● To identify the barriers to optimisation. − Technological − Cultural ● To understand the processes and approaches used. ● To highlight any potential future challenges. | 8 | The Past, Present and Future of Website Optimisation
  5. 5. | 9 | The Past, Present and Future of Website Optimisation
  6. 6. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 10 | The Past, Present and Future of Website Optimisation
  7. 7. | 11 | The Past, Present and Future of Website Optimisation Photo credit: 123dan321 on stock.xchng Why test anyway?
  8. 8. Testing is key to improving website performance | 12 | The Past, Present and Future of Website Optimisation Source: Conversion Rate Optimisation Report 2014
  9. 9. Testing is key to improving website performance ● Companies seeing significant increases in sales typically run more A/B and multivariate tests Source: Conversion Rate Optimisation Report 2014 | 13 | The Past, Present and Future of Website Optimisation
  10. 10. “If it is going to fail, fail fast… “We see a positive benefit from around a third of tests.” Chris Howard, Head of Digital, Shop Direct | 14 | The Past, Present and Future of Website Optimisation
  11. 11. Case Study: Topshop | 15 | The Past, Present and Future of Website Optimisation ● Wanted to improve US experience ● Tested using size formats more familiar to North American shoppers ● Measured not on conversion but revenue optimisation ● Increased revenue by 4.5% with 99% statistical significance
  12. 12. | 16 | The Past, Present and Future of Website Optimisation Photo credit: 123dan321 on stock.xchng What to test? What to aim for?
  13. 13. “The onus is on optimising the funnel and we can’t do it all at once. We’ve started having grown-up conversations about what is the best point to focus optimisation on.” Daniel Sale, Head of Digital Marketing, Investec Private Bank | 17 | The Past, Present and Future of Website Optimisation
  14. 14. Advanced Website Optimisation is yet to get over the adoption curve… | 18 | The Past, Present and Future of Website Optimisation
  15. 15. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 19 | The Past, Present and Future of Website Optimisation
  16. 16. How much to test? | 20 | The Past, Present and Future of Website Optimisation
  17. 17. “We’ve learned what degrees of success you can have. You can chip away and only move the dial so far unless you are doing something significantly wrong.” Paul Wishman, Group Ecommerce Director, LV | 21 | The Past, Present and Future of Website Optimisation
  18. 18. Size matters | 22 | The Past, Present and Future of Website Optimisation
  19. 19. “Realistically we can run three to four tests a month simultaneously… we never get the volume to the extent of retail banks for example. We need statistically significant samples and that takes time.” Daniel Sale, Head of Digital Marketing, Investec Private Bank | 23 | The Past, Present and Future of Website Optimisation
  20. 20. “Apart from choosing the right tools, most of the obstacles were related to internal culture and getting buy-in from internal stakeholders.” David Paice, Group Ecommerce and Admissions Director, Merlin Entertainments | 24 | The Past, Present and Future of Website Optimisation
  21. 21. “Testing has been an internal communications piece and especially because A/B testing involves so many different teams.” Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch | 25 | The Past, Present and Future of Website Optimisation
  22. 22. Decisions on technology | 26 | The Past, Present and Future of Website Optimisation
  23. 23. • Bought-in software, either off the shelf or customised? • Built in-house? • Managed services (when needs are constantly changing)? | 27 | The Past, Present and Future of Website Optimisation
  24. 24. Can your vendor grow with you? | 28 | The Past, Present and Future of Website Optimisation
  25. 25. Introduction and Methodology The Business Case Foundations and Resourcing Moving Towards Personalisation | 29 | The Past, Present and Future of Website Optimisation
  26. 26. “Personalisation is the most exciting challenge and the hardest. There’s so much we can do but it’s finding out what you should and what will add value.” Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch | 30 | The Past, Present and Future of Website Optimisation
  27. 27. Case Study: Farfetch | 31 | The Past, Present and Future of Website Optimisation ● Farfetch hypothesised that people convert more when they understand the business model ● Segment of users: new visitors with over 10 page views in their session ● A message layer slides in, prompting users to read FAQs ● Result: 17% increase in conversion
  28. 28. 93% of companies saw an uplift in conversion rates since implementing personalisation. | 32 | The Past, Present and Future of Website Optimisation Source: Conversion Rate Optimisation Report 2014
  29. 29. Ensure incremental | 33 | The Past, Present and Future of Website Optimisation gain
  30. 30. “Testing by revenue is critical to our business. Conversion tells only part of the story.” Simon Pritchard, Group Digital Director, Arcadia | 34 | The Past, Present and Future of Website Optimisation
  31. 31. One to One? | 35 | The Past, Present and Future of Website Optimisation
  32. 32. “It’s the nature of fashion for consumers largely to behave like the herd in many ways. Perhaps one-to-one conversion isn’t necessary. If you can identify groups of people, that’s good enough.” Stuart McMillan, Deputy Head of Ecommerce, Schuh | 36 | The Past, Present and Future of Website Optimisation
  33. 33. Case Study: DFS | 37 | The Past, Present and Future of Website Optimisation ● Hypothesised: European customers may need more information when shopping on UK websites ● Targeted users from outside the UK with a layer informing them that DFS shipped to their country ● Result: 23% uplift in conversions for segment
  34. 34. Over time the majority will adopt best practice | 38 | The Past, Present and Future of Website Optimisation
  35. 35. Final Takeaways ● Testing has seen a surge of growth but barriers still remain. ● People elements are often as significant as technological ones. ● A pragmatic yet continual approach is most likely to pay future dividends. | 39 | The Past, Present and Future of Website Optimisation
  36. 36. PEOPLE, PROCESS, TECHNOLOGY NAOSHI YAMAUCHI, CPO @Yamalytics
  37. 37. Brooks Bell The premier firm focused exclusively on enterprise-level A/B testing, optimization and personalization services. SOME OF OUR CLIENTS
  38. 38. TALENT, CULTURE, PROCESS ISSUES CAUSING ‘UTILIZATION GAP’ Optimization tools increasing in sophistication • Tracking across platforms • Personalization • Mobile • More data! 42 Capabilities
  39. 39. TALENT, CULTURE, PROCESS ISSUES CAUSING ‘UTILIZATION GAP’ Optimization tools increasing in sophistication • Tracking across platforms • Personalization • Mobile • More data! 43 Insufficient Team & Talent • Business savvy analysts • Technical talent • Data savvy executive influencers Capabilities Broken Process • Lack of standards • Minimal documentation Resistant Culture • Gut-driven decision making • Minimal training & sharing UTILIZATION GAP Use DRIVING ISSUES
  40. 40. 44 Utilization ranks 20th out of 24 total categories in Top Scoring Assessment Brooks Bell Strategic Assessment Categories Worst Scoring 24
  41. 41. So, What can you do? 4 TIPS ON HOW TO CLOSE THE UTLILZATION GAP
  42. 42. 46 ASK YOURSELF: “Do we really have the winning team?” If the answer is no, you’re not alone ☺. tip1
  43. 43. OPTIMIZATION TEAM Strategy Design Engineering Project Management Analytics Creative
  44. 44. EVANGELIZE & BUILD CULTURE GET INFLUENCERS RECOGNIZED IN TEST WINS tip2
  45. 45. WHO IS BROOKS BELL? WE RING A GONG EVERY TIME WE LAUNCH A TEST
  46. 46. CREATE A REPEATABLE PROCESS Standardize and then document campaign, dev, QA, test archives YOU CANNOT SCALE AND MAINTAIN MOMENTUM WITHOUT PROCESS & STRUCTURE tip3
  47. 47. 51 THE STORY OF TESTING AT BRAND X 2011 2012 2013 2014 COMPLEXITY DEMAND TEAM SIZE # OF TESTS PROCESS CULTURE DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014
  48. 48. tip4 START TARGETING SEGMENTS FOR TESTING. IT WILL MAKE IDEATION MORE EFFECTIVE. DOWNLOAD! Campaign QA Checklist – bbi.bz/qa-2014
  49. 49. Case Study: New v. Returning Segment: New customers Goal: Increase orders Issue: Confused as to what the different fits mean CONTROL CHALLENGER
  50. 50. 54 TIP 1: Ask yourself “Do we really have a winning team?” TIP 2: Get influencers recognized in test wins TIP 3: Create a repeatable process TIP 4: Start targeting segments for testing

×