Greg Kelton Multivariate Testing

1,884 views

Published on

Greg Kelton, Managing Director, Autonomy Optimost EMEA, presented at the Orange Online Meetup event in Bucharest

Published in: Business, Technology

Greg Kelton Multivariate Testing

  1. 1. Multivariate Testing at Orange Driving more revenue online Greg Kelton Managing Director, Autonomy Optimost EMEA Interwoven Confidential
  2. 2. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 2 2
  3. 3. 1 A A Control B B +75% 3
  4. 4. Generating a 75% increase in Clickthroughs • 14 template permutations analyzed Control (creative 2) Winner (creative 11) UAR-Shop = x% UAR-Shop = x% +75% 4
  5. 5. 2 A A +6.97% B B Control 5
  6. 6. Generating a 53.42% lift (99% confidence) Total Recipes Analysed: 2,938 B4 F=1 Control generating a 53.42% lift (99% confidence) 6
  7. 7. 3 Flash (animated) A A B B Control +4.1% 7
  8. 8. Winning creative beat control by 10.0% on UAR-Congratulations and 8.2% on UAR-Profile Information Over 1 billion template permutations analyzed Control (creative 121) Winner (creative 271) UAR-Congratulations = x% UAR-Congratulations = x% +10.0% UAR-Profile Information = x% UAR-Profile Information = x% +8.2% 8
  9. 9. Who should control your marketing? Shareholders/ Shareholders/ Investors Investors Product Product Marketing Marketing Managers Managers Dept Dept Creative Creative CEO CEO Agencies Agencies Executives Executives Analysts Analysts 11
  10. 10. Who should control your marketing? Shareholders/ Shareholders/ Investors Investors Product Product Marketing Marketing Managers Managers Dept Dept None of the above Creative Creative CEO CEO Agencies Agencies Executives Executives Analysts Analysts 12
  11. 11. Who should control your marketing? Your marketing should be driven by your customers Your Your Customers Customers 13
  12. 12. 14
  13. 13. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 15 15
  14. 14. The Content Selection Process is Complex & Faulty How do you make web content decisions? • Speculation • HiPPO • Intuition • Agency experience • Argument • Quasi-objective • Focus groups What if Your Customers • Usability testing Could Tell You What • Web Surveys Content They Like? 16
  15. 15. 17
  16. 16. Old Facebook New Facebook 18
  17. 17. Top facebook Applications See 25 Percent Drop In Traffic Since Redesign facebook Poll: 94% Of Users Don’t Like Redesign 19
  18. 18. 20
  19. 19. 22
  20. 20. 23
  21. 21. 24
  22. 22. 25
  23. 23. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 29 29
  24. 24. But…. What’s the point? • Your online competitive advantage • Your Sales Funnel: • Ease of buying & customer experience? • Multi-channel influences? • Technology investment (Flash, AJAX, Adobe AIR?) • Your Operational Advantage: • Change management? • Cost reduction? • Putting business in control of your site? • Speed to market? • Your Brand & Offering: • Regional customers • Price sensitivity? By region? By demographic? 30
  25. 25. Affiliate Marketing Page Simplify the Buying Process • Simple version increased conversion by 3.86% 32
  26. 26. Putting business in control of your online channel Merchandising Challenge - Brands, price brackets, colours vary depending on product category •How many price brackets? •Should the number of products in stock be displayed? •How many brands? •Which order is best? •Are collapsible menus preferred? Benefit –understand how your customers prefer to interact with site technologies eg. reviews, search, navigation, shopping carts. 33
  27. 27. Shop Home Page 35
  28. 28. Shop Home Page C A, B 36
  29. 29. 30 total Page Recipes analysed Quiz! B Control A +17% 39
  30. 30. Product Page 40
  31. 31. Product Page 41
  32. 32. 60 total Page Recipes analysed Quiz! A -9% B +13% 43
  33. 33. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 47 47
  34. 34. 1. Pick a Page to Test 48
  35. 35. 2. Add Single Javascript Callout to Page <script optimost.displayModule("main"); </script> 49
  36. 36. 3. Brainstorm ideas to test 1 2 3 vs None 4 51
  37. 37. 4. Optimost Generates Page Recipes 52
  38. 38. 5. Run the Experiment Live on your site! 53
  39. 39. 6. Optimost declares the Best Recipe This Version Increases Original Version Conversions by 28.4% 54
  40. 40. 7. REPEAT 55
  41. 41. Summary – MVT Allows you to: Listen to your Customers Improve Revenue… consistently Avoid Cost and Risk of Redesigns 57
  42. 42. Thank You! Greg Kelton gkelton@interwoven.com 58

×