Successfully reported this slideshow.
Your SlideShare is downloading. ×

Preparing for AI - Measurefest

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 99 Ad

More Related Content

Slideshows for you (20)

Similar to Preparing for AI - Measurefest (20)

Advertisement

More from Guido X Jansen (20)

Advertisement

Preparing for AI - Measurefest

  1. 1. © Randstad Group Communications, July 2017 | 1 preparing for AI: our journey to new optimization methods
  2. 2. how can we make the next step?
  3. 3. how can we maximize output (in terms of learnings and conversion uplift) with fixed resources?
  4. 4. faster optimization loops!
  5. 5. CRO is tough! © Randstad
  6. 6. optimizing for the “average user” does (s)he exist? © Randstad
  7. 7. I need more resources!!! get in line © Randstad
  8. 8. too many variables impacting conversion! ● Customer persona’s ● Countries ● Budgets ● Customer history ● Regulations ● Colors ● Flower seasons ● Product types ● Holiday peaks ● Time of day/week ● Purchasing power ● Culture ● Devices ● Social media platforms ● Acquisition channels ● Symbols ● Step in Customer Journey ● Tech adoption level ● Accessoires ● Multi-channel funnels ● Etiquette ● Consumer vs Corporate ● Platform ● Different payment systems ● Brand awareness/perception ● Logistic situation ● Aesthetics ● Traditions ● Social context ● Weather ● Competition campaigns ● Consumer state-of-mind ● Competitor A/B test ● ...
  9. 9. and to make it worse… © Randstad
  10. 10. these variables keep changing all the ffing time! © Randstad
  11. 11. but first some context © Randstad
  12. 12. Guido Jansen global CRO Lead randstad
  13. 13. || randstad in numbers (year-end 2016) second HR services provider worldwide. € 20.7 billion in revenue. 32,280 avg. corporate employees. 626,300 people employed every day. 183,900 number of permanent placements. 4,752 offices in 39 countries. 45.2 % women in leadership positions. © Randstad 2017 14
  14. 14. Previously: Chief Psychology Officer / CRO Manager
  15. 15. || CRO building blocks. © Randstad 2017 ExperimentsPsychologyQual/Quant Data
  16. 16. || our global presence. © Randstad 2017 Andorra Luxembourg Argentina Malaysia Australia Mexico Austria Monaco Belgium Netherlands Brazil New Zealand Canada Norway Chile Poland China Portugal Czech Republic Romania Denmark Sweden France Singapore Germany Spain Greece Switzerland Hong Kong Turkey Hungary United Arab Emirates India United Kingdom Ireland United States Italy Uruguay Japan (represents over 90% of the global HR services market.)
  17. 17. || our brands. © Randstad 2017
  18. 18. Live Testing Complete/Analysis Deployments Design Development Q&A Pending approval Approved Idea Hypotheses Full testing plan our optimization loop
  19. 19. slight warning… ...but slides will be shared! there will be data & details
  20. 20. how can we make the next step?
  21. 21. How are others doing this?
  22. 22. “[Booking.com’s] utilization of A/B testing … drives higher conversion across it’s entire platform, resulting in conversion levels 2-3x the industry average” Source: Evercore Equity Research
  23. 23. Sadly, when you test more, the % of tests with an uplift will go down.
  24. 24. When A/B testing a fully optimized site, the % of tests with an uplift is low(er).
  25. 25. 1 in 5 tests with uplift 1 in 10 tests with uplift Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017
  26. 26. How to get more tests with an uplift?
  27. 27. 10-20 concurrent test 1000 concurrent tests Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017
  28. 28. 1 optimization team 75 product teams Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017
  29. 29. Of course, we are not ...
  30. 30. ...but we do know a thing (or 2) about experiments...
  31. 31. ...so let’s experiment with the way we experiment!
  32. 32. Let’s look for solutions... © Randstad 35
  33. 33. (evolutionary) AI/ML? sounds cool, tell me more! © Randstad
  34. 34. Possible applications © Randstad 37
  35. 35. Different applications of AI in E-commerce ● Catalog optimization ← already tried this ● Personalization ● RTB / Advertising ● Chatbots ● Virtual personal shopper ● Improved optimization process ● ...
  36. 36. We tried AI catalog optimization
  37. 37. Previous experience: ● “Black box” ● No (perceived) control for E-commerce managers ● No proven uplift in CR ● Hard to do for (emotional) consumers buying for someone else?
  38. 38. but still… the AI promise is so tantalizing... MORE CAPABLE • Test your entire funnel in a single experiment ADAPTIVE • Evolves with your audience • Always-on optimization GREATER IMPACT • More tests = better results • Focus on marketing, not test administration FASTER AND MORE EFFICIENT • Tests dozens of ideas • Test 1,000,000s of combinations • Get to results faster TIME-SAVING • Fully automated • Eliminates tedious test administration • Handles the data science
  39. 39. © Randstad 42 AI-POWERED CONVERSION OPTIMIZATION
  40. 40. “Fractional Factorial MVT + Evolution”
  41. 41. MVT...
  42. 42. … + evolution
  43. 43. How the “evolution” part works... + 0 - + - + 0 + +
  44. 44. How the “evolution” part works... + 0 - + - + 0 + +
  45. 45. How the “evolution” part works...
  46. 46. How the “evolution” part works...
  47. 47. How the “evolution” part works...
  48. 48. How the “evolution” part works...
  49. 49. Evolutionary Algorithms
  50. 50. Evolutionary Algorithms
  51. 51. Evolutionary Algorithms
  52. 52. Evolutionary Algorithms
  53. 53. Evolutionary Algorithms
  54. 54. Evolutionary Algorithms
  55. 55. Evolutionary Algorithms + 8.7% PERFORMANCE OF BEST- PERFORMING CANDIDATE (VS. CONTROL) +4.5% +5.3% +6% … …
  56. 56. Multipage Funnel Optimization Home PDP Cart
  57. 57. Sentient Ascend Tooling: Implementation partner:
  58. 58. $143M in total funding 9 YEARS platform & technology development 35 PATENTS submitted to date; 11 patents issued 100+ EMPLOYEES HQ in San Francisco with offices in San Jose and Hong Kong Founded by the team that developed the core technology behind Siri
  59. 59. Sentient Ascend basic principles DEEP LEARNING BAYESIAN EVOLUTIONARY ALGORITHMS NEUROEVOLUTION
  60. 60. Tested in multiple industries FINANCE HEALTH CARE INSURANCE CYBERSECURITY AGRICULTURE
  61. 61. BUT... does it work for E-commerce…? © Randstad
  62. 62. Current brands experimenting with Sentient.ai ßeta
  63. 63. Sentient E- commerce ßeta © Randstad
  64. 64. Countries selected for trial The Netherlands Norway Sweden
  65. 65. 8 elements on PDP, 2 variants each ● Header - less content ● Image position - left to right ● Add progress bar - feed forward ● Social sentence (under CTA, Hobson’s choice +1) ● Product info - full width / less prominent ● Price partitioning ● USP Bar icons/text - different content ● USP Bar position - to the top
  66. 66. || our main questions • Does it work? • How much faster is it? • Is it worth it? © Randstad 2017
  67. 67. (1st) baseline active: end of Feb Live: March 20th End: June 3rd
  68. 68. NL 3 generations, 23 variants Sample size Full factorial MVT requires 1M users Sentient tested 125K users (12,5%)
  69. 69. NL - Time Series
  70. 70. NL - Best 20 variants Best Improvement over control: 2.5%-point (12.7%)
  71. 71. Significant? Nope... Too many variants/ Not enough visitors...
  72. 72. NO 4 generations, 28 variants Sample size Full factorial MVT requires 444K users Sentient tested 75K users (16,8%)
  73. 73. NO - Time Series
  74. 74. NO - Best 20 variants Best Improvement over control: 4.6%-point (12.6%)
  75. 75. Significant? Nope... Too many variants/ Not enough visitors...
  76. 76. SE 4 generations, 28 variants Sample size Full factorial MVT requires 1.2M users Sentient tested 351K users (29,3%)
  77. 77. SE - Time Series
  78. 78. SE - Best 20 variants Best Improvement over control: 2.8%-point (6.9%)
  79. 79. Significant? Yes! @ CI 95%, not @ CI 99%
  80. 80. Best variants NL Variant 8, generation 1 ● USP Bar Icons/Text SE Variant 26, generation 4 ● USP Bar Icons/Text ● Price perception ● Product info ● Call to action ● Progress Bar NO Variant 8, generation 1 ● USP Bar Icons/Text
  81. 81. Control Winning Variation
  82. 82. But can we trust the numbers from Sentient? © Randstad
  83. 83. A/B test the winner to validate Significant (frequentist; 90% confidence; 1-sided) ● Add-to-cart uplift: 3.9% (p-value 0.0161, Power: 95,9%) ● Conversion rate uplift: 5,3% (p-value 0.0361, Power 89,6%) Not significant ● Average Order Value
  84. 84. How much does it add to our bottom line?
  85. 85. Business case: ● Costs: 165K (subscription + support + salary) ● Expected uplift: 500K-1M profit ● Minimal CR uplift needed for break even: 1.25%
  86. 86. Final Thoughts © Randstad
  87. 87. ● Steered on CR, not on RPU ● GA data issues ● Sentient dashboard UX issues ● Calculator to determine ideal values and estimated runtime would be very helpful Specific Sentient caveats/ needed improvements to ßeta
  88. 88. Pro/Cons: Regular A/B/MvT testing Pro: ● More control ● Stricter hypothesis testing ● ... Con: ● Slow ● Less variants ● Needs a lot of traffic ● Limited test capacity ● ...
  89. 89. Pro/Cons: Testing with AI Pro: ● “Time per tested variant” much shorter ● End up with “uncommon” variation ● Requires less traffic (around 75% less) ● Higher ROI ● ... Con: ● More extensive setup process ● Black box (?) ● Don’t know exactly how fast generations will follow... ● Garbage in, Garbage out ● Completely freaks out our local Ecom managers. ● ...
  90. 90. “I just followed instructions” ‘Compu’er says no’
  91. 91. || our main questions • Does it work? • How much faster is it? • Is it worth it? © Randstad 2017
  92. 92. Did Sentient deliver on their promise...? MORE CAPABLE • Test your entire funnel in a single experiment ADAPTIVE • Evolves with your audience • Always-on optimization GREATER IMPACT • More tests = better results • Focus on marketing, not test administration FASTER AND MORE EFFICIENT • Tests dozens of ideas • Test 1,000,000s of combinations • Get to results faster TIME-SAVING • Fully automated • Eliminates tedious test administration • Handles the data science
  93. 93. © Randstad | will this make the CRO team obsolete? © Randstad
  94. 94. | well... ..I’m hiring :) © Randstad
  95. 95. Link to all slides: twitter.com/guido © Randstad

×