Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Euroflorist is preparing for Artificial Intelligence

1,366 views

Published on

My slides from Emerce Conversion where I share Euroflorists first steps into an optimization process with evolutionary artificial intelligence algorithms.

Published in: Data & Analytics
  • Be the first to comment

How Euroflorist is preparing for Artificial Intelligence

  1. 1. How Euroflorist is preparing for AI: Changes to our optimization strategies. euroflorist.com
  2. 2. There will be Data (& GIFs)
  3. 3. How can we make the next step?
  4. 4. How can we maximize output (in terms of learnings and conversion uplift) with fixed resources?
  5. 5. Faster optimization loops!
  6. 6. Conversion Optimization is hard!
  7. 7. Optimizing for “the average user” Who is (s)he anyway?
  8. 8. Need more resources! (but who doesn’t…?)
  9. 9. Too many variables impacting conversion! ● Customer persona’s ● Countries ● Budgets ● Customer history ● Regulations ● Colors ● Flower seasons ● Product types ● Holiday peaks ● Time of day/week ● Purchasing power ● Culture ● Devices ● Social media platforms ● Acquisition channels ● Symbols ● Step in Customer Journey ● Tech adoption level ● Accessoires ● Multi-channel funnels ● Etiquette ● Consumer vs Corporate ● Platform ● Different payment systems ● Brand awareness/perception ● Logistic situation ● Aesthetics ● Traditions ● Social context ● Weather ● Competition campaigns ● Consumer state-of-mind ● Competitor A/B test ● ...
  10. 10. And to make it even worse...
  11. 11. These variables keep changing all the ffing time!
  12. 12. Let’s take a step back...
  13. 13. Why does Euroflorist exist?
  14. 14. We help people to make others happy, noticed or remembered c
  15. 15. Who are we?
  16. 16. Who is Euroflorist? ● We sell flowers & others gifts ● We’ve been doing this since 1947 ● We do B2C & B2B, online and offline ● We are a Swedish company ● Our Dutch office is located in Amsterdam
  17. 17. Euroflorist in numbers (1/2) ● > 21 years in e-commerce ● > 60% revenue from online ● > 200 employees: 50% in CS, 35 working in online ● > 2M bouquets sent every year ● 19 sites in 11 countries and 8 languages
  18. 18. Euroflorist in numbers (2/2) ● 10M+ users on our sites in 2017 ● Sites with up to 20% conversion rate ● Market share ranging from 5-45% ● 8+ score on Trustpilot, NPS > 50 ● Growing revenue, teams and taking market share
  19. 19. Guido Jansen ● Chief Psychology Officer @ Euroflorist ● UX/CX Team Lead ● Full Customer Journey optimization ● Wannabee Data Scientist x@gui.do | @guido
  20. 20. What are we doing now (with CRO)?
  21. 21. Customer journey teams ● Branding/Marketing ● Traffic ● Landing pages ● PDP ● Product ● Checkout ● Post-order experience ● Delivery ● Customer Support ● E-mail / Retention 1 Business owner + 1 UX/CX team member
  22. 22. Live Testing Complete/Analysis Deployments Design Development Q&A Pending approval Approved Idea Hypotheses Full testing plan Our optimization loop
  23. 23. How can we make the next step?
  24. 24. How are others doing this?
  25. 25. “[Booking.com’s] utilization of A/B testing … drives higher conversion across it’s entire platform, resulting in conversion levels 2-3x the industry average” Source: Evercore Equity Research
  26. 26. Sadly, when you test more, the % of tests with an uplift will go down.
  27. 27. When A/B testing a fully optimized site, the % of tests with an uplift is low(er).
  28. 28. Euroflorist: 1 in 5 tests with uplift Booking.com: 1 in 10 tests with uplift Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017
  29. 29. How to get more tests with an uplift?
  30. 30. Euroflorist: 10-20 concurrent test Booking.com: 1000 concurrent tests Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017
  31. 31. Euroflorist: 1 optimization team Booking.com: 75 product teams Booking.com Source: Stuart Frisby, Booking.com - Conversions@Google 2017
  32. 32. Of course, we are not Booking.com... !=
  33. 33. ...but we do know a thing (or 2) about experiments...
  34. 34. ...so let’s experiment with the way we experiment!
  35. 35. Let’s start looking for a solution...
  36. 36. Evolutionary AI? Tell me more!
  37. 37. Possible applications
  38. 38. Different applications of AI in E-commerce ● Catalog optimization ← already tried this ● Personalization ● RTB / Advertising ● Chatbots ● Virtual personal shopper ● Improved optimization process ● ...
  39. 39. We tried AI catalog optimization
  40. 40. Previous experience: ● “Black box” ● No (perceived) control for E-commerce managers ● No proven uplift in CR ● Hard to do for (emotional) consumers buying for someone else?
  41. 41. The AI promise... ● Faster & more efficient ● Time-saving ● Test more ● Adaptive ● Greater impact ● ...
  42. 42. Sentient
  43. 43. “Fractional Factorial MVT + Evolution”
  44. 44. MVT...
  45. 45. … + Evolution
  46. 46. How the “evolution” part works... + 0 - + - + 0 + +
  47. 47. How the “evolution” part works... + 0 - + - + 0 + +
  48. 48. How the “evolution” part works...
  49. 49. How the “evolution” part works...
  50. 50. How the “evolution” part works...
  51. 51. How the “evolution” part works...
  52. 52. Always-on continuous optimization
  53. 53. Multipage Funnel Optimization Home PDP Cart
  54. 54. Sentient Ascend Tooling: Implementation partner:
  55. 55. $143M in total funding 9 YEARS platform & technology development 35 PATENTS submitted to date; 11 patents issued 100+ EMPLOYEES HQ in San Francisco with offices in San Jose and Hong Kong Founded by the team that developed the core technology behind Siri
  56. 56. Sentient Ascend DEEP LEARNING BAYESIAN EVOLUTIONARY ALGORITHMS NEUROEVOLUTION
  57. 57. Tested in multiple industries FINANCE HEALTH CARE INSURANCE CYBERSECURITY AGRICULTURE
  58. 58. But does it work for E-commerce…?
  59. 59. Current brands experimenting with Sentient.ai ßeta
  60. 60. Sentient E-commerce ßeta @ Euroflorist
  61. 61. Countries selected for trial The Netherlands Norway Sweden
  62. 62. 8 elements on PDP, 2 variants each ● Header - less content ● Image position - left to right ● Add progress bar - feed forward ● Social sentence (under CTA, Hobson’s choice +1) ● Product info - full width / less prominent ● Price partitioning ● USP Bar icons/text - different content ● USP Bar position - to the top
  63. 63. 28 = 256 variants
  64. 64. (1st) baseline active: end of Feb Live: March 20th End: June 3rd
  65. 65. NL 3 generations, 23 variants Sample size Full factorial MVT requires 1M users Sentient tested 125K users
  66. 66. NL - Time Series
  67. 67. NL - Best 20 variants Best Improvement over control: 2.5%-point (12.7%)
  68. 68. Significant? Nope... Too many variants/ Not enough visitors...
  69. 69. NO 4 generations, 28 variants Sample size Full factorial MVT requires 444K users Sentient tested 75K users
  70. 70. NO - Time Series
  71. 71. NO - Best 20 variants Best Improvement over control: 4.6%-point (12.6%)
  72. 72. Significant? Nope... Too many variants/ Not enough visitors...
  73. 73. SE 4 generations, 28 variants Sample size Full factorial MVT requires 1.2M users Sentient tested 351K users
  74. 74. SE - Time Series
  75. 75. SE - Best 20 variants Best Improvement over control: 2.8%-point (6.9%)
  76. 76. Significant? Yes! @ CI 95%, not @ CI 99%
  77. 77. Best variants NL Variant 8, generation 1 ● USP Bar Icons/Text SE Variant 26, generation 4 ● USP Bar Icons/Text ● Price perception ● Product info ● Call to action ● Progress Bar NO Variant 8, generation 1 ● USP Bar Icons/Text
  78. 78. Best variant: Common best element: USP Bar Icons/Text Baseline:
  79. 79. But can we trust the numbers?
  80. 80. A/B test the winner to validate Significant (frequentist; 90% confidence; 1-sided) ● Add-to-cart uplift: 3.9% (p-value 0.0161, Power: 95,9%) ● Conversion rate uplift: 5,3% (p-value 0.0361, Power 89,6%) Not significant ● Average Order Value
  81. 81. How much does it add to our bottom line? € € € € € € € € € €
  82. 82. Business case: ● Costs: 165K (subscription + support + salary) ● Expected uplift: 500K-1M profit ● Minimal CR uplift needed for break even: CR increase of 1.25%
  83. 83. Final thoughts
  84. 84. ● Steered on CRT, not on RPU ● GA data issues ● Sentient dashboard UX issues ● Calculator to determine ideal values would be very helpful Specific Sentient caveats/ needed improvements to ßeta
  85. 85. Regular A/B/MvT testing Pro: ● More control ● Stricter hypothesis testing ● ... Con: ● Slow ● Less variants ● Needs a lot of traffic ● Limited test capacity ● ...
  86. 86. Testing with AI Pro: ● “Time per tested variant” much shorter ● End up with “uncommon” variation ● Requires less traffic (around 75% less) ● Higher ROI ● ... Con: ● More extensive setup process ● Black box (?) ● Don’t know exactly how fast generations will follow... ● Garbage in, Garbage out ● Completely freaks out our local Ecom managers. ● ...
  87. 87. “I just followed instructions” ‘Compu’er says no’
  88. 88. Did Sentient deliver on their promise...? MORE CAPABLE • Test your entire funnel in a single experiment ADAPTIVE • Evolves with your audience • Always-on optimization GREATER IMPACT • More tests = better results • Focus on marketing, not test administration FASTER AND MORE EFFICIENT • Tests dozens of ideas • Test 1,000,000s of combinations • Get to results faster TIME-SAVING • Fully automated • Eliminates tedious test administration • Handles the data science
  89. 89. Will this make the CRO team obsolete?
  90. 90. We’re hiring! Optimization team: ● Web Analysts ● UX designers ● Growth hackers ● Interns ● AI/CKI background ● (Cognitive/Behavioral) Psychologists Other online: ● SEA Specialist ● Marketing & communications specialist ● CRM specialist ● Social Media Manager ● Interns ● ... Are you smart & curious? Looking for a growing international company where you can experiment & learn a lot? Join Euroflorist! euroflorist.com
  91. 91. THX! x@gui.do | @guido

×