Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modernize 2018: Driving Innovation Through a Culture of Experimentation, VIP Breakfast - Optimizely

591 views

Published on

Watch Modernize 2018's VIP breakfast presentation by Optimizely’s Global Head of Strategy, Hazjier Pourkhalkhali.

Hazjier covers the latest experimentation trends from around the world and shares how companies like Nike, the BBC, and H&M are building experimentation into standard operating procedures - unlocking innovation across their companies.

ABOUT MODERNIZE 2018:
Be a part of the future. Shape the customer experience with Accenture, Forrester, Qantas, REA, Intrepid and more -- and learn how leading organisations are harnessing their people, processes, and technologies to exceed customer’s expectations.

Visit: https://www.optimizely.com/anz/modernize2018

Published in: Technology
  • I'd advise you to use this service: ⇒ www.HelpWriting.net ⇐ The price of your order will depend on the deadline and type of paper (e.g. bachelor, undergraduate etc). The more time you have before the deadline - the less price of the order you will have. Thus, this service offers high-quality essays at the optimal price.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Modernize 2018: Driving Innovation Through a Culture of Experimentation, VIP Breakfast - Optimizely

  1. 1. D R I V I N G I N N O V A T I O N T H R O U G H A C U L T U R E O F E X P E R I M E N T A T I O N
  2. 2. 40% 19% 10% 7% 5% 4% Rest Market share Alexa 10K Websites We are the world’s largest digital laboratory
  3. 3. EXPERIMENTATION PLATFORM #1 FORTUNE-100 CLIENTS 27 EXPERIMENT PARTICIPANTS PER YEAR 78Billion
  4. 4. “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” J E F F B E Z O S
  5. 5. 2010 1 TERAFLOP / s 100 GIGAFLOP / s 10 GIGAFLOP / s 1 GIGAFLOP / s 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 234X
  6. 6. 2010 1 TERAFLOP / s 1 GIGAFLOP / s 2020 2030 2040 2050 2060 1 EXAFLOP / s 1 PETAFLOP / s 1 ZETTAFLOP / s 2070 MOORE’S LAW SLOWDOWN
  7. 7. 2010 1 TERAFLOP / s 1 GIGAFLOP / s 2020 2030 2040 2050 2060 1 EXAFLOP / s 1 PETAFLOP / s 1 ZETTAFLOP / s 2070 2021 2022 2026 2027 2044 2048 2046 2049 2062
  8. 8. Insights-Driven Businesses CULTURE OF CONTINUOUS EXPERIMENTATION AND LEARNING GROW 8X FASTER THAN GLOBAL GDP EARN $1.8 TRILLION BY 2021
  9. 9. Insights-Driven Businesses Will Outperform Global GDP by 8X 2016 2017 2018 2019 2020 2021 $1.8 Trillion SOURCE: Forrester, Economic Intelligence Unit, Morningstar, PitchBook $400 Billion VC-Funded Startups Insights-driven public companies
  10. 10. Building A Culture of Experimentation ▪ Focus on the experience ▪ Relentlessly improve ▪ Set goals, not deliverables ▪ Set course via experiments ▪ Attract key talent ▪ Invest in engineering SOLVE REAL PAIN POINTS ADOPT AN AGILE ROADMAP INVEST IN CAPABILITIES
  11. 11. Building A Culture of Experimentation ▪ Focus on the experience ▪ Relentlessly improve SOLVE REAL PAIN POINTS
  12. 12. “People’s attitudes and feelings about losses and gains are really not symmetric. We really feel more pain when we lose $10,000 than we feel pleasure when we get $10,000.” D A N I E L K A H N E M A N N O B E L - P R I Z E 2 0 0 2
  13. 13. ORIGINAL $20 GIFT CARD VARIATION NO GIFT CARD
  14. 14. 43.4% MORE REVENUE PER VISITOR
  15. 15. §Pre-order distracts from products §Only valuable if people purchase a second product §Without a follow-up purchase in mind, pre-order feels like losing $20
  16. 16. Problem Solution Result All great experiments begin with a customer problem, not a solution
  17. 17. Problem Solution Result § Identify the user’s pain points § Validate pain points through research § Prioritize the most urgent pain points § Identify 10 solutions § Problem solve on the most critical to test § Select 3-4 variations § Decide the primary metrics § Choose secondary metrics § Select monitoring goals Problem-Centric Hypothesis Framework
  18. 18. S T E F A N T H O M K E H A R V A R D R O N K O H A V I M I C R O S O F T
  19. 19. R O N K O H A V I M I C R O S O F T §Vague headlines §No CTA in headline §Unable to engage brand if unsure, in need of information Problem
  20. 20. ORIGINAL VARIATION
  21. 21. An analysis showed that the change had increased revenue by an astonishing 12% — which on an annual basis would come to more than $100 million in the United States alone — without hurting key user-experience metrics.
  22. 22. Building A Culture of Experimentation ▪ Set goals, not deliverables ▪ Set course via experiments ADOPT AN AGILE ROADMAP
  23. 23. There is a big problem with “A/B Testing” @Hazjier
  24. 24. 2 Variations 4 Variations 5 Variations >6 Variations 3 Variations A B C D E F @Hazjier
  25. 25. Three-fourths of all experiments only have 2 variations 2 Variations 3 Variations 4 Variations 5 Variations >6 Variations 77% 14% 5% 2% 1% @Hazjier
  26. 26. Testing 5 or more variations can improve your win rate by 75% +71% +75% +48% +32% 25% 33% 37% 44% 43% +75% 2 Variations 3 Variations 4 Variations 5 Variations >6 Variations Significant uplift Significant reduction Inconclusive @Hazjier
  27. 27. We can say with statistical certainty that more variations is better! 18,956 75,055 4,619 13,869 1,815 4,859 1,005 2,275 566 1,310 >99% winner >99% winner >99% winner >99% winner NUMBER OF VARIATIONS STATISTICALLY SIGNIFICANT UPLIFTS EXPERIMENTS EXPERIMENT WIN RATE IMPROVEMENT STATISTICAL SIGNIFICANCE STATUS 5 Variations is currently beating 2 Variations by 75% BEHAVIOR Testing More Variations The percentage of experiments that reached a statistically significant uplift +71.1% +47.9% +31.9% 25.3% 33.3% 37.4% 44.2% 43.2% +74.9% 2 Variations 3 Variations 4 Variations 5 Variations >6 Variations @Hazjier
  28. 28. Building A Culture of Experimentation ▪ Attract key talent ▪ Invest in engineering INVEST IN CAPABILITIES
  29. 29. We analyzed a quarter million variations by the number of lines of code @Hazjier
  30. 30. 8% 92% Over half of all variations are written with 5 or fewer lines of code 1-5 LOC 6-10 LOC 11-20 LOC 21-50 LOC 51-100 LOC 101-200 LOC >200 LOC 51% 11% 10% 10% 4% 2% 2% @Hazjier Significant uplift Significant reduction Inconclusive
  31. 31. Hire a developer! 1-5 LOC 6-10 LOC 11-20 LOC 21-50 LOC 51-100 LOC 101-200 LOC >200 LOC 17% 21% 25% 29% 35% 37% 36% +112% +106% +118% Hire a developer! Complex variations win 2.1–2.2x more often! @Hazjier Significant uplift Significant reduction Inconclusive
  32. 32. Building A Culture of Experimentation ▪ Focus on the experience ▪ Relentlessly improve ▪ Set goals, not deliverables ▪ Set course via experiments ▪ Attract key talent ▪ Invest in engineering SOLVE REAL PAIN POINTS ADOPT AN AGILE ROADMAP INVEST IN CAPABILITIES
  33. 33. CULTURE OF EXPERIMENTATION A/B TESTING
  34. 34. D R I V I N G I N N O V A T I O N T H R O U G H A C U L T U R E O F E X P E R I M E N T A T I O N H A Z J I E R P O U R K H A L K H A L I G L O B A L H E A D O F S T R A T E G Y

×