Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse

191 views

Published on

Harnessing the power of data and experimentation is central to Updater’s goal of revolutionizing the moving industry. However, while scaling their experimentation program, the team at Updater had to overcome the challenge that many of their conversions happen offline or on third-party sites and therefore couldn't be used in testing. During this session, you’ll learn how Updater fixed this blind spot in their funnels, tripled experiment velocity, and discovered deep user insights by integrating their experimentation platform and data warehouse.

Published in: Business
  • Be the first to comment

  • Be the first to like this

Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse

  1. 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
  2. 2. 2  Sessions today will be recorded and will be available after Opticon  Join the conversation on Twitter at #Opticon19  Like what you’ve seen today? Give feedback and rate sessions on the mobile app
  3. 3. Chris Byington Director of Analytics, Updater Dan Storms VP of Product, Updater Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
  4. 4. 4 Agenda 1. Updater & experimentation 2. A tale of two experimentation programs 3. Updater’s challenge 4. Our solution 5. Example outcomes 6. Conclusion & key takeaways 7. Q & A
  5. 5. 5 Build technology that is so good, people won’t move without it. • Revolutionizing the way 40 million people move every year in the U.S. • Offering new and innovative solutions to make relocation more efficient for everyone
  6. 6. 6 Updater’s business model Create value by bridging the communication gap Mover BusinessUpdater Save time, money, headache, & backache Reach consumers at a crucial behavior change moment
  7. 7. 7 Experimentation at Updater Learn Drive Release how to solve problems for our users revenue features with confidence The goal is to…
  8. 8. 8 Experimentation at Updater Crank the Flywheel • Faster experimentation process • Quicker feature development • More feature releases
  9. 9. 9 — Jeff Bezos Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.
  10. 10. 10 The blind spot Our true outcome metrics often occur off-platform, creating a blind spot in our funnels Off-platform metrics
  11. 11. 11
  12. 12. 12 • Weak off-platform transaction tracking • Slow experiments (if done at all) • Shallow user insights • Quick experimentation • Higher confidence experiments • Deeper user insights
  13. 13. 13 The dream… • Experiment more quickly • Experiment with higher confidence • Gain deeper user insights The reality • Bottlenecks in the funnel • Lack of visibility into what users are doing off-platform
  14. 14. 14 Experiment bottlenecks and lack of visibility Example: Setting Up TV & Internet Sign Up Enter TV/Internet Flow Answer Preference Questions Browse Plans Make a Reservation Call the Call Center Schedule Installation Installatio n Occurs On Platform Off-Platform Track and experiment No tracking
  15. 15. 15 Major hurdles Long evaluation cycle Low confidence in results Loss in team productivity
  16. 16. 16 Can it be better?The Challenge Analyst hours to evaluate 4 hours Elapsed time to evaluate 3 days Ticket required to evaluate Yes Lines of SQL needed ~35 Medium of evaluation MS Excel Ability to self-serve results No Maximum confidence level 50% Maximum experiment riskiness Medium Experimentation Process Effectiveness
  17. 17. 17 Our solution Identify the problem Combine forces Work with 3rd parties Enable feed into data platform Integrate data platform with Optimizely 1 2 3 4 5
  18. 18. 18 Our solution — Step 1 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
  19. 19. 19 Our solution — Step 2 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
  20. 20. 20 Our solution — Step 3 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
  21. 21. 21 Our solution — Step 4 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
  22. 22. 22 Our solution — Step 5 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
  23. 23. 23 Data platform Source Systems Extract & Load Data Lake & Warehouse Data Consumption & Informed Team Members UserExperienceSaaS Event Data Off-Platform Data Application DBs EventBus DataStoreOrchestration Snapshots Denormalized Tables Structured Data Business Intelligence Business Partners
  24. 24. 24 Simplified process
  25. 25. 25 Outcomes We’ve unlocked the ability to… Run Have Automate Gain many more experiments higher confidence a manual process deeper user insights
  26. 26. 26 Experimentation Process Effectiveness The Challenge The Solution Analyst hours to evaluate 4 hours 0 hours Elapsed time to evaluate 3 days 0 days Ticket required to evaluate Yes No Lines of SQL needed ~35 0 Medium of evaluation MS Excel Optimizely Ability to self-serve results No Yes Maximum confidence level 50% >99% Maximum experiment riskiness Medium Very high
  27. 27. 27 Experiment velocity gains Number of Experiments Launched per Month Number of Experiments Aug 2018 Sep 2018 Oct 2018 Nov 2018 Dec 2018 Jan 2019 Feb 2019 Mar 2019 Apr 2019 May 2019 Jun 2019 Optimizely Data Platform Integration
  28. 28. 28 Results A successful experiment Offline booking of TV and internet services Control: Plan Summary Only 15% Variant: Show Additional Plan Details off-platform Flat on platform
  29. 29. 29 Learning what doesn’t work A shorter user workflow Control: Include a Reservation Step Variant: Skip Reservation Page Results 23% conversion impact
  30. 30. 30 Experiment Runtime Syncing experiment results from Optimizely Analyzing a detailed funnel of user behavior… while the test is still running Sync user-variant mapping data Further optimization based on learnings Control: Book a New Service Variant: Transfer an Existing Service Extract experiment data Results 67% UX result improvement By 2/3 decrease in poor experience
  31. 31. 31 Experimentation velocity is crucial to success.So act like it.
  32. 32. 32 There’s power in combining forces Data + Product =
  33. 33. Thank you!
  34. 34. Up Next…  Head to the Expo Hall for a refreshment break  Fireside Chat with Ashton Kutcher at 4pm!  Happy Hour in the Expo Hall at 4:45pm

×