Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
•Download as PPTX, PDF•
0 likes•510 views
Harnessing the power of data and experimentation is central to Updater’s goal of revolutionizing the moving industry. However, while scaling their experimentation program, the team at Updater had to overcome the challenge that many of their conversions happen offline or on third-party sites and therefore couldn't be used in testing. During this session, you’ll learn how Updater fixed this blind spot in their funnels, tripled experiment velocity, and discovered deep user insights by integrating their experimentation platform and data warehouse.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
Similar to Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse (20)
More from Optimizely
More from Optimizely (20)
Recently uploaded
Recently uploaded (20)
Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
- 1. Congratulations to our Outperform Award Winners! Most Dramatic Business Impact Most Customer- Obsessed Company Culture Most Transformative Innovation Most Inspiring Social Impact
- 2. 2 Sessions today will be recorded and will be available after Opticon Join the conversation on Twitter at #Opticon19 Like what you’ve seen today? Give feedback and rate sessions on the mobile app
- 3. Chris Byington Director of Analytics, Updater Dan Storms VP of Product, Updater Triple Your Experiment Velocity by Integrating Optimizely with Your Data Warehouse
- 4. 4 Agenda 1. Updater & experimentation 2. A tale of two experimentation programs 3. Updater’s challenge 4. Our solution 5. Example outcomes 6. Conclusion & key takeaways 7. Q & A
- 5. 5 Build technology that is so good, people won’t move without it. • Revolutionizing the way 40 million people move every year in the U.S. • Offering new and innovative solutions to make relocation more efficient for everyone
- 6. 6 Updater’s business model Create value by bridging the communication gap Mover BusinessUpdater Save time, money, headache, & backache Reach consumers at a crucial behavior change moment
- 7. 7 Experimentation at Updater Learn Drive Release how to solve problems for our users revenue features with confidence The goal is to…
- 8. 8 Experimentation at Updater Crank the Flywheel • Faster experimentation process • Quicker feature development • More feature releases
- 9. 9 — Jeff Bezos Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.
- 10. 10 The blind spot Our true outcome metrics often occur off-platform, creating a blind spot in our funnels Off-platform metrics
- 11. 11
- 12. 12 • Weak off-platform transaction tracking • Slow experiments (if done at all) • Shallow user insights • Quick experimentation • Higher confidence experiments • Deeper user insights
- 13. 13 The dream… • Experiment more quickly • Experiment with higher confidence • Gain deeper user insights The reality • Bottlenecks in the funnel • Lack of visibility into what users are doing off-platform
- 14. 14 Experiment bottlenecks and lack of visibility Example: Setting Up TV & Internet Sign Up Enter TV/Internet Flow Answer Preference Questions Browse Plans Make a Reservation Call the Call Center Schedule Installation Installatio n Occurs On Platform Off-Platform Track and experiment No tracking
- 15. 15 Major hurdles Long evaluation cycle Low confidence in results Loss in team productivity
- 16. 16 Can it be better?The Challenge Analyst hours to evaluate 4 hours Elapsed time to evaluate 3 days Ticket required to evaluate Yes Lines of SQL needed ~35 Medium of evaluation MS Excel Ability to self-serve results No Maximum confidence level 50% Maximum experiment riskiness Medium Experimentation Process Effectiveness
- 17. 17 Our solution Identify the problem Combine forces Work with 3rd parties Enable feed into data platform Integrate data platform with Optimizely 1 2 3 4 5
- 18. 18 Our solution — Step 1 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 19. 19 Our solution — Step 2 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 20. 20 Our solution — Step 3 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 21. 21 Our solution — Step 4 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 22. 22 Our solution — Step 5 Identify Combine Work with Enable feed Integrate 1 2 3 4 5 the problem forces 3rd parties into data platform data platform with Optimizely
- 23. 23 Data platform Source Systems Extract & Load Data Lake & Warehouse Data Consumption & Informed Team Members UserExperienceSaaS Event Data Off-Platform Data Application DBs EventBus DataStoreOrchestration Snapshots Denormalized Tables Structured Data Business Intelligence Business Partners
- 25. 25 Outcomes We’ve unlocked the ability to… Run Have Automate Gain many more experiments higher confidence a manual process deeper user insights
- 26. 26 Experimentation Process Effectiveness The Challenge The Solution Analyst hours to evaluate 4 hours 0 hours Elapsed time to evaluate 3 days 0 days Ticket required to evaluate Yes No Lines of SQL needed ~35 0 Medium of evaluation MS Excel Optimizely Ability to self-serve results No Yes Maximum confidence level 50% >99% Maximum experiment riskiness Medium Very high
- 27. 27 Experiment velocity gains Number of Experiments Launched per Month Number of Experiments Aug 2018 Sep 2018 Oct 2018 Nov 2018 Dec 2018 Jan 2019 Feb 2019 Mar 2019 Apr 2019 May 2019 Jun 2019 Optimizely Data Platform Integration
- 28. 28 Results A successful experiment Offline booking of TV and internet services Control: Plan Summary Only 15% Variant: Show Additional Plan Details off-platform Flat on platform
- 29. 29 Learning what doesn’t work A shorter user workflow Control: Include a Reservation Step Variant: Skip Reservation Page Results 23% conversion impact
- 30. 30 Experiment Runtime Syncing experiment results from Optimizely Analyzing a detailed funnel of user behavior… while the test is still running Sync user-variant mapping data Further optimization based on learnings Control: Book a New Service Variant: Transfer an Existing Service Extract experiment data Results 67% UX result improvement By 2/3 decrease in poor experience
- 31. 31 Experimentation velocity is crucial to success.So act like it.
- 32. 32 There’s power in combining forces Data + Product =
- 33. Thank you!
- 34. Up Next… Head to the Expo Hall for a refreshment break Fireside Chat with Ashton Kutcher at 4pm! Happy Hour in the Expo Hall at 4:45pm
Editor's Notes
- We’ll show something here to highlight our business model to provide context for the presentation.
- Graphic representation of a flywheel… Making this process fast = making feature development fast = releasing more features ==> testing quickly is crucial to Updater’s success
- As a result, there’s real value in experiment velocity Be creative and thoughtful to improve it Aspire to drive a step-change increase in experimentation speed rather than optimizing around the edges
- a choice between the least worst of: Test using intermediate metrics that eroded confidence in test results & feature release Download user-variant lists for each experiment, request one-off reporting from 3rd party conversion sites, wait for them to send the reporting, and then tabulate experiment results by hand before evaluating experiments (took days to evaluate experiment once reached sample size)
- We wanted to experiment more quickly & with higher confidence and gain deeper insights about our users, but time and time again hit a bottleneck of not knowing what was happening off-platform -- blind spot in our funnels. This had a few consequences (consequences in the following slide):
- If our systems were integrated, we’d be able to treat offline conversions as first-class success metrics in Optimizely and learn and make decisions – with confidence – more quickly, allowing us to run riskier, higher-potential experiments, fail, and be more creative.
- Step 1. Identified the problem - Data team realized Experimentation speed is crucial to speed of product development It’s a bottleneck to not have offline conversion events in Optimizely in near-real time. Making this change will dramatically increase testing speed and confidence, and allow us to more deeply analyze user behavior in Data Platform to boot
- Step 2. Product & Data teams partner to vet & prioritize the project
- Step 3. Worked with third party partners to gain access to conversion event data feed
- Step 4. Data Engineering wrote ELT process to ingest this data feed into Data Platform
- Step 5. Take 3rd party conversion data in Data Platform & send it into Optimizely in a way that can be used for AB testing; get AB testing data back from Optimizely into our Data Platform so it can be used for deeper analysis
- Data Platform Core data store (Snowflake) Surrounding services Fivetran for ELT of stateful data Production databases 3rd party data like Salesforce, Iterable, etc. Consolidated event bus for all event data Apache Airflow for orchestration Optimizely
- Show the NEW user journey next, as it’s a result of the outcomes.
- Example: Displaying additional TV / Internet package info in-app, drove an increase in call center performance. Our internet booking product uses a call center to complete the transaction, and the funnel has two offline steps: calls to the call center and conversions. We made an in-product change to surface additional information about individual internet plans to the user, which we expected to increase conversion to the endpoint of our in-product experience. However, we saw a flat impact on in-product conversion. Thanks to having offline events in Optimizely in near real time, we were surprised to see a 15% increase in call-to-conversion rate, meaning a 15% lift to revenue for this product overall! But for this data integration, we would have written this experiment off as a loser and would not have known this product change helped our users and revenue at the same time.
- Example: removing the last page in our in-app user workflow for TV / Internet doesn’t help our users connect with our call center at a higher rate, even though there are fewer in-app steps. We decided to test removing the page in our user workflow where the user can view which internet package they reserved, the hypothesis being fewer pages would make it easier for users to get to the endpoint and convert. We saw the opposite, i.e. a negative impact on end-conversions. Thanks to being able to see end-conversion events in Optimizely, we were able to see that our treatment variant was performing worse than control well before our test reached its target sample size and turn the test off, thereby significantly limiting the negative impact the test would have had on our overall conversion rate.
- Example: analyzing a detailed funnel of user behavior in an experiment while the test is still running Again for our internet booking product, we tested adding an entirely new user workflow in parallel to our current workflow - one to book new service, one to transfer existing service Each workflow ends with a call to our call center, so in Optimizely we were quickly able to see that giving users the option to transfer resulted in more calls - this was great because it gave users something they needed while also generating revenue Never satisfied with our own success, while the test was running, using user-variant mapping data synced into Snowflake from Optimizely we analyzed each step of both funnels to more deeply understand how users were performing within each funnel We quickly identified an area for further optimization: transfer users calling our call center were converting at about ½ the rate that other users were. Armed with this insight, we collaborated with our call center to understand the driver of this lower conversion rate, and released optimizations to the product which significantly increased conversion rate from calls, further increasing revenue on top of the lift we saw in our initial experiment. Specifically, we made one particularly impactful discovery: Many users calling our call center were trying to transfer with a provider which wasn’t available at their new home, meaning that transfer was not possible and our alternative new service workflow was better suited to these users. We used this information to more conspicuously surface available providers to users within our app, decreasing this false transfer attempt rate by ⅔
- Moving as quickly as possible from observation >> hypothesis >> test >> evaluation >> action will map to releasing better product experiences to users faster
- Importance of Data/Product teams working together/joining forces Marry business understanding with technical understanding Data is a crucial source of learning, so Product teams that value learning quickly can benefit from partnering closely with Data teams Creativity and enthusiasm for the world of data can have a massive impact. Don’t just think, “how do we make more revenue with this one experiment?” - think about the second- and third-order business-outcome-driving questions such as “how do we increase experiment velocity to impact overall revenue and UX?”