SlideShare a Scribd company logo
How The Zebra Utilized Feature
Experiments To Increase Carrier Card
Engagement by 10%
Megan Bubley
Sr. Product Manager
THE ZEBRA
2
The Zebra
We make insurance black and white.
The Zebra was founded with a single goal: to
simplify insurance, and in doing so we’ve
become the nation’s leading insurance
comparison site.
Our vision is to be a trusted partner for
consumers, helping them to better manage and
understand all their insurance needs through
every stage of life, even in the face of the
unexpected.
Carrier Card Redesign
Auto Results Page - Multivariate Feature Test
3
Our goal with this redesign was to rapidly test and learn. We
validated our design choices with usability tests but still had
questions around which combination of features would create
the best user experience and how performance would differ
based on device.
We wanted to test a combination of carrier card features but
were concerned about the time it would take to analyze test
performance and the amount of engineering bandwidth
required to introduce a new round of A/B tests.
In the past running a multivariate experiment this robust would
require weeks of engineering time and would drastically
increase our delivery timelines, however through the use of
feature testing we were able to test multiple variants with
limited engineering effort in a short amount of time!
Differentiation by Platform
Desktop vs. Mobile
4
For our first round of tests we wanted to focus on
information layout and creating consistency across
platforms so we could better assess how design
impacted user engagement.
We knew that historically desktop and mobile have
performed very differently in terms of engagement.
● Mobile typically performed worse when more
elements were added to carrier cards vs.
desktop.
● Since desktop carrier cards have more real
estate to utilize, more content was typically
added to these cards creating a very different
user experience across platforms.
5
Step One: Work with design and engineering to identify the carrier
card elements we want to test along with the number of variants
we wanted to start with.
Step Two: Create feature variables within Optimizely that allow us
to turn on or off elements on the carrier card so we can customize
the design.
Step Three: Once the test goes live, use real time data to monitor
test performance and come up with future variants we want to test
based on carrier card conversion rates.
Development Strategy
Feature Variables
6
Square Logo
Rectangle Logo
Star Ratings
Coverage Value Prop
First Month Pricing
Price in CTA
Carrier Taglines
List View
Card View
7
Running The Test
The most challenging part of A/B testing has typically
been saving space for optimizations, particularly when a
feature is testing and developers are no longer actively
working it.
In the past we have had issues with prioritizing future
optimizations based on how we think the test will
perform in order to keep the momentum going with
engineers.
With feature based testing we were able to make changes
to our test based on the data without any additional
engineering effort—essentially we had the ability to create
a custom carrier card without any engineering resources.
Example of the feature test setup used for the carrier card
redesign.
7
8
Variant Designs
Mobile
Monitoring Performance & Use of Feature Flags
Achieving Our Long Term Vision
9
While we had hoped to implement a similar experience across
desktop and mobile we noticed that different variants performed
better on different platforms—for example rectangular logos
performed better on mobile and square logos did better on desktop.
Since we wanted to come back to some of these findings and retest
at a later date we utilized the feature flag functionality once we had
a winning variant to promote that feature to 100% of traffic without
eating into our traffic allotment in Optimizely.
10
10%
Increase in carrier card engagement across platforms
60%
Reduction in engineering effort and development time
Final Outcomes
We ran the carrier card feature test for over 6 months and
worked closely with stakeholders to make changes to our
variants as our product changed.
By running this test as a feature experiment we were able
to implement changes to our variants without any
additional development effort—something that would have
taken us weeks to implement in the past.
We were also able to test different variants on mobile and
desktop which allowed us to differentiate our experience
by platform based on user behavior.
As a result of these tests we saw significant improvement
to carrier card engagement across platforms.
11
Benefits
1. Lots of flexibility when it comes to
testing multiple designs.
2. Ability to turn on and off feature
variables with the flip of a switch—no
engineering effort required.
3. Ability to move the experiment behind a
feature flag to reduce the strain on
traffic allotments
4. Ability to retest features at a later date —
once new features have been
launched—to understand full impact of
the final product on the user experience.
Challenges
1. Lots of frontloading required to identify
the different feature variables and
testing strategy.
2. Additional engineering effort required up
front in order to set up the feature test—
was more challenging to implement
than a straight A/B test.
3. Additional testing required to ensure all
feature variables display correctly in
production.
4. Multiple dashboards needed to record
performance and there was not a super
easy way to track performance in
Optimizely.
12
PRESENTATIO
N TITLE
Questions?
Contact Information:
Email: mbubley@thezebra.com
Linkedin: www.linkedin.com/in/megan-bubley

More Related Content

What's hot

Ahead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeAhead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance Edge
Optimizely
 
Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - Atlassian
Optimizely
 
Apply A/B Testing with NGINX Routing Policy
Apply A/B Testing with NGINX Routing PolicyApply A/B Testing with NGINX Routing Policy
Apply A/B Testing with NGINX Routing Policy
Supachai Jaturaprom
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
Optimizely
 
Experimentation Excellence: Mastering the basics
Experimentation Excellence: Mastering the basicsExperimentation Excellence: Mastering the basics
Experimentation Excellence: Mastering the basics
Optimizely
 
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
Optimizely
 
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at ScaleOpticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
Optimizely
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
Optimizely
 
The Future of Software Development
The Future of Software DevelopmentThe Future of Software Development
The Future of Software Development
Optimizely
 
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Optimizely
 
[Webinar] Innovate Faster by Adopting The Modern Growth Stack
[Webinar] Innovate Faster by Adopting The Modern Growth Stack[Webinar] Innovate Faster by Adopting The Modern Growth Stack
[Webinar] Innovate Faster by Adopting The Modern Growth Stack
Optimizely
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
Optimizely
 
Opticon 2017 How Developers Can Take Experimentation
Opticon 2017 How Developers Can Take ExperimentationOpticon 2017 How Developers Can Take Experimentation
Opticon 2017 How Developers Can Take Experimentation
Optimizely
 
Delivering Excellent Digital Quality to Your Customers
Delivering Excellent Digital Quality to Your CustomersDelivering Excellent Digital Quality to Your Customers
Delivering Excellent Digital Quality to Your Customers
Applause
 
Maximizing Optimizely Website Performance
Maximizing Optimizely Website PerformanceMaximizing Optimizely Website Performance
Maximizing Optimizely Website Performance
Optimizely
 
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
Optimizely
 
Tackling Strategic Engineering Challenges
Tackling Strategic Engineering ChallengesTackling Strategic Engineering Challenges
Tackling Strategic Engineering Challenges
Applause
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Optimizely
 
[Webinar] How Vivid Seats Ensures Experimentation Success
[Webinar] How Vivid Seats Ensures Experimentation Success [Webinar] How Vivid Seats Ensures Experimentation Success
[Webinar] How Vivid Seats Ensures Experimentation Success
Optimizely
 
Boost Your Intelligent Assistants with UX Testing
Boost Your Intelligent Assistants with UX TestingBoost Your Intelligent Assistants with UX Testing
Boost Your Intelligent Assistants with UX Testing
Applause
 

What's hot (20)

Ahead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance EdgeAhead of the Curve: How 23andMe Improved UX with Performance Edge
Ahead of the Curve: How 23andMe Improved UX with Performance Edge
 
Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - Atlassian
 
Apply A/B Testing with NGINX Routing Policy
Apply A/B Testing with NGINX Routing PolicyApply A/B Testing with NGINX Routing Policy
Apply A/B Testing with NGINX Routing Policy
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
 
Experimentation Excellence: Mastering the basics
Experimentation Excellence: Mastering the basicsExperimentation Excellence: Mastering the basics
Experimentation Excellence: Mastering the basics
 
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
 
Opticon 2017 Decisions at Scale
Opticon 2017 Decisions at ScaleOpticon 2017 Decisions at Scale
Opticon 2017 Decisions at Scale
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
 
The Future of Software Development
The Future of Software DevelopmentThe Future of Software Development
The Future of Software Development
 
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
 
[Webinar] Innovate Faster by Adopting The Modern Growth Stack
[Webinar] Innovate Faster by Adopting The Modern Growth Stack[Webinar] Innovate Faster by Adopting The Modern Growth Stack
[Webinar] Innovate Faster by Adopting The Modern Growth Stack
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Opticon 2017 How Developers Can Take Experimentation
Opticon 2017 How Developers Can Take ExperimentationOpticon 2017 How Developers Can Take Experimentation
Opticon 2017 How Developers Can Take Experimentation
 
Delivering Excellent Digital Quality to Your Customers
Delivering Excellent Digital Quality to Your CustomersDelivering Excellent Digital Quality to Your Customers
Delivering Excellent Digital Quality to Your Customers
 
Maximizing Optimizely Website Performance
Maximizing Optimizely Website PerformanceMaximizing Optimizely Website Performance
Maximizing Optimizely Website Performance
 
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
The Optimizely Experience Keynote by Matt Althauser - Optimizely Experience L...
 
Tackling Strategic Engineering Challenges
Tackling Strategic Engineering ChallengesTackling Strategic Engineering Challenges
Tackling Strategic Engineering Challenges
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
[Webinar] How Vivid Seats Ensures Experimentation Success
[Webinar] How Vivid Seats Ensures Experimentation Success [Webinar] How Vivid Seats Ensures Experimentation Success
[Webinar] How Vivid Seats Ensures Experimentation Success
 
Boost Your Intelligent Assistants with UX Testing
Boost Your Intelligent Assistants with UX TestingBoost Your Intelligent Assistants with UX Testing
Boost Your Intelligent Assistants with UX Testing
 

Similar to How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagement by 10%

Ajinth_Resume
Ajinth_ResumeAjinth_Resume
Ajinth_Resume
Ajinth Gabriel
 
Sandipsingh kandari resume
Sandipsingh kandari resume Sandipsingh kandari resume
Sandipsingh kandari resume
Sandip Kandari
 
User Stories Lunch & Learn
User Stories Lunch & LearnUser Stories Lunch & Learn
User Stories Lunch & Learn
Christopher Say Go
 
mohit's-resume
mohit's-resumemohit's-resume
mohit's-resume
Mohit Sharma
 
Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...
Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...
Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...
kalichargn70th171
 
Ashish Baraiya
Ashish BaraiyaAshish Baraiya
Ashish Baraiya
Ashish Baraiya
 
10 must do’s for perfect customer experience (Cx) -Qualitest
10 must do’s for perfect customer experience (Cx) -Qualitest10 must do’s for perfect customer experience (Cx) -Qualitest
10 must do’s for perfect customer experience (Cx) -Qualitest
Qualitest
 
Experimentation at Blue Apron (webinar)
Experimentation at Blue Apron (webinar)Experimentation at Blue Apron (webinar)
Experimentation at Blue Apron (webinar)
Optimizely
 
Testing Strategy To Test A Ticketing Platform
Testing Strategy To Test A Ticketing PlatformTesting Strategy To Test A Ticketing Platform
Testing Strategy To Test A Ticketing Platform
Knoldus Inc.
 
Resume_Mahesh
Resume_MaheshResume_Mahesh
Resume_Mahesh
Mahesh P
 
Vinit Ranjan_Tester
Vinit Ranjan_TesterVinit Ranjan_Tester
Vinit Ranjan_Tester
Vinit Ranjan
 
Ship Confidently with progressive delivery and experimentation.pdf
Ship Confidently with progressive delivery and experimentation.pdfShip Confidently with progressive delivery and experimentation.pdf
Ship Confidently with progressive delivery and experimentation.pdf
HeyEmbedMe
 
Resume_Pratheeshkumar
Resume_PratheeshkumarResume_Pratheeshkumar
Resume_Pratheeshkumar
pratheesh kumar
 
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdfimplementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
sarah david
 
Pravin_CV_4+years
Pravin_CV_4+yearsPravin_CV_4+years
Pravin_CV_4+years
Pravin Gurav
 
Agile case studies
Agile case studiesAgile case studies
Agile case studies
Sébastien Donné
 
Kartheeka CV
Kartheeka CVKartheeka CV
Kartheeka CV
Kartheeka Bandlamudi
 
Punith_CV
Punith_CVPunith_CV
Punith_CV
Puneeth k
 
Sunil_Personal
Sunil_PersonalSunil_Personal
Sunil_Personal
sunil kumar
 
Tahseen Experience Resume
Tahseen Experience  ResumeTahseen Experience  Resume
Tahseen Experience Resume
Syed Ali
 

Similar to How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagement by 10% (20)

Ajinth_Resume
Ajinth_ResumeAjinth_Resume
Ajinth_Resume
 
Sandipsingh kandari resume
Sandipsingh kandari resume Sandipsingh kandari resume
Sandipsingh kandari resume
 
User Stories Lunch & Learn
User Stories Lunch & LearnUser Stories Lunch & Learn
User Stories Lunch & Learn
 
mohit's-resume
mohit's-resumemohit's-resume
mohit's-resume
 
Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...
Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...
Automated Front End Testing_ Navigating Types and Tools for Optimal Web Devel...
 
Ashish Baraiya
Ashish BaraiyaAshish Baraiya
Ashish Baraiya
 
10 must do’s for perfect customer experience (Cx) -Qualitest
10 must do’s for perfect customer experience (Cx) -Qualitest10 must do’s for perfect customer experience (Cx) -Qualitest
10 must do’s for perfect customer experience (Cx) -Qualitest
 
Experimentation at Blue Apron (webinar)
Experimentation at Blue Apron (webinar)Experimentation at Blue Apron (webinar)
Experimentation at Blue Apron (webinar)
 
Testing Strategy To Test A Ticketing Platform
Testing Strategy To Test A Ticketing PlatformTesting Strategy To Test A Ticketing Platform
Testing Strategy To Test A Ticketing Platform
 
Resume_Mahesh
Resume_MaheshResume_Mahesh
Resume_Mahesh
 
Vinit Ranjan_Tester
Vinit Ranjan_TesterVinit Ranjan_Tester
Vinit Ranjan_Tester
 
Ship Confidently with progressive delivery and experimentation.pdf
Ship Confidently with progressive delivery and experimentation.pdfShip Confidently with progressive delivery and experimentation.pdf
Ship Confidently with progressive delivery and experimentation.pdf
 
Resume_Pratheeshkumar
Resume_PratheeshkumarResume_Pratheeshkumar
Resume_Pratheeshkumar
 
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdfimplementing_ai_for_improved_performance_testing_the_key_to_success.pdf
implementing_ai_for_improved_performance_testing_the_key_to_success.pdf
 
Pravin_CV_4+years
Pravin_CV_4+yearsPravin_CV_4+years
Pravin_CV_4+years
 
Agile case studies
Agile case studiesAgile case studies
Agile case studies
 
Kartheeka CV
Kartheeka CVKartheeka CV
Kartheeka CV
 
Punith_CV
Punith_CVPunith_CV
Punith_CV
 
Sunil_Personal
Sunil_PersonalSunil_Personal
Sunil_Personal
 
Tahseen Experience Resume
Tahseen Experience  ResumeTahseen Experience  Resume
Tahseen Experience Resume
 

More from Optimizely

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
Optimizely
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Optimizely
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
Optimizely
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Optimizely
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Optimizely
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Optimizely
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Optimizely
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Optimizely
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Optimizely
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Optimizely
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
Optimizely
 
Run High Impact Experimentation with High-quality Customer Discovery
Run High Impact Experimentation with High-quality Customer DiscoveryRun High Impact Experimentation with High-quality Customer Discovery
Run High Impact Experimentation with High-quality Customer Discovery
Optimizely
 
Using Empathy to Build Custom Solutions at Scale
Using Empathy to Build Custom Solutions at ScaleUsing Empathy to Build Custom Solutions at Scale
Using Empathy to Build Custom Solutions at Scale
Optimizely
 
How to find data insights that will drive a 10X impact
How to find data insights that will drive a 10X impact How to find data insights that will drive a 10X impact
How to find data insights that will drive a 10X impact
Optimizely
 
Targeted Rollouts: How to Release Features to Multiple Audiences
Targeted Rollouts: How to Release Features to Multiple AudiencesTargeted Rollouts: How to Release Features to Multiple Audiences
Targeted Rollouts: How to Release Features to Multiple Audiences
Optimizely
 
Deploying Fearlessly in a Continuous World
Deploying Fearlessly in a Continuous WorldDeploying Fearlessly in a Continuous World
Deploying Fearlessly in a Continuous World
Optimizely
 
Detecting incorrectly implemented experiments
Detecting incorrectly implemented experimentsDetecting incorrectly implemented experiments
Detecting incorrectly implemented experiments
Optimizely
 
Move Fast in the Age of Uncertainty
Move Fast in the Age of UncertaintyMove Fast in the Age of Uncertainty
Move Fast in the Age of Uncertainty
Optimizely
 
The Future of Building Good Products: Progressive Delivery and Experimentation
The Future of Building Good Products: Progressive Delivery and ExperimentationThe Future of Building Good Products: Progressive Delivery and Experimentation
The Future of Building Good Products: Progressive Delivery and Experimentation
Optimizely
 
How Clorox Experiments Across Brands to Turn Visitors into Consumers
How Clorox Experiments Across Brands to Turn Visitors into ConsumersHow Clorox Experiments Across Brands to Turn Visitors into Consumers
How Clorox Experiments Across Brands to Turn Visitors into Consumers
Optimizely
 

More from Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Run High Impact Experimentation with High-quality Customer Discovery
Run High Impact Experimentation with High-quality Customer DiscoveryRun High Impact Experimentation with High-quality Customer Discovery
Run High Impact Experimentation with High-quality Customer Discovery
 
Using Empathy to Build Custom Solutions at Scale
Using Empathy to Build Custom Solutions at ScaleUsing Empathy to Build Custom Solutions at Scale
Using Empathy to Build Custom Solutions at Scale
 
How to find data insights that will drive a 10X impact
How to find data insights that will drive a 10X impact How to find data insights that will drive a 10X impact
How to find data insights that will drive a 10X impact
 
Targeted Rollouts: How to Release Features to Multiple Audiences
Targeted Rollouts: How to Release Features to Multiple AudiencesTargeted Rollouts: How to Release Features to Multiple Audiences
Targeted Rollouts: How to Release Features to Multiple Audiences
 
Deploying Fearlessly in a Continuous World
Deploying Fearlessly in a Continuous WorldDeploying Fearlessly in a Continuous World
Deploying Fearlessly in a Continuous World
 
Detecting incorrectly implemented experiments
Detecting incorrectly implemented experimentsDetecting incorrectly implemented experiments
Detecting incorrectly implemented experiments
 
Move Fast in the Age of Uncertainty
Move Fast in the Age of UncertaintyMove Fast in the Age of Uncertainty
Move Fast in the Age of Uncertainty
 
The Future of Building Good Products: Progressive Delivery and Experimentation
The Future of Building Good Products: Progressive Delivery and ExperimentationThe Future of Building Good Products: Progressive Delivery and Experimentation
The Future of Building Good Products: Progressive Delivery and Experimentation
 
How Clorox Experiments Across Brands to Turn Visitors into Consumers
How Clorox Experiments Across Brands to Turn Visitors into ConsumersHow Clorox Experiments Across Brands to Turn Visitors into Consumers
How Clorox Experiments Across Brands to Turn Visitors into Consumers
 

Recently uploaded

Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
Philip Schwarz
 
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
rachitkumar09887
 
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTIONBITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
ssuser2b426d1
 
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docxComprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Aardwolf Security
 
dachnug51 - All you ever wanted to know about domino licensing.pdf
dachnug51 - All you ever wanted to know about domino licensing.pdfdachnug51 - All you ever wanted to know about domino licensing.pdf
dachnug51 - All you ever wanted to know about domino licensing.pdf
DNUG e.V.
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
jealousviolet
 
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...
karim wahed
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
jhkdigitalmarketing
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
akshesh doshi
 
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to KnowThe Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
onemonitarsoftware
 
11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf
kalichargn70th171
 
NBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial CompanyNBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial Company
NBFC Softwares
 
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
Hironori Washizaki
 
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdfBuilding infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
mohitd6
 
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
aslasdfmkhan4750
 
WEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service ProvidersWEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service Providers
Severalnines
 
ANSYS Mechanical APDL Introductory Tutorials.pdf
ANSYS Mechanical APDL Introductory Tutorials.pdfANSYS Mechanical APDL Introductory Tutorials.pdf
ANSYS Mechanical APDL Introductory Tutorials.pdf
sachin chaurasia
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
kayash1656
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
ashiklo9823
 
React Native vs Flutter - SSTech System
React Native vs Flutter  - SSTech SystemReact Native vs Flutter  - SSTech System
React Native vs Flutter - SSTech System
SSTech System
 

Recently uploaded (20)

Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
 
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
 
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTIONBITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
BITCOIN HEIST RANSOMEWARE ATTACK PREDICTION
 
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docxComprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
Comprehensive Vulnerability Assessments Process _ Aardwolf Security.docx
 
dachnug51 - All you ever wanted to know about domino licensing.pdf
dachnug51 - All you ever wanted to know about domino licensing.pdfdachnug51 - All you ever wanted to know about domino licensing.pdf
dachnug51 - All you ever wanted to know about domino licensing.pdf
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
 
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...
AWS Cloud Practitioner Essentials (Second Edition) (Arabic) Final Course Know...
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
 
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to KnowThe Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
 
11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf
 
NBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial CompanyNBFC Software: Optimize Your Non-Banking Financial Company
NBFC Software: Optimize Your Non-Banking Financial Company
 
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...
 
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdfBuilding infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
 
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
 
WEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service ProvidersWEBINAR SLIDES: CCX for Cloud Service Providers
WEBINAR SLIDES: CCX for Cloud Service Providers
 
ANSYS Mechanical APDL Introductory Tutorials.pdf
ANSYS Mechanical APDL Introductory Tutorials.pdfANSYS Mechanical APDL Introductory Tutorials.pdf
ANSYS Mechanical APDL Introductory Tutorials.pdf
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
 
React Native vs Flutter - SSTech System
React Native vs Flutter  - SSTech SystemReact Native vs Flutter  - SSTech System
React Native vs Flutter - SSTech System
 

How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagement by 10%

  • 1. How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagement by 10% Megan Bubley Sr. Product Manager THE ZEBRA
  • 2. 2 The Zebra We make insurance black and white. The Zebra was founded with a single goal: to simplify insurance, and in doing so we’ve become the nation’s leading insurance comparison site. Our vision is to be a trusted partner for consumers, helping them to better manage and understand all their insurance needs through every stage of life, even in the face of the unexpected.
  • 3. Carrier Card Redesign Auto Results Page - Multivariate Feature Test 3 Our goal with this redesign was to rapidly test and learn. We validated our design choices with usability tests but still had questions around which combination of features would create the best user experience and how performance would differ based on device. We wanted to test a combination of carrier card features but were concerned about the time it would take to analyze test performance and the amount of engineering bandwidth required to introduce a new round of A/B tests. In the past running a multivariate experiment this robust would require weeks of engineering time and would drastically increase our delivery timelines, however through the use of feature testing we were able to test multiple variants with limited engineering effort in a short amount of time!
  • 4. Differentiation by Platform Desktop vs. Mobile 4 For our first round of tests we wanted to focus on information layout and creating consistency across platforms so we could better assess how design impacted user engagement. We knew that historically desktop and mobile have performed very differently in terms of engagement. ● Mobile typically performed worse when more elements were added to carrier cards vs. desktop. ● Since desktop carrier cards have more real estate to utilize, more content was typically added to these cards creating a very different user experience across platforms.
  • 5. 5 Step One: Work with design and engineering to identify the carrier card elements we want to test along with the number of variants we wanted to start with. Step Two: Create feature variables within Optimizely that allow us to turn on or off elements on the carrier card so we can customize the design. Step Three: Once the test goes live, use real time data to monitor test performance and come up with future variants we want to test based on carrier card conversion rates. Development Strategy
  • 6. Feature Variables 6 Square Logo Rectangle Logo Star Ratings Coverage Value Prop First Month Pricing Price in CTA Carrier Taglines List View Card View
  • 7. 7 Running The Test The most challenging part of A/B testing has typically been saving space for optimizations, particularly when a feature is testing and developers are no longer actively working it. In the past we have had issues with prioritizing future optimizations based on how we think the test will perform in order to keep the momentum going with engineers. With feature based testing we were able to make changes to our test based on the data without any additional engineering effort—essentially we had the ability to create a custom carrier card without any engineering resources. Example of the feature test setup used for the carrier card redesign. 7
  • 9. Monitoring Performance & Use of Feature Flags Achieving Our Long Term Vision 9 While we had hoped to implement a similar experience across desktop and mobile we noticed that different variants performed better on different platforms—for example rectangular logos performed better on mobile and square logos did better on desktop. Since we wanted to come back to some of these findings and retest at a later date we utilized the feature flag functionality once we had a winning variant to promote that feature to 100% of traffic without eating into our traffic allotment in Optimizely.
  • 10. 10 10% Increase in carrier card engagement across platforms 60% Reduction in engineering effort and development time Final Outcomes We ran the carrier card feature test for over 6 months and worked closely with stakeholders to make changes to our variants as our product changed. By running this test as a feature experiment we were able to implement changes to our variants without any additional development effort—something that would have taken us weeks to implement in the past. We were also able to test different variants on mobile and desktop which allowed us to differentiate our experience by platform based on user behavior. As a result of these tests we saw significant improvement to carrier card engagement across platforms.
  • 11. 11 Benefits 1. Lots of flexibility when it comes to testing multiple designs. 2. Ability to turn on and off feature variables with the flip of a switch—no engineering effort required. 3. Ability to move the experiment behind a feature flag to reduce the strain on traffic allotments 4. Ability to retest features at a later date — once new features have been launched—to understand full impact of the final product on the user experience. Challenges 1. Lots of frontloading required to identify the different feature variables and testing strategy. 2. Additional engineering effort required up front in order to set up the feature test— was more challenging to implement than a straight A/B test. 3. Additional testing required to ensure all feature variables display correctly in production. 4. Multiple dashboards needed to record performance and there was not a super easy way to track performance in Optimizely.
  • 12. 12 PRESENTATIO N TITLE Questions? Contact Information: Email: mbubley@thezebra.com Linkedin: www.linkedin.com/in/megan-bubley

Editor's Notes

  1. I'm a senior product manager and i’ve been at The Zebra a little over a year and a half now. My main focus has been on improving the user experience on our auto results page and this presentation will cover one of the features we launched through the use of Optimizely’s feature experiment tool. [ID: CODE B04] How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagement by 10% Speaker: Megan Bubley, Sr. Product Manager, The Zebra SESSION TYPE: Customer TRACK: Product, Engineering
  2. The Zebra is an online insurance marketplace where users can search and compare multiple carriers to find a policy that works best for them. We realize shopping for insurance can be overwhelming for users so our goal is to take some of the stress and confusion out of that process and essentially make the purchasing process black and white for users.
  3. Before we jump in I wanted to provide a bit of insight on our results page. Essentially our main product - users need to complete a funnel with questions about their driving history and personal background in order for us to return rates from carriers. We have two types of carrier cards on our results page - ads and quotes - and each of these carrier cards has slightly different information - for example quotes return prices and ads do not. Prior to this redesign we conducted a lot of user research around our current results page experience to better understand some of our users pain points and help us prioritize the features we planned to test as part of this experiment This test was focused on redesigning our results page carrier cards. Prior to this redesign there were a lot of discrepancies between what content was visible on desktop vs. mobile - which I’ll get into a bit more later on in the presentation. As part of this test we wanted the flexibility of testing a lot of different variants in a short amount of time. Our goal with this experiment was to be able to rapidly test, learn, and iterate. Even though we had a lot of user research guiding our testing strategy we wanted the ability to incorporate both qual and quant findings into our testing decisions and if we could make changes to the test in real time based on what the data was telling us we could create a carrier card that best served our users.
  4. We knew we wanted to try and create as much consistency as possible for users no matter what platform they were shopping on but also noted that historically user behavior was pretty different when it came to user engagement with carrier cards on desktop vs. mobile. On mobile, when more content was added to carrier cards we typically saw a decrease in users who clicked on those cards - our assumption here was that more information on smaller screen sizes was overwhelming for users and increased the likelihood of decision fatigue. On desktop we had more real estate to play around with - however we felt the content we were displaying on the cards was not useful to users. There was a lot of duplicate information on these cards and our assumption here was users would skim right over this information and not find any real value the way the information was presented While we had high hopes that we could find a happy medium across both platforms we also understood that we may need to change up our testing strategy based on how the test was performing and running this as a feature experiment provided us with the flexibility to do that without any additional engineering effort.
  5. We had very specific goals we were trying to achieve with this test and we wanted the freedom to move quickly but the flexibility to test a lot of different things - which can be incredibly challenging especially when thinking about engineering resources and bandwidth. Knew we wanted to run this as a feature experiment where we could essentially “create our own carrier card” by creating individual feature variables for all the components on the card. This required a lot of thought and effort up front to ensure we knew all the different variables we wanted to test and make sure those variables would display correctly on the cards. - More to come on this Once we identified all the variables we wanted to test we were able to add them into Optimizely and begin developing the feature.
  6. Here’s some insight into how we broke out the different feature variables on the carrier cards - creating separate variable for each component on the card gave us the ability to create our own carrier card essentially with the flip of a switch.
  7. As a company that is incredibly focused on A/B testing and validating new designs by implementing them fully into our main product, one of the biggest challenges has been saving space and time to optimize our tests based on the data were gathering. As a result we either pivot to a new feature without fully achieving our final vision for the feature or we begin to prioritize optimizations based on how we think the test will perform - I think everyone will agree this is not an ideal way to operate. The benefit of utilizing feature testing is that we have the ability to make changes to the product without engineering resources. In the past this has always been the most difficult part of prioritization - thinking through the time it will take for devs to deinstrument a previous A/B, build a new feature, and then spin up a new A/B test has taken weeks. We also had to be cognizant of the time it took for a test to run and what developers would be working on while the test was in progress - do we move onto a new feature and try to circle back to the one that’s currently testing when we have the time? Do we wait for the test to finish before kicking off any new feature work - is this the best use of developers time? Needless to say these questions made it difficult for us to feel confident in our next steps. Feature testing took a lot of pressure off both product and engineering since the development work to support multiple variants was done up front and in doing so we could essentially run dozens of A/B tests in a fraction of the time it would normally take us. Here on the screen you can see an example of the variable key with all the different carrier card elements. We were able to turn on an element by switching the value to true and within minutes the new design would be visible in production.
  8. As I mentioned earlier we decided to launch the carrier card redesign with 3 potential designs - control and 2 variants - which you can see on the screen. These are the first round mobile designs we tested with - and even though its not pictured on the screen we did follow the same strategy on desktop as well. We planned to let the data help us make the call on what we did next so if control won we could assume adding more content to the cards was distracting and limit the number of changes we made in the next round of testing. If a variant won we planned to test that against a slightly different design to see if there was a potential to optimize the experience further.
  9. Once the test is live monitoring performance is pretty much the fun part. Our hope was that a similar trend would emerge across desktop and mobile but that was ultimately not the case so we ended up testing slightly different experience across platforms in order to identify a winning variant. Once we had a winner on both platforms we utilized the feature flag functionality in order to promote the winner to 100% without having to deinstrument the experiment. The nice part about using feature flags is that it allowed us to keep our experiment intact in case we wanted to come back to the experiment at a future date when new features had been introduced and retest without eating away at our traffic allocation in optimizely. This turned out to be a really smart move as we did come back to this test a few times over the course of 6 months to test different minor changes to the carrier cards which ended up being really impactful, not only because we were able to make these adjustments without any additional engineering resources, but we were also able to address stakeholder concerns and feedback in real time which helped us build even stronger collaboration process.
  10. As I mentioned we ended up running this test for about 6 months with great success. We were able to launch 3 different versions of carrier cards with our first round of testing to create a performance baseline we could build off of going forward. We were able to make come back to the experiment and make changes to carrier cards to better understand how individual elements on the cards impacted user engagement. Once the test was live we were able to rapidly test and iterate with no additional engineering effort, something that would have taken us months of time and effort in the past, and as a result we were able to reduce development time by 60%. This setup also had a really positive impact on stakeholder relationships and collaboration since we were able to loop them into our decision making process more regularly and provide them with more freedom to guide our iteration process. Finally we were able to improve the experience for our users, particularly in terms of carrier card conversion rates, by using data to drive real time changes to our carrier cards and regularly change up the experience based on how that feature was performing even after we introduced new product features on our results page.
  11. For my last slide I wanted to quickly run through some of the benefits and challenges of using feature testing for this carrier card redesign experiment.