SlideShare a Scribd company logo
1 of 47
Download to read offline
Designing, Building, and
Implementing Product
Experiments
Cian Mac Mahon
Designing, Implementing, And Analysing Product Experiments
A Brief History of Experimentation at
HubSpot Track all user
interactions, dig into
it post-launch to
understand user
impact
Combine qualitative
and quantitive data
to make predictions,
set goals based on
those predictions
prior to launch, hold
ourselves to account
with data
Try it out, wait
until we hear
feedback
from users
2014 2017 2019 - now
I have heard a lot of
things about building
experiments
Problems
• It’s difficult to come up with ideas
• Research and Engineering work
involved in building a randomised
experiment is too heavy
• You need a maths degree to understand
the results
Dirty Filthy Lies
• It’s difficult to come up with ideas
• Research and Engineering work
involved in building a randomised
experiment is too heavy
• You need a maths degree to understand
the results
Agenda
Who am I? ✅ How do I come up
with ideas?
How do I implement
experiments?
How do I analyse
the results?
Designing, Implementing, And Analysing Product Experiments
How do I come
up with ideas?
Guardrails
1. Experiments should have the aim of
creating a better user experience
2. Experiments should answer risky
questions
3. Experiments should reach statistical
significance
Obviously
1. Create better user
experiences
Less obviously
2. Answer risky
questions
High risk, no
existing data
Low risk, no existing
data
Low risk, existing
data
High risk, existing
data
No existing data
Existing data
Low
risk
High
risk
u wot?
3. Reach Statistical
Significance
r u havin a giggle?
u best not be havin a laff m8
Everybody
(10,000 people)
Statistical Significance
An equation which helps us estimate the chance
that the effect we see in our experimental
sample also exists in the wider population.*
*this is what it means for the usecase of somebody who isn’t a data analyst don’t @ me
Experimental
Cohort
(1,000
people)
Coin Flipping
16COHORT FLIPS HEADS TAILS
Control
Trustworthy coin
10 4 6
Variant
Suspicious coin
10 7 3
COHORT FLIPS HEADS TAILS
Control
Trustworthy coin
100 48 52
Variant
Suspicious coin
100 53 47
You
UXDX attendee
“But Cian - how do I
actually come up with
ideas for experiments?”
What do we
want to achieve?
Pick two or three metrics
which you will see a
change in over a short
period of time
01 02 03 04
Talk it out
Spent 15 minutes
chatting about the
metrics. Do you have
any hypothesis about
how to improve them?
Survival of the
Fittest
Each person pick a
metric, and spend 30
minutes trying to
disprove each
hypotheses using both
qualitative and
quantitive data
Write
experiment docs
Each person takes a
surviving hypothesis
and designs an
experiment to discover
if it is true
Experiment Afternoon
Experiment 1:
Importing contacts
to HubSpot
Importing contacts to
HubSpot CRM
• In order to fully understand the value of
HubSpot CRM a user needs to have some
data in the Contact Database.
• During user interviews, more than 95% of
users told us they wanted to import
contacts first thing
• Only 10% of users who see our setup
checklist start the Import Contacts task
• Of those who start, only 1/4 finish
importing their contacts
Where do we lose you?
The contact importing journey
0
20
40
60
80
100
Start task
Upload file
Complete
import
How do I
build
experiments?
Experiment Doc
• What is your hypothesis?
• What evidence do you have to support it?
• What is the change you will be making?
• What is your success metric?
• What is the minimum improvement you’d
accept in order to consider the experiment a
success?
What is your hypothesis?
“We know that importing contacts is the best way to
get started using HubSpot, and we know that users
want to get their contacts into the system. But we
believe that it is too hard, which we believe
discourages users from doing so.”
What is your evidence?
• People tell us over and over again in user research
that they want to import their contacts.
• 1/5 users who start importing contacts finish doing
so.
• If a user selects a file to upload, they’re very likely to
complete the process.
What is the change you will be making?
We will split our signup
cohort from July 1st to July
14th in 3. Control will see
the existing Onboarding
experience. Variant 1 will
see a Quick Import banner,
Variant 2 will have a new
“Quick Import” checklist
item.
COHORT SIZE CHANGE
Control 33.3%
No change, show existing
onboarding content
Variant 1
In your face variant
33.3%
Show users a large Quick
Import banner
Variant 2
Integrated variant
33.3%
Swap the normal “Import”
checklist task with a
“Quick Import” task
What is the change you will be making?
Control Variant 1 Variant 2
What is the change you will be making?
What is the change you will be making?
What is your success metric
“We will judge success on the percentage of users who
complete an import from the Getting Started checklist”
What is your minimum improvement
“Our minimum improvement is 25% improvement on
the existing metric”
“In order to detect a minimum improvement of 25%
with 95% significance (p-value of 0.05) we will need to
run for 14 days.”
Tell me again!
• What is your hypothesis?
• What evidence do you have to support it?
• What is the change you will be making?
• What is your success metric?
• What is the minimum improvement you’d
accept in order to consider the experiment a
success?
• bit.ly/hs-experiment-doc
Time to build!
Time to build!
• Randomly assign user cohorts
• Assignment should be functional and
stateless
• Consider if you want to assign users, or
accounts, in a product where accounts can
have multiple users
• Send enrolment event to analytics product
• Be sure to include cohort!
• Send success metric event to analytics product
PlanOut
facebook.github.io/planout
• Python
• Ported to JS, Java, and PHP
• Functionally assign cohorts
• Log experimental events
• Namespace experiments to avoid clashes
• Our blog post on using it with React:
bit.ly/react-experiments
Laboratory
HubSpot’s Experiment
Infrastructure
• Based on PlanOut’s algorithm
• Cohort assignment and data gathering
happens on the backend
• All team members can enable, disable,
administer parameters
• Automatically builds analysis based on
team KPIs
• Builds a library of documentation around
that experiment
How do I
analyse
experiments?
Importing contacts to HubSpot
Control
Variant 1 (Banner)
0%
20%
40%
60%
80%
100%
View Getting Started Complete Import
12%
100%
14%
100%
11%
100%
Variant 2 (Checklist task)
+27% +9%
Designing, Implementing, And Analysing Product Experiments
Experiment 2:
Users believe our
free tools cost
money
What is your hypothesis?
“We believe that if we re-iterate through the demo tour
that all the functionality shown is free, users will be
more motivated to try them.”
What is your evidence?
• User research indicates that after completing the
demo tour, some users are uncertain what free tools
they have access to.
• User research indicates that after testing the demo,
some users say that they assumed the shown tools
require payment to use
What is the change you will be making?
We will split our signup
cohort for two weeks in two.
Control will see the existing
demo tour copy.
Variant will see updated
demo tour copy which
regularly mentions that all
demonstrated functionality is
free.
COHORT SIZE CHANGE
Control 50%
No change, show existing
copy
Variant 1
It’s Free!
50%
Re-iterate through the
demo tour that all
demonstrated
functionality is free
What is your success metric
We will judge success based on activation of our free
marketing tools.
What is your minimum improvement
Our minimum improvement is 0.4 percentage points
over the control.
We will need to run the experiment for 14 days in order
to reach a p-value of 0.02.
Users believe our free tools cost money
Control
Variant
View Help Widget Activate our free Marketing tools
Thank you
Cian@HubSpot.com
(We’re hiring)
@Cianmm
( )

More Related Content

More from UXDXConf

Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...UXDXConf
 
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...UXDXConf
 
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023UXDXConf
 
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023UXDXConf
 
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...UXDXConf
 
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023UXDXConf
 
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...UXDXConf
 
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023UXDXConf
 
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023UXDXConf
 
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023UXDXConf
 
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...UXDXConf
 
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023UXDXConf
 
Building Sustainable Design Systems
Building Sustainable Design SystemsBuilding Sustainable Design Systems
Building Sustainable Design SystemsUXDXConf
 
Building a Seamless Online to Offline Experience
Building a Seamless Online to Offline ExperienceBuilding a Seamless Online to Offline Experience
Building a Seamless Online to Offline ExperienceUXDXConf
 
Your Product Manager Toolkit
Your Product Manager ToolkitYour Product Manager Toolkit
Your Product Manager ToolkitUXDXConf
 
Team Downsized: How Do You Move Forward?
Team Downsized: How Do You Move Forward?Team Downsized: How Do You Move Forward?
Team Downsized: How Do You Move Forward?UXDXConf
 
Applying the JTBD Framework to Build User Personas that Stick
Applying the JTBD Framework to Build User Personas that StickApplying the JTBD Framework to Build User Personas that Stick
Applying the JTBD Framework to Build User Personas that StickUXDXConf
 
How Might We Get Engineers Involved in Discovery - Tim Simms
How Might We Get Engineers Involved in Discovery - Tim SimmsHow Might We Get Engineers Involved in Discovery - Tim Simms
How Might We Get Engineers Involved in Discovery - Tim SimmsUXDXConf
 
Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...
Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...
Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...UXDXConf
 
Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...
Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...
Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...UXDXConf
 

More from UXDXConf (20)

Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...Operational Evolution_ Harnessing Design for Internal Business Transformation...
Operational Evolution_ Harnessing Design for Internal Business Transformation...
 
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
Design Delivery Principles, Accelerating Product Delivery Outcomes in a Compl...
 
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
Innovation Without Adoption Is Meaningless_ Rhiannon White_ UXDX_ EMEA_2023
 
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
Learning Balanced Leadership with Horses_Martina Hodges-Schell_ UXDX_ EMEA_2023
 
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
An Unexpected Approach to Design: Uncovering Hidden Parallels With Writing St...
 
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
Embracing Uncertainty in Product Delivery_ Martin Reilly_ UXDX_ EMEA_2023
 
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
Embracing Career Pivots, Navigating Change, and Building a Motivated Remote T...
 
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
Uncharted waters of design leadership_ Anderson Gomes_ UXDX_ EMEA_ 2023
 
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
Enhancing the Impact of User Research_ Fahad Osmani_ UXDX_EMEA_2023
 
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
How to Know Enough to Know You Don't Know_ Shilpi Sinha_UXDX_EMEA_2023
 
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
Lessons Learned From Our Accessibility-First Approach to Data Visualisation- ...
 
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
Fostering a customer-centric culture at an enterprise_UXDX_EMEA_2023
 
Building Sustainable Design Systems
Building Sustainable Design SystemsBuilding Sustainable Design Systems
Building Sustainable Design Systems
 
Building a Seamless Online to Offline Experience
Building a Seamless Online to Offline ExperienceBuilding a Seamless Online to Offline Experience
Building a Seamless Online to Offline Experience
 
Your Product Manager Toolkit
Your Product Manager ToolkitYour Product Manager Toolkit
Your Product Manager Toolkit
 
Team Downsized: How Do You Move Forward?
Team Downsized: How Do You Move Forward?Team Downsized: How Do You Move Forward?
Team Downsized: How Do You Move Forward?
 
Applying the JTBD Framework to Build User Personas that Stick
Applying the JTBD Framework to Build User Personas that StickApplying the JTBD Framework to Build User Personas that Stick
Applying the JTBD Framework to Build User Personas that Stick
 
How Might We Get Engineers Involved in Discovery - Tim Simms
How Might We Get Engineers Involved in Discovery - Tim SimmsHow Might We Get Engineers Involved in Discovery - Tim Simms
How Might We Get Engineers Involved in Discovery - Tim Simms
 
Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...
Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...
Embracing Uncertainty: How to Get Untangled & Make Constant Change Work for Y...
 
Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...
Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...
Standardising UX: A Roadmap for Success by Duaa Gettani, Senior UX Researcher...
 

Recently uploaded

The Roles of Aviation Auditors - Presentation
The Roles of Aviation Auditors - PresentationThe Roles of Aviation Auditors - Presentation
The Roles of Aviation Auditors - PresentationTilak Ramaprakash
 
What Are The Must-Know Tips For First-Time Jet Skiers In Aruba
What Are The Must-Know Tips For First-Time Jet Skiers In ArubaWhat Are The Must-Know Tips For First-Time Jet Skiers In Aruba
What Are The Must-Know Tips For First-Time Jet Skiers In ArubaDelphi Watersports
 
Sicily Holidays Guide Book: Unveiling the Treasures of Italy's Jewel
Sicily Holidays Guide Book: Unveiling the Treasures of Italy's JewelSicily Holidays Guide Book: Unveiling the Treasures of Italy's Jewel
Sicily Holidays Guide Book: Unveiling the Treasures of Italy's JewelTime for Sicily
 
5 beautyfull places visiting in uttrakhand
5 beautyfull places visiting in uttrakhand5 beautyfull places visiting in uttrakhand
5 beautyfull places visiting in uttrakhandaradhya3287
 
5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)
5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)
5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)Mazie Garcia
 
Lucknow to Sitapur Cab | Lucknow to Sitapur Taxi
Lucknow to Sitapur Cab | Lucknow to Sitapur TaxiLucknow to Sitapur Cab | Lucknow to Sitapur Taxi
Lucknow to Sitapur Cab | Lucknow to Sitapur TaxiCab Bazar
 
Revolutionalizing Travel: A VacAI Update
Revolutionalizing Travel: A VacAI UpdateRevolutionalizing Travel: A VacAI Update
Revolutionalizing Travel: A VacAI Updatejoymorrison10
 
What Safety Precautions Are Recommended For Na Pali Snorkeling Adventure
What Safety Precautions Are Recommended For Na Pali Snorkeling AdventureWhat Safety Precautions Are Recommended For Na Pali Snorkeling Adventure
What Safety Precautions Are Recommended For Na Pali Snorkeling AdventureHanalei Charters
 
It’s Time Get Refresh Travel Around The World
It’s Time Get Refresh Travel Around The WorldIt’s Time Get Refresh Travel Around The World
It’s Time Get Refresh Travel Around The WorldParagliding Billing Bir
 
69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)
69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)
69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)Escort Service
 
How Safe Is It To Witness Whales In Maui’s Waters
How Safe Is It To Witness Whales In Maui’s WatersHow Safe Is It To Witness Whales In Maui’s Waters
How Safe Is It To Witness Whales In Maui’s WatersMakena Coast Charters
 
Disney Dreams in Europe: A Guide to Disneyland Paris
Disney Dreams in Europe: A Guide to Disneyland ParisDisney Dreams in Europe: A Guide to Disneyland Paris
Disney Dreams in Europe: A Guide to Disneyland ParisisangoTravel
 
Authentic Travel Experience 2024 Greg DeShields.pptx
Authentic Travel Experience 2024 Greg DeShields.pptxAuthentic Travel Experience 2024 Greg DeShields.pptx
Authentic Travel Experience 2024 Greg DeShields.pptxGregory DeShields
 
Transportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdf
Transportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdfTransportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdf
Transportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdfGlobalbustours
 
Canada PR - Eligibility, Steps to apply and Visa processing fees
Canada PR - Eligibility, Steps to apply and Visa processing feesCanada PR - Eligibility, Steps to apply and Visa processing fees
Canada PR - Eligibility, Steps to apply and Visa processing feesY-Axis Overseas Careers
 
Paragliding Billing Bir at Himachal Pardesh
Paragliding Billing Bir at Himachal PardeshParagliding Billing Bir at Himachal Pardesh
Paragliding Billing Bir at Himachal PardeshParagliding Billing Bir
 
What Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To Grasp
What Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To GraspWhat Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To Grasp
What Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To GraspHanalei Surf School
 
Story Of Neem Karoli Baba -Kainchi Dham Yatra
Story Of Neem Karoli Baba -Kainchi Dham YatraStory Of Neem Karoli Baba -Kainchi Dham Yatra
Story Of Neem Karoli Baba -Kainchi Dham YatraSuYatra
 
Top Five Best Places to Visit in India.pdf
Top Five Best Places to Visit in India.pdfTop Five Best Places to Visit in India.pdf
Top Five Best Places to Visit in India.pdfonlinevisaindia
 
a presentation for foreigners about how to travel in Germany.
a presentation for foreigners about how to travel in Germany.a presentation for foreigners about how to travel in Germany.
a presentation for foreigners about how to travel in Germany.moritzmieg
 

Recently uploaded (20)

The Roles of Aviation Auditors - Presentation
The Roles of Aviation Auditors - PresentationThe Roles of Aviation Auditors - Presentation
The Roles of Aviation Auditors - Presentation
 
What Are The Must-Know Tips For First-Time Jet Skiers In Aruba
What Are The Must-Know Tips For First-Time Jet Skiers In ArubaWhat Are The Must-Know Tips For First-Time Jet Skiers In Aruba
What Are The Must-Know Tips For First-Time Jet Skiers In Aruba
 
Sicily Holidays Guide Book: Unveiling the Treasures of Italy's Jewel
Sicily Holidays Guide Book: Unveiling the Treasures of Italy's JewelSicily Holidays Guide Book: Unveiling the Treasures of Italy's Jewel
Sicily Holidays Guide Book: Unveiling the Treasures of Italy's Jewel
 
5 beautyfull places visiting in uttrakhand
5 beautyfull places visiting in uttrakhand5 beautyfull places visiting in uttrakhand
5 beautyfull places visiting in uttrakhand
 
5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)
5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)
5S - House keeping (Seiri, Seiton, Seiso, Seiketsu, Shitsuke)
 
Lucknow to Sitapur Cab | Lucknow to Sitapur Taxi
Lucknow to Sitapur Cab | Lucknow to Sitapur TaxiLucknow to Sitapur Cab | Lucknow to Sitapur Taxi
Lucknow to Sitapur Cab | Lucknow to Sitapur Taxi
 
Revolutionalizing Travel: A VacAI Update
Revolutionalizing Travel: A VacAI UpdateRevolutionalizing Travel: A VacAI Update
Revolutionalizing Travel: A VacAI Update
 
What Safety Precautions Are Recommended For Na Pali Snorkeling Adventure
What Safety Precautions Are Recommended For Na Pali Snorkeling AdventureWhat Safety Precautions Are Recommended For Na Pali Snorkeling Adventure
What Safety Precautions Are Recommended For Na Pali Snorkeling Adventure
 
It’s Time Get Refresh Travel Around The World
It’s Time Get Refresh Travel Around The WorldIt’s Time Get Refresh Travel Around The World
It’s Time Get Refresh Travel Around The World
 
69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)
69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)
69 Girls ✠ 9599264170 ✠ Call Girls In East Of Kailash (VIP)
 
How Safe Is It To Witness Whales In Maui’s Waters
How Safe Is It To Witness Whales In Maui’s WatersHow Safe Is It To Witness Whales In Maui’s Waters
How Safe Is It To Witness Whales In Maui’s Waters
 
Disney Dreams in Europe: A Guide to Disneyland Paris
Disney Dreams in Europe: A Guide to Disneyland ParisDisney Dreams in Europe: A Guide to Disneyland Paris
Disney Dreams in Europe: A Guide to Disneyland Paris
 
Authentic Travel Experience 2024 Greg DeShields.pptx
Authentic Travel Experience 2024 Greg DeShields.pptxAuthentic Travel Experience 2024 Greg DeShields.pptx
Authentic Travel Experience 2024 Greg DeShields.pptx
 
Transportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdf
Transportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdfTransportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdf
Transportation Options_ Getting to Keukenhof Gardens from Amsterdam.pdf
 
Canada PR - Eligibility, Steps to apply and Visa processing fees
Canada PR - Eligibility, Steps to apply and Visa processing feesCanada PR - Eligibility, Steps to apply and Visa processing fees
Canada PR - Eligibility, Steps to apply and Visa processing fees
 
Paragliding Billing Bir at Himachal Pardesh
Paragliding Billing Bir at Himachal PardeshParagliding Billing Bir at Himachal Pardesh
Paragliding Billing Bir at Himachal Pardesh
 
What Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To Grasp
What Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To GraspWhat Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To Grasp
What Unwritten Rules Of Surfing Etiquette Are Crucial For Beginners To Grasp
 
Story Of Neem Karoli Baba -Kainchi Dham Yatra
Story Of Neem Karoli Baba -Kainchi Dham YatraStory Of Neem Karoli Baba -Kainchi Dham Yatra
Story Of Neem Karoli Baba -Kainchi Dham Yatra
 
Top Five Best Places to Visit in India.pdf
Top Five Best Places to Visit in India.pdfTop Five Best Places to Visit in India.pdf
Top Five Best Places to Visit in India.pdf
 
a presentation for foreigners about how to travel in Germany.
a presentation for foreigners about how to travel in Germany.a presentation for foreigners about how to travel in Germany.
a presentation for foreigners about how to travel in Germany.
 

Designing, Implementing, And Analysing Product Experiments

  • 1. Designing, Building, and Implementing Product Experiments Cian Mac Mahon
  • 3. A Brief History of Experimentation at HubSpot Track all user interactions, dig into it post-launch to understand user impact Combine qualitative and quantitive data to make predictions, set goals based on those predictions prior to launch, hold ourselves to account with data Try it out, wait until we hear feedback from users 2014 2017 2019 - now
  • 4. I have heard a lot of things about building experiments
  • 5. Problems • It’s difficult to come up with ideas • Research and Engineering work involved in building a randomised experiment is too heavy • You need a maths degree to understand the results
  • 6. Dirty Filthy Lies • It’s difficult to come up with ideas • Research and Engineering work involved in building a randomised experiment is too heavy • You need a maths degree to understand the results
  • 7. Agenda Who am I? ✅ How do I come up with ideas? How do I implement experiments? How do I analyse the results?
  • 9. How do I come up with ideas?
  • 10. Guardrails 1. Experiments should have the aim of creating a better user experience 2. Experiments should answer risky questions 3. Experiments should reach statistical significance
  • 11. Obviously 1. Create better user experiences
  • 12. Less obviously 2. Answer risky questions
  • 13. High risk, no existing data Low risk, no existing data Low risk, existing data High risk, existing data No existing data Existing data Low risk High risk
  • 14. u wot? 3. Reach Statistical Significance r u havin a giggle? u best not be havin a laff m8
  • 15. Everybody (10,000 people) Statistical Significance An equation which helps us estimate the chance that the effect we see in our experimental sample also exists in the wider population.* *this is what it means for the usecase of somebody who isn’t a data analyst don’t @ me Experimental Cohort (1,000 people)
  • 16. Coin Flipping 16COHORT FLIPS HEADS TAILS Control Trustworthy coin 10 4 6 Variant Suspicious coin 10 7 3 COHORT FLIPS HEADS TAILS Control Trustworthy coin 100 48 52 Variant Suspicious coin 100 53 47
  • 17. You UXDX attendee “But Cian - how do I actually come up with ideas for experiments?”
  • 18. What do we want to achieve? Pick two or three metrics which you will see a change in over a short period of time 01 02 03 04 Talk it out Spent 15 minutes chatting about the metrics. Do you have any hypothesis about how to improve them? Survival of the Fittest Each person pick a metric, and spend 30 minutes trying to disprove each hypotheses using both qualitative and quantitive data Write experiment docs Each person takes a surviving hypothesis and designs an experiment to discover if it is true Experiment Afternoon
  • 20. Importing contacts to HubSpot CRM • In order to fully understand the value of HubSpot CRM a user needs to have some data in the Contact Database. • During user interviews, more than 95% of users told us they wanted to import contacts first thing • Only 10% of users who see our setup checklist start the Import Contacts task • Of those who start, only 1/4 finish importing their contacts
  • 21. Where do we lose you? The contact importing journey 0 20 40 60 80 100 Start task Upload file Complete import
  • 23. Experiment Doc • What is your hypothesis? • What evidence do you have to support it? • What is the change you will be making? • What is your success metric? • What is the minimum improvement you’d accept in order to consider the experiment a success?
  • 24. What is your hypothesis? “We know that importing contacts is the best way to get started using HubSpot, and we know that users want to get their contacts into the system. But we believe that it is too hard, which we believe discourages users from doing so.”
  • 25. What is your evidence? • People tell us over and over again in user research that they want to import their contacts. • 1/5 users who start importing contacts finish doing so. • If a user selects a file to upload, they’re very likely to complete the process.
  • 26. What is the change you will be making? We will split our signup cohort from July 1st to July 14th in 3. Control will see the existing Onboarding experience. Variant 1 will see a Quick Import banner, Variant 2 will have a new “Quick Import” checklist item. COHORT SIZE CHANGE Control 33.3% No change, show existing onboarding content Variant 1 In your face variant 33.3% Show users a large Quick Import banner Variant 2 Integrated variant 33.3% Swap the normal “Import” checklist task with a “Quick Import” task
  • 27. What is the change you will be making? Control Variant 1 Variant 2
  • 28. What is the change you will be making?
  • 29. What is the change you will be making?
  • 30. What is your success metric “We will judge success on the percentage of users who complete an import from the Getting Started checklist”
  • 31. What is your minimum improvement “Our minimum improvement is 25% improvement on the existing metric” “In order to detect a minimum improvement of 25% with 95% significance (p-value of 0.05) we will need to run for 14 days.”
  • 32. Tell me again! • What is your hypothesis? • What evidence do you have to support it? • What is the change you will be making? • What is your success metric? • What is the minimum improvement you’d accept in order to consider the experiment a success? • bit.ly/hs-experiment-doc
  • 34. Time to build! • Randomly assign user cohorts • Assignment should be functional and stateless • Consider if you want to assign users, or accounts, in a product where accounts can have multiple users • Send enrolment event to analytics product • Be sure to include cohort! • Send success metric event to analytics product
  • 35. PlanOut facebook.github.io/planout • Python • Ported to JS, Java, and PHP • Functionally assign cohorts • Log experimental events • Namespace experiments to avoid clashes • Our blog post on using it with React: bit.ly/react-experiments
  • 36. Laboratory HubSpot’s Experiment Infrastructure • Based on PlanOut’s algorithm • Cohort assignment and data gathering happens on the backend • All team members can enable, disable, administer parameters • Automatically builds analysis based on team KPIs • Builds a library of documentation around that experiment
  • 38. Importing contacts to HubSpot Control Variant 1 (Banner) 0% 20% 40% 60% 80% 100% View Getting Started Complete Import 12% 100% 14% 100% 11% 100% Variant 2 (Checklist task) +27% +9%
  • 40. Experiment 2: Users believe our free tools cost money
  • 41. What is your hypothesis? “We believe that if we re-iterate through the demo tour that all the functionality shown is free, users will be more motivated to try them.”
  • 42. What is your evidence? • User research indicates that after completing the demo tour, some users are uncertain what free tools they have access to. • User research indicates that after testing the demo, some users say that they assumed the shown tools require payment to use
  • 43. What is the change you will be making? We will split our signup cohort for two weeks in two. Control will see the existing demo tour copy. Variant will see updated demo tour copy which regularly mentions that all demonstrated functionality is free. COHORT SIZE CHANGE Control 50% No change, show existing copy Variant 1 It’s Free! 50% Re-iterate through the demo tour that all demonstrated functionality is free
  • 44. What is your success metric We will judge success based on activation of our free marketing tools.
  • 45. What is your minimum improvement Our minimum improvement is 0.4 percentage points over the control. We will need to run the experiment for 14 days in order to reach a p-value of 0.02.
  • 46. Users believe our free tools cost money Control Variant View Help Widget Activate our free Marketing tools