SlideShare a Scribd company logo
1 of 36
Download to read offline
Alicia Shiu
Growth Product Manager, Amplitude
How to Find Your Product's North
Star Metric
Tune in on Twitter: #testandlearn19
Biomedical
Engineering
Growth Product
Manager
Neuroscience
Alicia Shiu
Growth Product Manager
30K Products Powered
>7T User Actions Analyzed
Marketing
About me..
How does your product development team
define success?
# of features shipped
did we ship them on time
The most successful product teams measure
impact on business outcomes, not outputs
Lower barriers to
entry
Massive explosion in
product choice &
noise
Customer experience
and value trump
everything
Product is the best
competitive advantage
Product-led companies are winning
5m+ apps 0.01% survival rate
What does it mean to be Product-led?
Vision of Solving
Customer Pain
Revenue and
Shareholder Value
Impact the CEO
cares about
Customer value created
by the Product team
Product Analytics
Alerts App
Redesign
New User
Activation
Help Product
LaunchTrial Conversion
Profile
UX
North
Star
The 3 Games of Product
What Makes a Good North Star?
Evolving Amplitude’s North Star
From North Star to
Experimentation
→
→
→
→
Finding Your
North Star
The 3 Games of Product
What Makes a Good North Star?
Evolving Amplitude’s North Star
From North Star to
Experimentation
→
→
→
→
Finding Your
North Star
What game
are you
playing?
The 3 games of Product
attention transaction productivity
What game are you playing?
attention transaction productivity
Define the game you’re playing
attention
Typical Product North Stars Examples
● Total time spent
engaging with content
● # Users spending > 5
min/day
● # Stories viewed
● # Subscribers streaming
> 60 hrs/mo
hypothetical
Define the game you’re playing
Typical Product North Stars Examples
● Total # of Purchases
completed
● Avg # Purchases per
Prime Subscriber
● # Searches completed
● # Seat Upgrades
purchasedtransaction
hypothetical
Define the game you’re playing
Typical Product North Stars Examples
● Total # of records created
● % of Emails engaged
● # of messages sent
● # Active Users who
logged/completed a task
productivity
hypothetical
The 3 Games of Product
What Makes a Good North Star?
Evolving Amplitude’s North Star
From North Star to
Experimentation
→
→
→
→
Finding Your
North Star
3 attributes
of a good
North Star
metric
Measure product engagement that
is a LEADING INDICATOR of
revenue
Reflect the VALUE that customers
are getting from your product
Represent the core product
outcome you need to ALIGN with
your company strategy
1
2
3
The 3 Games of Product
What Makes a Good North Star?
Evolving Amplitude’s North Star
From North Star to
Experimentation
→
→
→
→
Finding Your
North Star
Example:
Evolving
our North
Star
2017-2018
2017-2018
Building product is a team sport!
?
As end user adoption increases,
businesses exchange greater value
with Amplitude
Customers get value by asking
questions and sharing learnings
Our unique strategy is to enable
data democracy in companies
1
2
3
Weekly
Learning
Users
(learnings shared with > 2 people)
As end user adoption increases,
businesses exchange greater value
with Amplitude
Customers get value by asking
questions and sharing learnings
Our unique strategy is to enable
data democracy in companies
1
2
3
Getting closer to customer value
Users that ask a question
(Weekly Querying Users)
Getting closer to customer value
Users that share learnings
(Weekly Learning Users)
Users that ask a question
(Weekly Querying Users)
Active and had a learning
shared w/ at least 2 people
in last 7 days
The 3 Games of Product
What Makes a Good North Star?
Evolving Amplitude’s North Star
From North Star to
Experimentation
→
→
→
→
Finding Your
North Star
Product experimentation to impact your North Star
Hypothesis: More user profile context & relevancy signals will encourage users
to click into Team Space and view team members’ content
Product experimentation to impact your North Star
Hypothesis: More user profile context & relevancy signals will encourage users
to click into Team Space and view team members’ content
Control Test
Getting started
● Define your product’s (or team’s)
North Star
● Create a hypothesis
● Kick off a new (small) project (or
align a current one)
● Test the hypothesis to see if you
can move the metric!
Define which game you’re playing: attention, transaction, or
productivity. Articulate tradeoffs you’re willing to make.
A good North Star reflects customer value, aligns product
teams with your company strategy, and is a leading indicator
of revenue.
Organize product experimentation to move your North Star
metric (or inputs / leading indicators of it).
→
→
→
Takeaways
Keep Learning!
For more details on the 3 product games,
ladder of engagement, & loops:
https://amplitude.com/mastering-engagement
Additional reading on North Star metrics:
- https://amplitude.com/blog/evolving-the-
product-north-star-metric
- https://amplitude.com/blog/evolving-the-
product-north-star-metric
Thank You! Questions?
Appendix
Breadth Depth Frequency Efficiencyx x x
# Items
within
Order
# Orders
completed
% Items
Delivered
within time
# Users
placing
Orders each
month
Break the North Star down into actionable input metrics
hypothetical
Total
monthly
items
received
on time
Actionable inputs: Efficiency
Breadth Depth Frequency Efficiencyx x x
# Items
within
Order
# Orders
Completed
% Items
Delivered
within time
ML Algorithm
Customer chat
Flash Sales
Notifications
Stock mgt
Recommendations
Shopping Lists
Discount Coupons
New Signups
Express Sub Trials
Trial Conversion
# Users
placing
Orders each
month
Roadmap
hypothetical
Total
monthly
items
received
on time
Next Session
Moving Fast, Breaking Things, and Fixing Them
as Quickly as Possible
Lukas Vermeer
Director of Experimentation, Booking.com
Tune in on Twitter: #testandlearn19

More Related Content

What's hot

心理的安全性の構造 デブサミ2019夏 structure of psychological safety
心理的安全性の構造 デブサミ2019夏 structure of psychological safety心理的安全性の構造 デブサミ2019夏 structure of psychological safety
心理的安全性の構造 デブサミ2019夏 structure of psychological safetyTokoroten Nakayama
 
The Power of a Proper North Star Metric
The Power of a Proper North Star MetricThe Power of a Proper North Star Metric
The Power of a Proper North Star MetricRocketSource
 
ビジネスパーソンのためのDX入門講座エッセンス版
ビジネスパーソンのためのDX入門講座エッセンス版ビジネスパーソンのためのDX入門講座エッセンス版
ビジネスパーソンのためのDX入門講座エッセンス版Tokoroten Nakayama
 
Salesforceの標準オブジェクトについて復習してみた
Salesforceの標準オブジェクトについて復習してみたSalesforceの標準オブジェクトについて復習してみた
Salesforceの標準オブジェクトについて復習してみたy-maeda
 
B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020Gokul Rangarajan
 
A/B Testing for New Product Launches by Booking.com Sr PM
A/B Testing for New Product Launches by Booking.com Sr PMA/B Testing for New Product Launches by Booking.com Sr PM
A/B Testing for New Product Launches by Booking.com Sr PMProduct School
 
Airbyte - Seed deck
Airbyte  - Seed deckAirbyte  - Seed deck
Airbyte - Seed deckAirbyte
 
既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!
既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!
既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!Recruit Lifestyle Co., Ltd.
 
Product Discovery @ Nubank
Product Discovery @ NubankProduct Discovery @ Nubank
Product Discovery @ NubankGustavo Mendes
 
User Story Mapping, Discover the whole story
User Story Mapping, Discover the whole storyUser Story Mapping, Discover the whole story
User Story Mapping, Discover the whole storyJeff Patton
 
What is a North Star Metric?
What is a North Star Metric?What is a North Star Metric?
What is a North Star Metric?GrowthHackers
 
Identifying your North Star Metric & Building a Model to Predict Growth
Identifying your North Star Metric & Building a Model to Predict GrowthIdentifying your North Star Metric & Building a Model to Predict Growth
Identifying your North Star Metric & Building a Model to Predict GrowthGrowthHackers
 
Dropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deck
Dropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deckDropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deck
Dropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deckAA BB
 
SalesforceにおけるCDC(変更データキャプチャ)の実装・活用法について
SalesforceにおけるCDC(変更データキャプチャ)の実装・活用法についてSalesforceにおけるCDC(変更データキャプチャ)の実装・活用法について
SalesforceにおけるCDC(変更データキャプチャ)の実装・活用法についてTakashi Hatamoto
 
A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...
A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...
A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...Dan Olsen
 
How to Build a Robust Product Roadmap by Salesforce VP of Product
How to Build a Robust Product Roadmap by Salesforce VP of ProductHow to Build a Robust Product Roadmap by Salesforce VP of Product
How to Build a Robust Product Roadmap by Salesforce VP of ProductProduct School
 
10分でわかったつもりになるLean Analytics_10min lean analytics
10分でわかったつもりになるLean Analytics_10min lean analytics10分でわかったつもりになるLean Analytics_10min lean analytics
10分でわかったつもりになるLean Analytics_10min lean analytics圭 進藤
 
「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-
「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-
「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-yabbysan
 

What's hot (20)

心理的安全性の構造 デブサミ2019夏 structure of psychological safety
心理的安全性の構造 デブサミ2019夏 structure of psychological safety心理的安全性の構造 デブサミ2019夏 structure of psychological safety
心理的安全性の構造 デブサミ2019夏 structure of psychological safety
 
The Power of a Proper North Star Metric
The Power of a Proper North Star MetricThe Power of a Proper North Star Metric
The Power of a Proper North Star Metric
 
ビジネスパーソンのためのDX入門講座エッセンス版
ビジネスパーソンのためのDX入門講座エッセンス版ビジネスパーソンのためのDX入門講座エッセンス版
ビジネスパーソンのためのDX入門講座エッセンス版
 
Salesforceの標準オブジェクトについて復習してみた
Salesforceの標準オブジェクトについて復習してみたSalesforceの標準オブジェクトについて復習してみた
Salesforceの標準オブジェクトについて復習してみた
 
B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020B2B SaaS north star metric 2019 2020
B2B SaaS north star metric 2019 2020
 
A/B Testing for New Product Launches by Booking.com Sr PM
A/B Testing for New Product Launches by Booking.com Sr PMA/B Testing for New Product Launches by Booking.com Sr PM
A/B Testing for New Product Launches by Booking.com Sr PM
 
Airbyte - Seed deck
Airbyte  - Seed deckAirbyte  - Seed deck
Airbyte - Seed deck
 
既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!
既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!
既存Redshift/ETLからSpectrum/Glueへの移行を徹底解明!
 
Product Discovery @ Nubank
Product Discovery @ NubankProduct Discovery @ Nubank
Product Discovery @ Nubank
 
User Story Mapping, Discover the whole story
User Story Mapping, Discover the whole storyUser Story Mapping, Discover the whole story
User Story Mapping, Discover the whole story
 
What is a North Star Metric?
What is a North Star Metric?What is a North Star Metric?
What is a North Star Metric?
 
Identifying your North Star Metric & Building a Model to Predict Growth
Identifying your North Star Metric & Building a Model to Predict GrowthIdentifying your North Star Metric & Building a Model to Predict Growth
Identifying your North Star Metric & Building a Model to Predict Growth
 
Dropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deck
Dropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deckDropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deck
Dropbox: $15K VC investment turned into $16.8B. Dropbox's initial pitch deck
 
WTF is a Product Roadmap?
WTF is a Product Roadmap?WTF is a Product Roadmap?
WTF is a Product Roadmap?
 
SalesforceにおけるCDC(変更データキャプチャ)の実装・活用法について
SalesforceにおけるCDC(変更データキャプチャ)の実装・活用法についてSalesforceにおけるCDC(変更データキャプチャ)の実装・活用法について
SalesforceにおけるCDC(変更データキャプチャ)の実装・活用法について
 
Rewind Pitch Deck
Rewind Pitch DeckRewind Pitch Deck
Rewind Pitch Deck
 
A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...
A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...
A Playbook for Achieving Product-Market Fit by Dan Olsen at Lean Startup Conf...
 
How to Build a Robust Product Roadmap by Salesforce VP of Product
How to Build a Robust Product Roadmap by Salesforce VP of ProductHow to Build a Robust Product Roadmap by Salesforce VP of Product
How to Build a Robust Product Roadmap by Salesforce VP of Product
 
10分でわかったつもりになるLean Analytics_10min lean analytics
10分でわかったつもりになるLean Analytics_10min lean analytics10分でわかったつもりになるLean Analytics_10min lean analytics
10分でわかったつもりになるLean Analytics_10min lean analytics
 
「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-
「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-
「開発がスクラム導入するんだって!試験どーしよ!?」 -サイボウズQAスクラム奮闘記-
 

Similar to Test & Learn: How to Find Your Product's North Star Metric

Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...
Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...
Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...Amplitude
 
Product Analytics Workshop
Product Analytics WorkshopProduct Analytics Workshop
Product Analytics WorkshopAmplitude
 
Powering the Next Generation of Products by Amplitude Principal of Product Ma...
Powering the Next Generation of Products by Amplitude Principal of Product Ma...Powering the Next Generation of Products by Amplitude Principal of Product Ma...
Powering the Next Generation of Products by Amplitude Principal of Product Ma...Product School
 
Leverage The Power of Small Data
Leverage The Power of Small DataLeverage The Power of Small Data
Leverage The Power of Small DataKaryn Zuidinga
 
How to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessHow to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessKissmetrics on SlideShare
 
Product Era is Here
Product Era is HereProduct Era is Here
Product Era is HereAmplitude
 
The five essential steps to building a data product
The five essential steps to building a data productThe five essential steps to building a data product
The five essential steps to building a data productBirst
 
SpigitEngage_Datasheet
SpigitEngage_DatasheetSpigitEngage_Datasheet
SpigitEngage_DatasheetConor O'Gorman
 
Market Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdfMarket Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdfAneel Mitra
 
How GetNinjas uses data to make smarter product decisions
How GetNinjas uses data to make smarter product decisionsHow GetNinjas uses data to make smarter product decisions
How GetNinjas uses data to make smarter product decisionsBernardo Srulzon
 
How to Develop New Products by Financial Times Head of Product
How to Develop New Products by Financial Times Head of ProductHow to Develop New Products by Financial Times Head of Product
How to Develop New Products by Financial Times Head of ProductProduct School
 
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...Optimizely
 
Why We Win (Or Lose): Powering Your CI Program with Win/Loss Analysis
Why We Win (Or Lose): Powering Your CI Program with Win/Loss AnalysisWhy We Win (Or Lose): Powering Your CI Program with Win/Loss Analysis
Why We Win (Or Lose): Powering Your CI Program with Win/Loss AnalysisKristine Culp
 
Spark a CX revolution: tips from the trenches
Spark a CX revolution: tips from the trenchesSpark a CX revolution: tips from the trenches
Spark a CX revolution: tips from the trenchesUserTesting
 
What's Next: The Value of Data
What's Next: The Value of DataWhat's Next: The Value of Data
What's Next: The Value of DataOgilvy Consulting
 
Intro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product ManagerIntro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product ManagerProduct School
 
A Practical Approach to Agile Methodology by Weedmaps Sr PM
A Practical Approach to Agile Methodology by Weedmaps Sr PMA Practical Approach to Agile Methodology by Weedmaps Sr PM
A Practical Approach to Agile Methodology by Weedmaps Sr PMProduct School
 
How to Get Started with Gamification
How to Get Started with GamificationHow to Get Started with Gamification
How to Get Started with GamificationBadgeville, Inc.
 
A Product-Led Growth Strategy for Enterprise
A Product-Led Growth Strategy for EnterpriseA Product-Led Growth Strategy for Enterprise
A Product-Led Growth Strategy for EnterpriseRocketSource
 

Similar to Test & Learn: How to Find Your Product's North Star Metric (20)

Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...
Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...
Product Executive Roundtable hosted by Sandhya Hegde, Director of Product, Am...
 
Product Analytics Workshop
Product Analytics WorkshopProduct Analytics Workshop
Product Analytics Workshop
 
Powering the Next Generation of Products by Amplitude Principal of Product Ma...
Powering the Next Generation of Products by Amplitude Principal of Product Ma...Powering the Next Generation of Products by Amplitude Principal of Product Ma...
Powering the Next Generation of Products by Amplitude Principal of Product Ma...
 
Leverage The Power of Small Data
Leverage The Power of Small DataLeverage The Power of Small Data
Leverage The Power of Small Data
 
How to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your BusinessHow to Use Data to Inform Your Design and Drive Your Business
How to Use Data to Inform Your Design and Drive Your Business
 
Product Era is Here
Product Era is HereProduct Era is Here
Product Era is Here
 
The five essential steps to building a data product
The five essential steps to building a data productThe five essential steps to building a data product
The five essential steps to building a data product
 
SpigitEngage_Datasheet
SpigitEngage_DatasheetSpigitEngage_Datasheet
SpigitEngage_Datasheet
 
Market Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdfMarket Validation (Scale) (4).pdf
Market Validation (Scale) (4).pdf
 
How GetNinjas uses data to make smarter product decisions
How GetNinjas uses data to make smarter product decisionsHow GetNinjas uses data to make smarter product decisions
How GetNinjas uses data to make smarter product decisions
 
How to Develop New Products by Financial Times Head of Product
How to Develop New Products by Financial Times Head of ProductHow to Develop New Products by Financial Times Head of Product
How to Develop New Products by Financial Times Head of Product
 
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
7 Habits of Highly Effective Personalisation Organisations | Optimizely ANZ W...
 
Product-Led In Practice
Product-Led In PracticeProduct-Led In Practice
Product-Led In Practice
 
Why We Win (Or Lose): Powering Your CI Program with Win/Loss Analysis
Why We Win (Or Lose): Powering Your CI Program with Win/Loss AnalysisWhy We Win (Or Lose): Powering Your CI Program with Win/Loss Analysis
Why We Win (Or Lose): Powering Your CI Program with Win/Loss Analysis
 
Spark a CX revolution: tips from the trenches
Spark a CX revolution: tips from the trenchesSpark a CX revolution: tips from the trenches
Spark a CX revolution: tips from the trenches
 
What's Next: The Value of Data
What's Next: The Value of DataWhat's Next: The Value of Data
What's Next: The Value of Data
 
Intro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product ManagerIntro to Product Management by Trunk Club Product Manager
Intro to Product Management by Trunk Club Product Manager
 
A Practical Approach to Agile Methodology by Weedmaps Sr PM
A Practical Approach to Agile Methodology by Weedmaps Sr PMA Practical Approach to Agile Methodology by Weedmaps Sr PM
A Practical Approach to Agile Methodology by Weedmaps Sr PM
 
How to Get Started with Gamification
How to Get Started with GamificationHow to Get Started with Gamification
How to Get Started with Gamification
 
A Product-Led Growth Strategy for Enterprise
A Product-Led Growth Strategy for EnterpriseA Product-Led Growth Strategy for Enterprise
A Product-Led Growth Strategy for Enterprise
 

More from Optimizely

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationOptimizely
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Optimizely
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing RightOptimizely
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleOptimizely
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Optimizely
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueOptimizely
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsOptimizely
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Optimizely
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Optimizely
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingOptimizely
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideOptimizely
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentOptimizely
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOptimizely
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...Optimizely
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyOptimizely
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueOptimizely
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' EyesOptimizely
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubOptimizely
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationOptimizely
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely
 

More from Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server Side
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 

Recently uploaded

08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptxLBM Solutions
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxMaking_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetHyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetEnjoy Anytime
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Artificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning eraArtificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning eraDeakin University
 
Snow Chain-Integrated Tire for a Safe Drive on Winter Roads
Snow Chain-Integrated Tire for a Safe Drive on Winter RoadsSnow Chain-Integrated Tire for a Safe Drive on Winter Roads
Snow Chain-Integrated Tire for a Safe Drive on Winter RoadsHyundai Motor Group
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 

Recently uploaded (20)

08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Key Features Of Token Development (1).pptx
Key  Features Of Token  Development (1).pptxKey  Features Of Token  Development (1).pptx
Key Features Of Token Development (1).pptx
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptx
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxMaking_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your BudgetHyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
Hyderabad Call Girls Khairatabad ✨ 7001305949 ✨ Cheap Price Your Budget
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
The transition to renewables in India.pdf
The transition to renewables in India.pdfThe transition to renewables in India.pdf
The transition to renewables in India.pdf
 
Artificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning eraArtificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning era
 
Snow Chain-Integrated Tire for a Safe Drive on Winter Roads
Snow Chain-Integrated Tire for a Safe Drive on Winter RoadsSnow Chain-Integrated Tire for a Safe Drive on Winter Roads
Snow Chain-Integrated Tire for a Safe Drive on Winter Roads
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 

Test & Learn: How to Find Your Product's North Star Metric

  • 1. Alicia Shiu Growth Product Manager, Amplitude How to Find Your Product's North Star Metric Tune in on Twitter: #testandlearn19
  • 2. Biomedical Engineering Growth Product Manager Neuroscience Alicia Shiu Growth Product Manager 30K Products Powered >7T User Actions Analyzed Marketing About me..
  • 3. How does your product development team define success?
  • 4. # of features shipped did we ship them on time The most successful product teams measure impact on business outcomes, not outputs
  • 5. Lower barriers to entry Massive explosion in product choice & noise Customer experience and value trump everything Product is the best competitive advantage Product-led companies are winning 5m+ apps 0.01% survival rate
  • 6. What does it mean to be Product-led? Vision of Solving Customer Pain Revenue and Shareholder Value
  • 7. Impact the CEO cares about Customer value created by the Product team Product Analytics Alerts App Redesign New User Activation Help Product LaunchTrial Conversion Profile UX North Star
  • 8. The 3 Games of Product What Makes a Good North Star? Evolving Amplitude’s North Star From North Star to Experimentation → → → → Finding Your North Star
  • 9. The 3 Games of Product What Makes a Good North Star? Evolving Amplitude’s North Star From North Star to Experimentation → → → → Finding Your North Star
  • 11. The 3 games of Product attention transaction productivity
  • 12. What game are you playing? attention transaction productivity
  • 13. Define the game you’re playing attention Typical Product North Stars Examples ● Total time spent engaging with content ● # Users spending > 5 min/day ● # Stories viewed ● # Subscribers streaming > 60 hrs/mo hypothetical
  • 14. Define the game you’re playing Typical Product North Stars Examples ● Total # of Purchases completed ● Avg # Purchases per Prime Subscriber ● # Searches completed ● # Seat Upgrades purchasedtransaction hypothetical
  • 15. Define the game you’re playing Typical Product North Stars Examples ● Total # of records created ● % of Emails engaged ● # of messages sent ● # Active Users who logged/completed a task productivity hypothetical
  • 16. The 3 Games of Product What Makes a Good North Star? Evolving Amplitude’s North Star From North Star to Experimentation → → → → Finding Your North Star
  • 17. 3 attributes of a good North Star metric Measure product engagement that is a LEADING INDICATOR of revenue Reflect the VALUE that customers are getting from your product Represent the core product outcome you need to ALIGN with your company strategy 1 2 3
  • 18. The 3 Games of Product What Makes a Good North Star? Evolving Amplitude’s North Star From North Star to Experimentation → → → → Finding Your North Star
  • 22. ? As end user adoption increases, businesses exchange greater value with Amplitude Customers get value by asking questions and sharing learnings Our unique strategy is to enable data democracy in companies 1 2 3
  • 23. Weekly Learning Users (learnings shared with > 2 people) As end user adoption increases, businesses exchange greater value with Amplitude Customers get value by asking questions and sharing learnings Our unique strategy is to enable data democracy in companies 1 2 3
  • 24. Getting closer to customer value Users that ask a question (Weekly Querying Users)
  • 25. Getting closer to customer value Users that share learnings (Weekly Learning Users) Users that ask a question (Weekly Querying Users) Active and had a learning shared w/ at least 2 people in last 7 days
  • 26. The 3 Games of Product What Makes a Good North Star? Evolving Amplitude’s North Star From North Star to Experimentation → → → → Finding Your North Star
  • 27. Product experimentation to impact your North Star Hypothesis: More user profile context & relevancy signals will encourage users to click into Team Space and view team members’ content
  • 28. Product experimentation to impact your North Star Hypothesis: More user profile context & relevancy signals will encourage users to click into Team Space and view team members’ content Control Test
  • 29. Getting started ● Define your product’s (or team’s) North Star ● Create a hypothesis ● Kick off a new (small) project (or align a current one) ● Test the hypothesis to see if you can move the metric!
  • 30. Define which game you’re playing: attention, transaction, or productivity. Articulate tradeoffs you’re willing to make. A good North Star reflects customer value, aligns product teams with your company strategy, and is a leading indicator of revenue. Organize product experimentation to move your North Star metric (or inputs / leading indicators of it). → → → Takeaways
  • 31. Keep Learning! For more details on the 3 product games, ladder of engagement, & loops: https://amplitude.com/mastering-engagement Additional reading on North Star metrics: - https://amplitude.com/blog/evolving-the- product-north-star-metric - https://amplitude.com/blog/evolving-the- product-north-star-metric
  • 34. Breadth Depth Frequency Efficiencyx x x # Items within Order # Orders completed % Items Delivered within time # Users placing Orders each month Break the North Star down into actionable input metrics hypothetical Total monthly items received on time
  • 35. Actionable inputs: Efficiency Breadth Depth Frequency Efficiencyx x x # Items within Order # Orders Completed % Items Delivered within time ML Algorithm Customer chat Flash Sales Notifications Stock mgt Recommendations Shopping Lists Discount Coupons New Signups Express Sub Trials Trial Conversion # Users placing Orders each month Roadmap hypothetical Total monthly items received on time
  • 36. Next Session Moving Fast, Breaking Things, and Fixing Them as Quickly as Possible Lukas Vermeer Director of Experimentation, Booking.com Tune in on Twitter: #testandlearn19