SlideShare a Scribd company logo
Why Product Managers
Should Not Be Data-Driven
Jens-Fabian Goetzmann
Head of Product, RevenueCat
#mtpcon London + EMEA
13 October 2022
In the olden days…
Software shipped on disks
No ability to collect and
analyze usage data
No ability to change product
after shipping
Long feedback and shipping
cycles
Today: treasure trove of data, rapid feedback cycles
Rapid shipping Analytics tooling
Agile & lean processes
SaaS
DevOps & CI/CD
Lean
Startup
Agile /
Scrum
The appeal of data-driven decisions
Objective
Neutral
Fast
Unambiguous
Data doesn’t fall prey to cognitive biases or preconceived notions
(“data doesn’t lie”)
Data doesn’t take sides; it doesn’t matter if it agrees with the
CEO’s opinion (HiPPO) or not
You can generate data as fast as you can ship, and don’t need to have
access to customers
It’s easy to compare two numbers and see which one is bigger, qualitative
human reactions are harder to compare
Data is…
The rise of the data-driven product manager
Being data-driven is a trap
It’s a trap!
Why being data-driven is a trap
Goodhart’s Law and the difficulties of measuring
Local maximum problem and incrementality
Thinking opportunistically instead of strategically
Analysis paralysis: Perfect is the enemy of done
1
2
3
4
Goodhart’s Law and the difficulties of measuring
“When a measure becomes a target,
it ceases to be a good measure.”
Engagement
E.g., if we target increasing…
Revenue
Profit
We might end up sending a bunch of push notifications that have no actual value
We might be incentivized to grow inorganically with bad unit economics
We may cut cost and underinvest in the future
Regardless of what you measure, you risk over-optimizing for that measure and for the short term.
Local maximum problem and incrementality
Teams will prefer incremental solutions, since it’s easier to find “winners”
● Being data-driven means acting opportunistically, whatever moves KPIs
● KPIs are not a strategy – at most, they proxy progress toward a strategy
● Strategy begins with deep insight into the problem to be solved, and a core
hypothesis how the problem can be better solved than today
● Acting strategically means pursuing the hypothesis even after setbacks—
if it was obviously true, someone else would have already built the product!
Thinking opportunistically instead of strategically
You can’t A/B test your way to a great product!
Analysis paralysis: Perfect is the enemy of done
● Product development is fundamentally risky and uncertain
● Being data-driven can lead to wanting to collect data for every decision
● This can lead to analysis paralysis, not moving forward unless there is sufficient
and unambiguous data
The solution: data-informed, not data-driven
Quantitative
analysis
Qualitative
research
data-informed
Strategy and
product vision
The role of strategy and product vision
Product strategy Product vision
Diagnosis of threats & opportunities
Focused guiding policy
Coherent actions to deploy capital
Forward-looking picture of how the
product will deliver on the strategy
Quantitative and qualitative research
In service of product strategy & product vision
Feedback
● To generate hypotheses to test: Deeply understand customer goals and problems
● To get to the “why”: Customer behavior is hard to understand from just metrics movement
● To open up new opportunities: Make your product solve problems of tomorrow’s customers
● To validate where data is unreliable: E.g., low volume, sales blocker, interaction effects
● To get answers faster: Validate prototypes without coding and shipping
● To keep you humble: Check your own biases—you don’t know all customers & use cases
Why you need qualitative research to complement data
Use the right tool for the job
Innovation Optimization
Success criteria are unknown
High uncertainty
Few big bets
High potential reward
Success metrics well known
Relatively little uncertainty
Many small bets
Incremental gains
Mostly qualitative research Mostly quantitative analytics
How quantitative and qualitative insights work together
Quantitative
Find potential product issues Understand product issues
Test and analyze impact Discover opportunities
Hypothesize impact Validate solution concepts
Qualitative
Jens-Fabian Goetzmann
jefago.com @MrJefago
Thank you! Time for Q&A
Quantitative
analysis
Qualitative
research
data-informed
Strategy and
product vision

More Related Content

What's hot

Become a Great Product Manager
Become a Great Product ManagerBecome a Great Product Manager
Become a Great Product Manager
Roman Pichler
 
How to Build a Product Roadmap by eBay Director of Product
How to Build a Product Roadmap by eBay Director of ProductHow to Build a Product Roadmap by eBay Director of Product
How to Build a Product Roadmap by eBay Director of Product
Product School
 
How to Build a Product Roadmap by Walmart Senior Product Manager
How to Build a Product Roadmap by Walmart Senior Product ManagerHow to Build a Product Roadmap by Walmart Senior Product Manager
How to Build a Product Roadmap by Walmart Senior Product Manager
Product School
 
How To Optimize Your Product Using Analytics by Dan Olsen
How To Optimize Your Product Using Analytics by Dan OlsenHow To Optimize Your Product Using Analytics by Dan Olsen
How To Optimize Your Product Using Analytics by Dan Olsen
Product School
 
The Lean Product Playbook with Dan Olsen
The Lean Product Playbook with Dan OlsenThe Lean Product Playbook with Dan Olsen
The Lean Product Playbook with Dan Olsen
Product School
 
Customer First Creating data-driven products with a human touch by Deliveroo ...
Customer First Creating data-driven products with a human touch by Deliveroo ...Customer First Creating data-driven products with a human touch by Deliveroo ...
Customer First Creating data-driven products with a human touch by Deliveroo ...
Product School
 
[Product Camp 2021] Product Roadmaps Relaunched
[Product Camp 2021] Product Roadmaps Relaunched[Product Camp 2021] Product Roadmaps Relaunched
[Product Camp 2021] Product Roadmaps Relaunched
Product Camp Brasil
 
Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth.
Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth. Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth.
Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth.
Traction Conf
 
What Are the Basics of Product Manager Interviews by Google PM
What Are the Basics of Product Manager Interviews by Google PMWhat Are the Basics of Product Manager Interviews by Google PM
What Are the Basics of Product Manager Interviews by Google PM
Product School
 
Data Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thData Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8th
Jonathan Woodward
 
Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldaba
Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika AldabaLightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldaba
Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldaba
ux singapore
 
Y Combinator Seed Deck Template
Y Combinator Seed Deck TemplateY Combinator Seed Deck Template
Y Combinator Seed Deck Template
Razin Mustafiz
 
Startups are Hard. Like, Really Hard. @luketucker
Startups are Hard. Like, Really Hard. @luketuckerStartups are Hard. Like, Really Hard. @luketucker
Startups are Hard. Like, Really Hard. @luketucker
Empowered Presentations
 
Developing a Product Vision by Amazon Sr Product Manager
Developing a Product Vision by Amazon Sr Product ManagerDeveloping a Product Vision by Amazon Sr Product Manager
Developing a Product Vision by Amazon Sr Product Manager
Product School
 
Product Roadmap Prioritization by Amazon Principal PM
Product Roadmap Prioritization by Amazon Principal PMProduct Roadmap Prioritization by Amazon Principal PM
Product Roadmap Prioritization by Amazon Principal PM
Product School
 
Build Features, Not Apps
Build Features, Not AppsBuild Features, Not Apps
Build Features, Not Apps
Natasha Murashev
 
Diversity and high-performing teams by TIER mobility CPO.pdf
Diversity and high-performing teams by TIER mobility CPO.pdfDiversity and high-performing teams by TIER mobility CPO.pdf
Diversity and high-performing teams by TIER mobility CPO.pdf
Product School
 
Inclusive design workshop
Inclusive design workshopInclusive design workshop
Inclusive design workshop
Brittany Mederos
 
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Dan Olsen
 
User Research at GDS
User Research at GDSUser Research at GDS
User Research at GDS
William Roissetter
 

What's hot (20)

Become a Great Product Manager
Become a Great Product ManagerBecome a Great Product Manager
Become a Great Product Manager
 
How to Build a Product Roadmap by eBay Director of Product
How to Build a Product Roadmap by eBay Director of ProductHow to Build a Product Roadmap by eBay Director of Product
How to Build a Product Roadmap by eBay Director of Product
 
How to Build a Product Roadmap by Walmart Senior Product Manager
How to Build a Product Roadmap by Walmart Senior Product ManagerHow to Build a Product Roadmap by Walmart Senior Product Manager
How to Build a Product Roadmap by Walmart Senior Product Manager
 
How To Optimize Your Product Using Analytics by Dan Olsen
How To Optimize Your Product Using Analytics by Dan OlsenHow To Optimize Your Product Using Analytics by Dan Olsen
How To Optimize Your Product Using Analytics by Dan Olsen
 
The Lean Product Playbook with Dan Olsen
The Lean Product Playbook with Dan OlsenThe Lean Product Playbook with Dan Olsen
The Lean Product Playbook with Dan Olsen
 
Customer First Creating data-driven products with a human touch by Deliveroo ...
Customer First Creating data-driven products with a human touch by Deliveroo ...Customer First Creating data-driven products with a human touch by Deliveroo ...
Customer First Creating data-driven products with a human touch by Deliveroo ...
 
[Product Camp 2021] Product Roadmaps Relaunched
[Product Camp 2021] Product Roadmaps Relaunched[Product Camp 2021] Product Roadmaps Relaunched
[Product Camp 2021] Product Roadmaps Relaunched
 
Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth.
Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth. Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth.
Aatif Awan, Head of Growth LinkedIn - Growth Hacking is Dead. Long Live Growth.
 
What Are the Basics of Product Manager Interviews by Google PM
What Are the Basics of Product Manager Interviews by Google PMWhat Are the Basics of Product Manager Interviews by Google PM
What Are the Basics of Product Manager Interviews by Google PM
 
Data Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8thData Culture Keynote and Exec Track Birm Dec 8th
Data Culture Keynote and Exec Track Birm Dec 8th
 
Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldaba
Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika AldabaLightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldaba
Lightning Talk #9: How UX and Data Storytelling Can Shape Policy by Mika Aldaba
 
Y Combinator Seed Deck Template
Y Combinator Seed Deck TemplateY Combinator Seed Deck Template
Y Combinator Seed Deck Template
 
Startups are Hard. Like, Really Hard. @luketucker
Startups are Hard. Like, Really Hard. @luketuckerStartups are Hard. Like, Really Hard. @luketucker
Startups are Hard. Like, Really Hard. @luketucker
 
Developing a Product Vision by Amazon Sr Product Manager
Developing a Product Vision by Amazon Sr Product ManagerDeveloping a Product Vision by Amazon Sr Product Manager
Developing a Product Vision by Amazon Sr Product Manager
 
Product Roadmap Prioritization by Amazon Principal PM
Product Roadmap Prioritization by Amazon Principal PMProduct Roadmap Prioritization by Amazon Principal PM
Product Roadmap Prioritization by Amazon Principal PM
 
Build Features, Not Apps
Build Features, Not AppsBuild Features, Not Apps
Build Features, Not Apps
 
Diversity and high-performing teams by TIER mobility CPO.pdf
Diversity and high-performing teams by TIER mobility CPO.pdfDiversity and high-performing teams by TIER mobility CPO.pdf
Diversity and high-performing teams by TIER mobility CPO.pdf
 
Inclusive design workshop
Inclusive design workshopInclusive design workshop
Inclusive design workshop
 
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
Product Management by Numbers: Using Metrics To Optimize Your Product by Dan ...
 
User Research at GDS
User Research at GDSUser Research at GDS
User Research at GDS
 

Similar to mtpcon London+EMEA 2022 – Why Product Managers should not be data-driven.pdf

Giving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business QuestionsGiving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business Questions
OReillyStrata
 
The Hidden ABCs of Product Management: Reveal Your Product Blind Spots
The Hidden ABCs of Product Management: Reveal Your Product Blind SpotsThe Hidden ABCs of Product Management: Reveal Your Product Blind Spots
The Hidden ABCs of Product Management: Reveal Your Product Blind Spots
Wes Galliher
 
Giving Organisations new capabilities to ask the right business questions 1.7
Giving Organisations new capabilities to ask the right business questions 1.7Giving Organisations new capabilities to ask the right business questions 1.7
Giving Organisations new capabilities to ask the right business questions 1.7
OReillyStrata
 
PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)
PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)
PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)
Chief Analytics Officer Forum
 
Webinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PM
Webinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PMWebinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PM
Webinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PM
Product School
 
Measuring What Matters by Google PM
Measuring What Matters by Google PMMeasuring What Matters by Google PM
Measuring What Matters by Google PM
Product School
 
Nancy's webinar
Nancy's webinarNancy's webinar
Nancy's webinar
VWO
 
1555 track 1 huang_using his mac
1555 track 1 huang_using his mac1555 track 1 huang_using his mac
1555 track 1 huang_using his mac
Rising Media, Inc.
 
Common Data Driven Mistakes with HAVI's Sr. Director of Advanced Analytics
Common Data Driven Mistakes with HAVI's Sr. Director of Advanced AnalyticsCommon Data Driven Mistakes with HAVI's Sr. Director of Advanced Analytics
Common Data Driven Mistakes with HAVI's Sr. Director of Advanced Analytics
Promotable
 
Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22
Matthias Schuurmans
 
MonetizingStatistics
MonetizingStatisticsMonetizingStatistics
MonetizingStatistics
Aaron Sankey
 
NPD hypothesis validation toolkit
NPD hypothesis validation toolkit NPD hypothesis validation toolkit
NPD hypothesis validation toolkit
CX NPD ltd
 
Leverage The Power of Small Data
Leverage The Power of Small DataLeverage The Power of Small Data
Leverage The Power of Small Data
Karyn Zuidinga
 
Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...
Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...
Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...
Lviv Startup Club
 
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The Data Driven Enterprise - Roadmap to Big Data & Analytics SuccessThe Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
BigInsights
 
Why Predictive Analytics Should Be Part of Your 2015 Strategy Final
Why Predictive Analytics Should Be Part of Your 2015 Strategy FinalWhy Predictive Analytics Should Be Part of Your 2015 Strategy Final
Why Predictive Analytics Should Be Part of Your 2015 Strategy Final
Joe Brandenburg
 
Big data analytics and innovation
Big data analytics and innovationBig data analytics and innovation
Big data analytics and innovation
Ahmed Fattah
 
How to succeed at data without even trying!
How to succeed at data without even trying!How to succeed at data without even trying!
How to succeed at data without even trying!
Dylan
 
Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...
Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...
Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...
Jamie Clouting (CSPO)
 
Better Living Through Analytics - Strategies for Data Decisions
Better Living Through Analytics - Strategies for Data DecisionsBetter Living Through Analytics - Strategies for Data Decisions
Better Living Through Analytics - Strategies for Data Decisions
Product School
 

Similar to mtpcon London+EMEA 2022 – Why Product Managers should not be data-driven.pdf (20)

Giving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business QuestionsGiving Organisations new Capabilities to ask the Right Business Questions
Giving Organisations new Capabilities to ask the Right Business Questions
 
The Hidden ABCs of Product Management: Reveal Your Product Blind Spots
The Hidden ABCs of Product Management: Reveal Your Product Blind SpotsThe Hidden ABCs of Product Management: Reveal Your Product Blind Spots
The Hidden ABCs of Product Management: Reveal Your Product Blind Spots
 
Giving Organisations new capabilities to ask the right business questions 1.7
Giving Organisations new capabilities to ask the right business questions 1.7Giving Organisations new capabilities to ask the right business questions 1.7
Giving Organisations new capabilities to ask the right business questions 1.7
 
PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)
PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)
PWC presentation at the Chief Analytics Officer Forum East Coast USA (#CAOForum)
 
Webinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PM
Webinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PMWebinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PM
Webinar: How to be More Effective Data-Driven PM by fmr Groupon Sr PM
 
Measuring What Matters by Google PM
Measuring What Matters by Google PMMeasuring What Matters by Google PM
Measuring What Matters by Google PM
 
Nancy's webinar
Nancy's webinarNancy's webinar
Nancy's webinar
 
1555 track 1 huang_using his mac
1555 track 1 huang_using his mac1555 track 1 huang_using his mac
1555 track 1 huang_using his mac
 
Common Data Driven Mistakes with HAVI's Sr. Director of Advanced Analytics
Common Data Driven Mistakes with HAVI's Sr. Director of Advanced AnalyticsCommon Data Driven Mistakes with HAVI's Sr. Director of Advanced Analytics
Common Data Driven Mistakes with HAVI's Sr. Director of Advanced Analytics
 
Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22Behind The Scenes Data Science Coolblue 2018-03-22
Behind The Scenes Data Science Coolblue 2018-03-22
 
MonetizingStatistics
MonetizingStatisticsMonetizingStatistics
MonetizingStatistics
 
NPD hypothesis validation toolkit
NPD hypothesis validation toolkit NPD hypothesis validation toolkit
NPD hypothesis validation toolkit
 
Leverage The Power of Small Data
Leverage The Power of Small DataLeverage The Power of Small Data
Leverage The Power of Small Data
 
Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...
Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...
Bogdan Onyshchenko: Як стати кращим Продакт Менеджером? 11 порад з особистого...
 
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The Data Driven Enterprise - Roadmap to Big Data & Analytics SuccessThe Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
 
Why Predictive Analytics Should Be Part of Your 2015 Strategy Final
Why Predictive Analytics Should Be Part of Your 2015 Strategy FinalWhy Predictive Analytics Should Be Part of Your 2015 Strategy Final
Why Predictive Analytics Should Be Part of Your 2015 Strategy Final
 
Big data analytics and innovation
Big data analytics and innovationBig data analytics and innovation
Big data analytics and innovation
 
How to succeed at data without even trying!
How to succeed at data without even trying!How to succeed at data without even trying!
How to succeed at data without even trying!
 
Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...
Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...
Going Beyond ‘What Success Looks Like’ – Using Data to Achieve Successful Pro...
 
Better Living Through Analytics - Strategies for Data Decisions
Better Living Through Analytics - Strategies for Data DecisionsBetter Living Through Analytics - Strategies for Data Decisions
Better Living Through Analytics - Strategies for Data Decisions
 

Recently uploaded

Chart--Time Management.pdf How to time is spent
Chart--Time Management.pdf How to time is spentChart--Time Management.pdf How to time is spent
Chart--Time Management.pdf How to time is spent
spandane
 
Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...
Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...
Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...
Alexey Krivitsky
 
一比一原版(QU毕业证)皇后大学毕业证如何办理
一比一原版(QU毕业证)皇后大学毕业证如何办理一比一原版(QU毕业证)皇后大学毕业证如何办理
一比一原版(QU毕业证)皇后大学毕业证如何办理
8p28uk6g
 
Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...
Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...
Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...
dsnow9802
 
Resource-mobilization-guide-for-community-based-organizations1.pdf
Resource-mobilization-guide-for-community-based-organizations1.pdfResource-mobilization-guide-for-community-based-organizations1.pdf
Resource-mobilization-guide-for-community-based-organizations1.pdf
FeteneA
 
Integrity in leadership builds trust by ensuring consistency between words an...
Integrity in leadership builds trust by ensuring consistency between words an...Integrity in leadership builds trust by ensuring consistency between words an...
Integrity in leadership builds trust by ensuring consistency between words an...
Ram V Chary
 
All the Small Things - XP2024 Bolzano/Bozen
All the Small Things - XP2024 Bolzano/BozenAll the Small Things - XP2024 Bolzano/Bozen
All the Small Things - XP2024 Bolzano/Bozen
Alberto Brandolini
 
Ganpati Kumar Choudhary Indian Ethos PPT.pptx
Ganpati Kumar Choudhary Indian Ethos PPT.pptxGanpati Kumar Choudhary Indian Ethos PPT.pptx
Ganpati Kumar Choudhary Indian Ethos PPT.pptx
GanpatiKumarChoudhar
 
Sethurathnam Ravi: A Legacy in Finance and Leadership
Sethurathnam Ravi: A Legacy in Finance and LeadershipSethurathnam Ravi: A Legacy in Finance and Leadership
Sethurathnam Ravi: A Legacy in Finance and Leadership
Anjana Josie
 
Myrna Story of Leadership and Management
Myrna Story of Leadership and ManagementMyrna Story of Leadership and Management
Myrna Story of Leadership and Management
Waleed Kamal
 
20240608 QFM019 Engineering Leadership Reading List May 2024
20240608 QFM019 Engineering Leadership Reading List May 202420240608 QFM019 Engineering Leadership Reading List May 2024
20240608 QFM019 Engineering Leadership Reading List May 2024
Matthew Sinclair
 
Stuart Wilson the teams I have led - 2024
Stuart Wilson the teams I have led - 2024Stuart Wilson the teams I have led - 2024
Stuart Wilson the teams I have led - 2024
stuwilson.co.uk
 
Strategic Org Design with Org Topologies™
Strategic Org Design with Org Topologies™Strategic Org Design with Org Topologies™
Strategic Org Design with Org Topologies™
Alexey Krivitsky
 
Addiction to Winning Across Diverse Populations.pdf
Addiction to Winning Across Diverse Populations.pdfAddiction to Winning Across Diverse Populations.pdf
Addiction to Winning Across Diverse Populations.pdf
Bill641377
 
The Management Guide: From Projects to Portfolio
The Management Guide: From Projects to PortfolioThe Management Guide: From Projects to Portfolio
The Management Guide: From Projects to Portfolio
Ahmed AbdelMoneim
 
Enriching engagement with ethical review processes
Enriching engagement with ethical review processesEnriching engagement with ethical review processes
Enriching engagement with ethical review processes
strikingabalance
 
Credit-Management seminar for cooperative power point presentation
Credit-Management seminar for cooperative power point presentationCredit-Management seminar for cooperative power point presentation
Credit-Management seminar for cooperative power point presentation
bernanbumatay1
 
innovation in nursing practice, education and management.pptx
innovation in nursing practice, education and management.pptxinnovation in nursing practice, education and management.pptx
innovation in nursing practice, education and management.pptx
TulsiDhidhi1
 
Risk-Management-presentation for cooperatives
Risk-Management-presentation for cooperativesRisk-Management-presentation for cooperatives
Risk-Management-presentation for cooperatives
bernanbumatay1
 
Employment Practices Regulation and Multinational Corporations
Employment PracticesRegulation and Multinational CorporationsEmployment PracticesRegulation and Multinational Corporations
Employment Practices Regulation and Multinational Corporations
RoopaTemkar
 

Recently uploaded (20)

Chart--Time Management.pdf How to time is spent
Chart--Time Management.pdf How to time is spentChart--Time Management.pdf How to time is spent
Chart--Time Management.pdf How to time is spent
 
Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...
Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...
Designing and Sustaining Large-Scale Value-Centered Agile Ecosystems (powered...
 
一比一原版(QU毕业证)皇后大学毕业证如何办理
一比一原版(QU毕业证)皇后大学毕业证如何办理一比一原版(QU毕业证)皇后大学毕业证如何办理
一比一原版(QU毕业证)皇后大学毕业证如何办理
 
Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...
Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...
Colby Hobson: Residential Construction Leader Building a Solid Reputation Thr...
 
Resource-mobilization-guide-for-community-based-organizations1.pdf
Resource-mobilization-guide-for-community-based-organizations1.pdfResource-mobilization-guide-for-community-based-organizations1.pdf
Resource-mobilization-guide-for-community-based-organizations1.pdf
 
Integrity in leadership builds trust by ensuring consistency between words an...
Integrity in leadership builds trust by ensuring consistency between words an...Integrity in leadership builds trust by ensuring consistency between words an...
Integrity in leadership builds trust by ensuring consistency between words an...
 
All the Small Things - XP2024 Bolzano/Bozen
All the Small Things - XP2024 Bolzano/BozenAll the Small Things - XP2024 Bolzano/Bozen
All the Small Things - XP2024 Bolzano/Bozen
 
Ganpati Kumar Choudhary Indian Ethos PPT.pptx
Ganpati Kumar Choudhary Indian Ethos PPT.pptxGanpati Kumar Choudhary Indian Ethos PPT.pptx
Ganpati Kumar Choudhary Indian Ethos PPT.pptx
 
Sethurathnam Ravi: A Legacy in Finance and Leadership
Sethurathnam Ravi: A Legacy in Finance and LeadershipSethurathnam Ravi: A Legacy in Finance and Leadership
Sethurathnam Ravi: A Legacy in Finance and Leadership
 
Myrna Story of Leadership and Management
Myrna Story of Leadership and ManagementMyrna Story of Leadership and Management
Myrna Story of Leadership and Management
 
20240608 QFM019 Engineering Leadership Reading List May 2024
20240608 QFM019 Engineering Leadership Reading List May 202420240608 QFM019 Engineering Leadership Reading List May 2024
20240608 QFM019 Engineering Leadership Reading List May 2024
 
Stuart Wilson the teams I have led - 2024
Stuart Wilson the teams I have led - 2024Stuart Wilson the teams I have led - 2024
Stuart Wilson the teams I have led - 2024
 
Strategic Org Design with Org Topologies™
Strategic Org Design with Org Topologies™Strategic Org Design with Org Topologies™
Strategic Org Design with Org Topologies™
 
Addiction to Winning Across Diverse Populations.pdf
Addiction to Winning Across Diverse Populations.pdfAddiction to Winning Across Diverse Populations.pdf
Addiction to Winning Across Diverse Populations.pdf
 
The Management Guide: From Projects to Portfolio
The Management Guide: From Projects to PortfolioThe Management Guide: From Projects to Portfolio
The Management Guide: From Projects to Portfolio
 
Enriching engagement with ethical review processes
Enriching engagement with ethical review processesEnriching engagement with ethical review processes
Enriching engagement with ethical review processes
 
Credit-Management seminar for cooperative power point presentation
Credit-Management seminar for cooperative power point presentationCredit-Management seminar for cooperative power point presentation
Credit-Management seminar for cooperative power point presentation
 
innovation in nursing practice, education and management.pptx
innovation in nursing practice, education and management.pptxinnovation in nursing practice, education and management.pptx
innovation in nursing practice, education and management.pptx
 
Risk-Management-presentation for cooperatives
Risk-Management-presentation for cooperativesRisk-Management-presentation for cooperatives
Risk-Management-presentation for cooperatives
 
Employment Practices Regulation and Multinational Corporations
Employment PracticesRegulation and Multinational CorporationsEmployment PracticesRegulation and Multinational Corporations
Employment Practices Regulation and Multinational Corporations
 

mtpcon London+EMEA 2022 – Why Product Managers should not be data-driven.pdf

  • 1. Why Product Managers Should Not Be Data-Driven Jens-Fabian Goetzmann Head of Product, RevenueCat #mtpcon London + EMEA 13 October 2022
  • 2. In the olden days… Software shipped on disks No ability to collect and analyze usage data No ability to change product after shipping Long feedback and shipping cycles
  • 3. Today: treasure trove of data, rapid feedback cycles Rapid shipping Analytics tooling Agile & lean processes SaaS DevOps & CI/CD Lean Startup Agile / Scrum
  • 4. The appeal of data-driven decisions Objective Neutral Fast Unambiguous Data doesn’t fall prey to cognitive biases or preconceived notions (“data doesn’t lie”) Data doesn’t take sides; it doesn’t matter if it agrees with the CEO’s opinion (HiPPO) or not You can generate data as fast as you can ship, and don’t need to have access to customers It’s easy to compare two numbers and see which one is bigger, qualitative human reactions are harder to compare Data is…
  • 5. The rise of the data-driven product manager
  • 6. Being data-driven is a trap It’s a trap!
  • 7. Why being data-driven is a trap Goodhart’s Law and the difficulties of measuring Local maximum problem and incrementality Thinking opportunistically instead of strategically Analysis paralysis: Perfect is the enemy of done 1 2 3 4
  • 8. Goodhart’s Law and the difficulties of measuring “When a measure becomes a target, it ceases to be a good measure.” Engagement E.g., if we target increasing… Revenue Profit We might end up sending a bunch of push notifications that have no actual value We might be incentivized to grow inorganically with bad unit economics We may cut cost and underinvest in the future Regardless of what you measure, you risk over-optimizing for that measure and for the short term.
  • 9. Local maximum problem and incrementality Teams will prefer incremental solutions, since it’s easier to find “winners”
  • 10. ● Being data-driven means acting opportunistically, whatever moves KPIs ● KPIs are not a strategy – at most, they proxy progress toward a strategy ● Strategy begins with deep insight into the problem to be solved, and a core hypothesis how the problem can be better solved than today ● Acting strategically means pursuing the hypothesis even after setbacks— if it was obviously true, someone else would have already built the product! Thinking opportunistically instead of strategically You can’t A/B test your way to a great product!
  • 11. Analysis paralysis: Perfect is the enemy of done ● Product development is fundamentally risky and uncertain ● Being data-driven can lead to wanting to collect data for every decision ● This can lead to analysis paralysis, not moving forward unless there is sufficient and unambiguous data
  • 12. The solution: data-informed, not data-driven Quantitative analysis Qualitative research data-informed Strategy and product vision
  • 13. The role of strategy and product vision Product strategy Product vision Diagnosis of threats & opportunities Focused guiding policy Coherent actions to deploy capital Forward-looking picture of how the product will deliver on the strategy Quantitative and qualitative research In service of product strategy & product vision Feedback
  • 14. ● To generate hypotheses to test: Deeply understand customer goals and problems ● To get to the “why”: Customer behavior is hard to understand from just metrics movement ● To open up new opportunities: Make your product solve problems of tomorrow’s customers ● To validate where data is unreliable: E.g., low volume, sales blocker, interaction effects ● To get answers faster: Validate prototypes without coding and shipping ● To keep you humble: Check your own biases—you don’t know all customers & use cases Why you need qualitative research to complement data
  • 15. Use the right tool for the job Innovation Optimization Success criteria are unknown High uncertainty Few big bets High potential reward Success metrics well known Relatively little uncertainty Many small bets Incremental gains Mostly qualitative research Mostly quantitative analytics
  • 16. How quantitative and qualitative insights work together Quantitative Find potential product issues Understand product issues Test and analyze impact Discover opportunities Hypothesize impact Validate solution concepts Qualitative
  • 17. Jens-Fabian Goetzmann jefago.com @MrJefago Thank you! Time for Q&A Quantitative analysis Qualitative research data-informed Strategy and product vision