Product managers today have access to detailed data and analytics around how their product is being used. This has led to the rise of the "data-driven product manager" who bases their decisions on data as much as possible. In this talk, Jens explains why that is a trap, and what to do instead.

1. Why Product Managers
Should Not Be Data-Driven
Jens-Fabian Goetzmann
Head of Product, RevenueCat
#mtpcon London + EMEA
13 October 2022

2. In the olden days…
Software shipped on disks
No ability to collect and
analyze usage data
No ability to change product
after shipping
Long feedback and shipping
cycles

3. Today: treasure trove of data, rapid feedback cycles
Rapid shipping Analytics tooling
Agile & lean processes
SaaS
DevOps & CI/CD
Lean
Startup
Agile /
Scrum

4. The appeal of data-driven decisions
Objective
Neutral
Fast
Unambiguous
Data doesn’t fall prey to cognitive biases or preconceived notions
(“data doesn’t lie”)
Data doesn’t take sides; it doesn’t matter if it agrees with the
CEO’s opinion (HiPPO) or not
You can generate data as fast as you can ship, and don’t need to have
access to customers
It’s easy to compare two numbers and see which one is bigger, qualitative
human reactions are harder to compare
Data is…

5. The rise of the data-driven product manager

6. Being data-driven is a trap
It’s a trap!

7. Why being data-driven is a trap
Goodhart’s Law and the difficulties of measuring
Local maximum problem and incrementality
Thinking opportunistically instead of strategically
Analysis paralysis: Perfect is the enemy of done
1
2
3
4

8. Goodhart’s Law and the difficulties of measuring
“When a measure becomes a target,
it ceases to be a good measure.”
Engagement
E.g., if we target increasing…
Revenue
Profit
We might end up sending a bunch of push notifications that have no actual value
We might be incentivized to grow inorganically with bad unit economics
We may cut cost and underinvest in the future
Regardless of what you measure, you risk over-optimizing for that measure and for the short term.

9. Local maximum problem and incrementality
Teams will prefer incremental solutions, since it’s easier to find “winners”

10. ● Being data-driven means acting opportunistically, whatever moves KPIs
● KPIs are not a strategy – at most, they proxy progress toward a strategy
● Strategy begins with deep insight into the problem to be solved, and a core
hypothesis how the problem can be better solved than today
● Acting strategically means pursuing the hypothesis even after setbacks—
if it was obviously true, someone else would have already built the product!
Thinking opportunistically instead of strategically
You can’t A/B test your way to a great product!

11. Analysis paralysis: Perfect is the enemy of done
● Product development is fundamentally risky and uncertain
● Being data-driven can lead to wanting to collect data for every decision
● This can lead to analysis paralysis, not moving forward unless there is sufficient
and unambiguous data

12. The solution: data-informed, not data-driven
Quantitative
analysis
Qualitative
research
data-informed
Strategy and
product vision

13. The role of strategy and product vision
Product strategy Product vision
Diagnosis of threats & opportunities
Focused guiding policy
Coherent actions to deploy capital
Forward-looking picture of how the
product will deliver on the strategy
Quantitative and qualitative research
In service of product strategy & product vision
Feedback

14. ● To generate hypotheses to test: Deeply understand customer goals and problems
● To get to the “why”: Customer behavior is hard to understand from just metrics movement
● To open up new opportunities: Make your product solve problems of tomorrow’s customers
● To validate where data is unreliable: E.g., low volume, sales blocker, interaction effects
● To get answers faster: Validate prototypes without coding and shipping
● To keep you humble: Check your own biases—you don’t know all customers & use cases
Why you need qualitative research to complement data

15. Use the right tool for the job
Innovation Optimization
Success criteria are unknown
High uncertainty
Few big bets
High potential reward
Success metrics well known
Relatively little uncertainty
Many small bets
Incremental gains
Mostly qualitative research Mostly quantitative analytics

16. How quantitative and qualitative insights work together
Quantitative
Find potential product issues Understand product issues
Test and analyze impact Discover opportunities
Hypothesize impact Validate solution concepts
Qualitative

17. Jens-Fabian Goetzmann
jefago.com @MrJefago
Thank you! Time for Q&A
Quantitative
analysis
Qualitative
research
data-informed
Strategy and
product vision