This presentation discusses OKRs, KPIs, and hypothesis-driven product development. It emphasizes defining clear and measurable OKRs aligned to customer needs, with KPIs to track impact. OKRs and KPIs should guide all work and hold teams accountable. The presentation also discusses testing hypotheses, not just features, through experiments and A/B testing to validate assumptions before fully building out features. All work should be justified by and help achieve OKRs by impacting relevant KPIs. The goal is focusing efforts only on work that addresses clear hypotheses aligned to customer and business goals.

1. OKRs, KPIs and Hypothesis Driven
Product Development
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A
1
Armen Bandikian - Technical Product Manager at Stash
(an introduction)

2. 2Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A
This presentation is 32 pages long (don’t worry, it’ll go fast).

3. Quick Intro
3Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

4. A little about me
4
◉ Native New Yorker
◉ Lived and worked in Armenia for 2 years
◉ Currently a TPM at Stash working on the Debit product
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

5. What is this talk about?
◉ In short, it’s all about knowing and doing the things that
matter
5Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

6. Thank you.
6

7. ...Just kidding
7Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

8. What you will take away from this presentation
◉ Learn to only do the things that matter
◉ Define good OKRs, KPIs and understand their impacts
◉ Understand hypothesis driven product development
◉ Hopefully have more questions that’ll help you build better,
more impactful products
8
wow
such relevance
much qualitymake history
so engaging
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

9. OKRs and KPIs
9Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

10. Why have solid OKRs and KPIs?
10Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

11. Why have solid OKRs and KPIs?
11
◉ Answers the most important question: Why are we doing this?
◉ Setting OKRs means setting the direction your team/company
will be working toward
◉ It holds everyone accountable for the work they do and the
impact they have
◉ Aligns company goals with customer needs
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

12. A Good OKR...
12
◉ Has a measurable impact and has Key Results that are clearly defined
◉ Is beneficial both to the customer and the company
◉ Has a KPI that can be tracked to validate results
◉ Is specific enough to justify (why) any work being done but doesn’t dive
into how it should be implemented.
◉ Influences your decisions, the product roadmap, the work you do and
how you get it done!
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

13. Tips for setting good OKRs and KPIs
13
◉ Think about the customer (empathy)
◉ Don’t work in a silo, encourage everyone to take part in
defining OKRs and KPIs
◉ Consider the goals for the quarter while taking the mid and
longer term goals into account
◉ Have all stakeholders buy in to the goal that’s set and raise
any concerns and even challenge decisions at this point
◉ Define the metrics that need to be impacted (either in the O
or the KR)
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

14. Good OKR, Bad OKR
14Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

15. Good OKR, Bad OKR
15
◉ Objective: Increase ad revenue
per user by 15%
◉ Key Results:
◉ Increase MAU
◉ Increase CTR by 20%
◉ Increase advertiser spend
Organization OKR/Goal: Increase Revenue
◉ Objective: Increase revenue
◉ Key Results:
◉ Show more ads to users
◉ Increase CTR
Good OKR Bad OKR
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

16. But What About Feature Parity?
16
Kano Model MoSCoW method
Must Have
Should Have
Could Have
Won’t Have
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

17. Did you know you can set OKRs and KPIs for
your team?
17Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

18. Being a Metrics Driven Organization
18
◉ Give as many people access to your KPI
dashboards as possible (this will keep
everyone accountable and excited)
◉ Measure your team’s impact and work. Set
an OKR and KPI for it!
◉ Metrics are great, understanding the
metrics are better.
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

19. Hypothesis Driven
Products
19Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

20. What is Hypothesis Driven Product
Development?
20
◉ Basically - it’s doing work that has a verified hypothesis or doing work
that verifies or disproves a hypothesis - and nothing in-between.
◉ At a high-level, it ensures that meaningful work you’re doing is also
impactful.
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

21. Why is it Great and What does it have to do
with OKRs and KPIs?
21
◉ Hypotheses are meant to be tested with measurable results. They help
you implement the things your customers care about the most while
reducing risk and time to market.
◉ Can be included in every story/ticket being written (whether the
hypothesis is validated or not).
◉ Puts the “why are we doing this” front and center for all the things you
do.
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

22. Why is it Great and What does it have to do
with OKRs and KPIs? (part 2)
22
◉ Just like everything else, hypotheses must align with an OKR and
impact a KPI.
◉ Usually answers the why and the how in a single statement.
◉ Once a hypothesis is tested and validated, it gives you a clear direction
to take the product in and helps you understand your customer
better.
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

23. A good hypothesis...
23
◉ Aligns to an OKR.
◉ Can be measured by and impacts a KPI.
◉ Is specific and well documented.
◉ States what the expected impact is.
◉ Has a clear definition of success or failure (or validated).
◉ Should measure only one thing and be isolated when being
experimented on.
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

24. Good Hypothesis, Bad Hypothesis
24Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

25. 25
◉ We believe that by showing 5
suggested connections in a user’s
connections list, a statistically
significant portion of users will
have three or more additional
connections as a result of visiting
this page.
◉ Statistical Significance: x Users or
y %
◉ We believe that by
exposing more
suggested connections
to a user, a significant
portion of users will
invite new users to the
platform.
Good Hypothesis Bad Hypothesis
Good Hypothesis, Bad Hypothesis
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

26. Testing a Hypothesis
26
◉ A/B Testing in production
◉ Data
◉ User Research (interviews, mock walkthroughs, surveys)
◉ Note: you don’t have to build features in their entirety to test them.
Something as simple as a waitlist could make a big impact.
◉ Something called...counterfactual time series and causal impact?
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

27. Causal Impact!
27Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A
How does it work?
Given a response time series (e.g., clicks) and a
set of control time series (e.g., clicks in
non-affected markets or clicks on other sites), the
package constructs a Bayesian structural
time-series model. This model is then used to try
and predict the counterfactual, i.e., how the
response metric would have evolved after the
intervention if the intervention had never occurred.
https://google.github.io/CausalImpact/CausalI
mpact.html

28. Tying It All Together
28Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

29. Revisiting this...
29
Kano Model MoSCoW method
Must Have
Should Have
Could Have
Won’t Have
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

30. In Summary
30
◉ Create clear, measurable, and impactful OKRs, tie them to KPIs and get
stakeholder buy-in.
◉ Don’t lose sight of your customer for a higher level OKR (like increasing
revenue or decreasing CAC).
◉ Test hypotheses, not features (be experimental).
◉ Be specific...be very specific!
◉ After testing, assign a level of confidence for your tests conclusions if
there is room for interpretation.
◉ Always be clear about “why” you’re doing something.
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

31. In Summary
31
Do the things that matter!
Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A

32. Q & A
32Intro OKRs & KPIs
Hypothesis
Driven Products
Tying it all
together
Q & A