This document discusses an agile approach to measuring business metrics. It proposes using "Analytics Sprints" where each sprint a key performance indicator (KPI) is selected and hypotheses for improving it are tested. At the end of each sprint, results are assessed and the team either pivots experiments or perseveres on the next KPI. Having a dedicated "Metrics Master" role to facilitate this process with stakeholders is also suggested. Case studies from Spotify using iterative metric refinement are provided. Challenges with early metrics and combining incremental values are also outlined. The approach aims to discover and measure true business value in an adaptive, learning-focused way.

1. An Agile approach to
Business Metrics
by @Pablo_Valcarcel
pvalcarcel@wemanity.com

2. Product Owner’s
Adventures in
Dataland

3. Who is this guy?
@Pablo_Valcarcel
pvalcarcel@wemanity.com

4. ‘Conventional’ Agile Metrics
Lead Time*
Defect count
Work in Progress
Code coverage
Unplanned Changes
Velocity (Stpts per sprint)
Return on investment*
Innovations per sprint
Artifacts generated
Slack time
Failure Load (firefighting time)
Iteration Burn-Down
Unfinished Stories
Customer Satisfaction*
LOC (lines of code)
Un-deployed Stories
# Blocks
Budget/Schedule Compliance
Flow Efficiency (lead time / touch
time)
Release Burn-Up
*Italics: Connected to business value.

5. Your average Sprint Review...
KPI
KPI KPI
KPI
Stakeholder
Scrum MasterProduct Owner

6. Is this ‘Agile’ enough?
KPI
KPI KPI
KPI
Stakeholder
Scrum MasterProduct Owner
● Does it help increase customer satisfaction?
● We deliver business value on every iteration,
but how do we measure it?
● Shouldn’t the definition of metrics be an
iterative+feedback-based process on itself?
● Priorities and markets change all the time.

7. Enterprise Metrics: The Ocean of Data

8. The Startup Way: Go Pirate
‘AARRR!’
-Dave McLure
500 Startups

9. The Lean Metrics Process (Croll & Yoskovitz)

10. The Lean Sprint (Ash Maurya)

11. The Thesis: The Analytics Sprint
Agile discipline and
‘Lean Analytics’
thinking can provide a
better method to
measure value.

12. The Analytics Sprint
Planning:
At the beginning of each sprint discuss:
● With stakeholders: What are the OKRs/key metrics? What’s the
KPI we should work on the next iteration? What’s the baseline?
● With team: Hypothesis are defined on why/how to tackle that
KPI.
All of this: KPI, baseline, hypothesis and experiments are
documented in writing.

13. The Analytics Sprint
Reviewing:
At the end of each sprint:
● There’s an assessment. Did we hit the goal? If not pivot into
another experiment. If we did we persevere onto the next KPI.
● Data is turned into validated learning. Even failed experiments
provide learning (think Google with OKR).

14. The Analytics Sprint: The Metrics Master
Metrics Master: Role responsible for the
definition, revision and accounting of the key
metrics.
● Facilitates discussion with stakeholders on
business value and key metrics.
● Discusses hypothesis and experiments with
team.
● Tracks and does innovation accounting.
● Reports back on the results and either pivots or
perseveres on the next metric.
● It can be the Product Owner (business value) or
a dedicated team-member (e.g. data-scientist)..

15. But wait, I already do that!
Even if you’re already measuring a
business value metric every sprint:
● When was its value defined?
● Who defined it?
● How’s the reporting being done to
stakeholders and customers?
(Theatre of success or. customer
discovery?).

16. Case Study: Spotify Metrics-Driven
Development
Every retrospective they measure squad performance, and they refine the
metric.
They also set goal targets at the start of every integration project we do
(which may span multiple sprints). For example: “we will know we’re
successful when this integration brings us x-amount of users.”
Source: Lynn Root

17. Case Study: Spotify Metrics-Driven
Development
● They consciously avoid waste by
measuring everything.
● They also set baselines with
historical data.
● Goals are a shorter decision-
making cycle as well as make
more informed decisions about
strategy and partnerships.

18. Some pro-tips
● Write it down. Prevents
biases.
● Always define a baseline or
benchmark.
● Avoid ‘bad metrics’.
● Results will be murky.
● Conversation with
stakeholders, customers and
team is a value on itself.

19. Further Reading:
Spotify: Metrics Driven Development by Lynn Root.
Lean Analytics by Alistair Croll & Ben Yoskovitz.
Appropriate Agile Measurement by Deborah Hartmann & Robin
Dymond.
Five Tips For Testing Your Idea by Anita Newton, Cindy Alvarez &
Alistair Croll.
Scaling Lean by Ash Maurya.
Managing Metrics in an Iterative Incremental Development
Environment by John D. McGregor.
Agile Metrics, post by André Dhont.

20. A call to action:
We like thinking of Agility not as a cargo cult, but as a set of values
and principles that we adapt to the specifics of every single case.
So, we need you to learn.Please, hit me with your ideas, case
studies and questions at pvalcarcel@wemanity.com

21. Thanks for your attention!!!
@Pablo_Valcarcel
pvalcarcel@wemanity.com

22. Reasons for an Agile approach
● We don’t know what we should be measuring.
● We discover a better way to measure it.
● Business priorities and political priorities change.
● The product’s incremental development requires an incremental
an adaptative approach to its metrics.
● Project (or transition) is too big for our resources and we have
to pick our battles.
We want to measure true business value as early as possible!

23. Challenges with this approach
From a business point of view:
● First, what information is available early in the process and how
reliable is that information?
● Second, how can we use the early estimates to predict the
actual values?
● The incremental aspect of the process requires that there be a
plan for combining the values of certain metrics in order to
obtain project-wide values (e.g. Addition, Averaging, Min-Max
values).

24. Startup Vs. Intrapreneurial metrics (by A. Croll)