Metrics for Agile Teams Forget Velocity: 42 Other Things to Ponder

Metrics for Agile Teams
Forget Velocity, 42 Other Things to Consider!
Presented by
Andy Cleff
Sr. Agile Coach and Trainer

Metrics for Agile Teams • Andy Cleff • AgileVelocity.Com
Overview
12 rules
for measurement
5 Categories &
42+ metrics
How to choose?
Workshop
2
andycleff.com/metrics

Poll: Who’s here?
What is your primary role? (pick one)
• Product Owner
• Dev-Team
• Scrum Master
• Agile Coach
• Management
• Other
3
POLL

Poll Results: Who’s here?
4
POLL

Andy.Cleﬀ@
agilevelocity.com
AgileVelocity.com
Sr. Agile Coach
and Trainer
Andy Cleff
5

12 rules for measurement
1: Measure for a purpose
2: Shrink the unknown
3. Seek to improve
4: Delight all stakeholders
5: Distrust all numbers
6: Set imprecise targets
7: Own your metrics
8: Don’t connect metrics to rewards
9: Promote values and transparency
10: Visualize and humanize
11: Measure early and often
12: Try something else
change and innovation practices
MANAGEMENT 3.0
12 Rules for Measurement
When selecting metrics, ask:
Rule 1: Measure for a purpose
You must always understand why you are measuring.
The metric is not a goal in itself. Never forget that it’s
just a means to an end. It all starts with why.
Rule 2: Shrink the unknown
A metric is just a surrogate for what you really want to
know. Don’t jump to conclusions. Always try to reduce
the size of what is still unknown.
Rule 3. Seek to improve
Don’t only measure things that will make you look good.
There is plenty of data around, but you must focus on
what enables you to do better work.
Rule 4: Delight all stakeholders
Your work depends on others, and others depend on
you. Never optimize for just one stakeholder. Instead,
measure your work from multiple perspectives.
Rule 5: Distrust all numbers
attitude towards any reported numbers.
Rule 6: Set imprecise targets
When people have targets, they have an inclination to
focus on the targets instead of the real purpose. Avoid
this tendency by keeping your targets vague.
Rule 7: Own your metrics
Everyone is responsible for their own work, and metrics
help us improve that work. Therefore, everyone should
be responsible for their own metrics.
Rule 8: Don’t connect metrics to rewards
Rewards often kill intrinsic motivation and lead
to dysfunctional behaviors in organizations. Don’t
incentivize people to do work they should like doing.
Rule 9: Promote values and transparency
To prevent gaming, be transparent about values,
intentions, and the metrics everyone is using.
Rule 10: Visualize and humanize
Numbers tend to dehumanize everything. Replace digits
with colors and pictures, and keep the measurements
close to where the actual work is done.
Rule 11: Measure early and often
Most people don’t measure often enough. Measure
sooner and faster to prevent risks and problems from
growing too big for you to handle.
Rule 12: Try something else
It’s rarely a good idea to do the same things over and
over. The environment changes all the time. The same
should apply to how and what you measure.
Why “this metric?” – Why does it matter?
What insights might we gain from it?
What is expected to change? What is expected variability,
consistency – are we looking for trends or absolute values?
Working to improve one thing may temporarily reduce
promote knowledge sharing, collaboration with other
teams and trust with our sponsors?
Is this metric a leading or lagging indicator?
6
Shout out to Jurgen Appelo & Management 3.0
Big thanks to Ralph van Rosmalen for getting the ball rolling

Rule 1- Measure
for a purpose
You must always understand
what you are measuring. The
metric is not a goal in itself.
Never forget that it’s just a
means to an end. It all starts
with why.
7

Measure for outcomes, not outputs
Employee Engagement
Continuous Improvement
Predictability
Customer Satisfaction
Market Responsiveness
Productivity
Speed
Quality
Innovation
8

Reasons why we do measure
To know revenues and drive resource & people allocation accordingly
To make decisions that are not based on gut feelings alone
Monitor alignment with mission / vision / goal
Observe quality of product / process
Judge customer happiness / employee satisfaction
9

Reasons why we don’t measure
Measurements might be used as
weapons
Some metrics are not useful or actionable
Implementing measures would cost too
much time / effort
Some things might just be immeasurable
10

Rule 2 - Shrink
the unknown
A metric is just a surrogate for
what you really want to know.
Don’t jump to conclusions.
Always try to reduce the size of
what is still unknown.
11

Cyneﬁn Framework
12

Ordered
Domains
Cyneﬁn Framework
13
OBVIOUS
CLEAR

Here Things
Get Interesting…
Cyneﬁn Framework
14
APORETIC
or
CONFUSED

Rule 3 - Seek to
improve
Don’t only measure things that
will make you look and feel
good. There is plenty of data
around, but you must focus on
what enables you to do better
work.
15

Actionable metrics
“A good metric changes the way you behave. This is by far
the most important criterion for a metric: what will you
do differently based on changes in the metric?”
Lean Analytics, Alistair Croll and Benjamin Yoskovitz
16

Vanity metrics
“When we rely on vanity metrics, a funny thing happens.
When the numbers go up, I've personally witnessed
everyone in the company naturally attributing that rise to
whatever they were working on at the time. That's not too
bad, except for this correlate: when the numbers go
down, we invariably blame someone else”
Eric Ries
17

Rule 4 - Delight all
stakeholders
Your work depends on others,
and others depend on you.
Never optimize for just one
stakeholder. Instead, measure
your work from multiple
perspectives.
18

It is impossible to please everyone, but you would like to
know who is pleased at certain moments and who is not.
19
What do your partners care about?

Rule 5 - Distrust all
numbers
Observers usually influence their
own metrics, and they suffer
from all kinds of biases. Have a
healthy, skeptical attitude
towards any reported numbers.
20

Metrics for Agile Teams • Andy Cleff • AgileVelocity.Com 21
Cognitive biases

Story
Hawthorne Works, Chicago, ca. 1925.
22

Story
23

Story
24

Story
25
The Hawthorne, or Observer, Effect

Rule 6 - Set
imprecise targets
When people have targets, they
have an inclination to focus on
the targets instead of the real
purpose. Avoid this tendency by
keeping your targets vague.
26

"When a measure becomes a target, it ceases to
be a good measure."
Goodhart's law
27

Rule 7 - Own your
metrics
Everyone is responsible for their
own work, and metrics help us
improve that work. Therefore,
everyone should be responsible
for their own metrics.
28

Important considerations
How many metrics should a team “own”?
Which ones?
How long should they use the ones selected?
29

Anti-Patterns
People know/suspect they are being watched and judged (Hawthorn)
Striving for ever increasing values instead of striving for consistency
and stability (Goodhart)
30

More Anti-Patterns
Correlation is not necessarily causation (Milton Friedman’s Thermostat)
31

More Anti-Patterns
Comparing metrics
across teams that
are very different
32

Poll: Scale of your ecosystem
Number of Teams (pick one)
• Less than 5
• Between 6 and 9
• More than 10
33
POLL

Poll Results: Scale of your ecosystem
34
POLL

Rule 8 - Don’t
connect metrics to
rewards
Rewards often kill intrinsic
motivation and lead to
dysfunctional behaviors in
organizations. Don’t incentivize
people to do work they should
like doing.
35

DILBERT © 1999 Scott Adams.
Used By permission of ANDREWS MCMEEL SYNDICATION. All rights reserved.
36
Anti-patterns

Rule 9 - Promote
values and
transparency
Human beings are smart and
able to game any system. To
prevent gaming, be transparent
about values, intentions, and the
metrics everyone is using.
37

Values, intention, purpose
Do we get paid a sustainable value for what we do?
Are we great at what we do in the eyes of our customers?
Do our employees / teammates love what we do and the
way we do it?
Will what we do make the world a better place for our
grandchildren?
38

Values, intention, purpose

Rule 10 - Visualize
and humanize
Numbers tend to dehumanize
everything. Replace digits with
colors and pictures, and keep
the measurements close to
where the actual work is done.
40

Story
41

Story
42

Story
43

Story
44

Story
45

Rule 11 - Measure
early and often
Most people don’t measure often
enough. Measure sooner and
faster to prevent risks and
problems from growing too big
for you to handle.
46

“The only way to win is to learn faster than
anyone else”
Eric Ries
“What you want to do as a company is maximize
the number of experiments you can do per unit of
time.”
Jeff Bezos
On winning…

Rule 12 - Try
something else
It’s rarely a good idea to do the
same things over and over. The
environment changes all the
time. The same should apply to
how and what you measure.
48

Limited lifespan of all metrics
That which is measured will improve, at a cost.
When a measure becomes a target, it ceases to be a good
measure. (Goodhart)
Correlation is not causation, but it sure is a hint. (Friedman)
Use multiple viewpoints - technical as well as human - to
get a holistic perspective
49

12 rules for measurement
1: Measure for a purpose
2: Shrink the unknown
3. Seek to improve
4: Delight all stakeholders
5: Distrust all numbers
6: Set imprecise targets
7: Own your metrics
8: Don’t connect metrics to rewards
9: Promote values and transparency
10: Visualize and humanize
11: Measure early and often
12: Try something else
50

Discussion / Q & A
Is anything confusing so
far?
What are you doing
already that's working?
What's not working for
you?
51

Optional
✦ When you return from the
break, ask yourself the
following question: “What is
the most important
concept I’ve learned so far?”
Break Over In:
Sponge Activity

Toolbox of metrics
53

Business Outcomes

Poll: Your biggest “why”
What business outcome is the most important now? (pick one)
• Employee Engagement
• Continuous Improvement
• Customer Satisfaction
• Predictability
• Market Responsiveness
• Productivity
• Speed
• Quality
• Innovation
55
POLL

Poll Results: Your biggest “why”
56
POLL

Five categories of metrics
Process Health
Release
Product Development
Technical / Code
People / Teams
57
Shout out to:
Jason Tice @TheAgileFactor

Process health metrics
Assess day-to-day delivery team activities and evaluate process changes:
• Cycle time
• Cumulative flow diagrams
• Control charts
• Number of experiments performed
• Number of improvements made to
process over time
• Flow efficiency
• “Concept to cash” delivery lead time
• Batch size
• Successful iteration completion
• Escaped defect resolution time
• Time thievery
58

Shine a light on the time thieves
1. Too much work in progress (WIP)
2. External dependencies
3. Unplanned work
4. Conflicting priorities
5. Neglected work
59

Cumulative Flow Diagram (CFD)
60

61
kanbanize.com
Arrivals
Departures
WIP

62
kanbanize.com

63
Cycle Time
kanbanize.com

Release metrics
Focus on identifying impediments to continuous delivery:
• Deployment frequency
• Release success rate
• Failed build frequency
• Mean time to restore (MTTR)
• Release time
• Cost per release
• Escaped defects
• Release net promoter score
• Release adoption / install rate
64

Net promoter score (NPS)
65
surveymonkey.com

Product development metrics
Help measure alignment of product features to user needs:
• Customer value delivered
• Risk burn down
• Value stream mapping
• Sales velocity
• Product net promoter score (NPS)
• User analytics / DAU / MAU
• Backlog health index
• Number of validated business-level
hypotheses developed
• Number of times a week you talk to
an actual customer
• Ratio of implemented to non-
implemented customer-driven
changes
• HEART
• AARRR
66

DAU/MAU
67
geckoboard.com

HEART Framework
68
Goal Signals Metrics
Happiness For users to feel like the site
is unique A satisfaction rating from a survey
Satisfaction
Perceived ease of use
Net promoter score
Engagement For users to keep
discovering more content
The amount to time people spend
viewing from analytics
Number of videos watched per user
per week
Number of uploads
Number of shares
Adoption For users to use the site or
app
The number of signup per days,
from analytics
Upgrades to latest version
New subscriptions
Purchased by new users
Retention For users to continue to use
the site or app
The number of return customers
from customer data
Number of active users
Renewal rate
Repeat purchase
Task Success For users to accomplish
their goal
The number of successful
completes from a study
Search result success
Time to upload
Profile creation completion
Google Ventures

AARRR (Pirate Metrics)
• Acquisition
• Activation
• Retention
• Referral
• Revenue
69

Technical & code metrics
Help determine quality of implementation and architecture:
• Automated test coverage
• Number of tests written before coding
• Unit / regression test coverage
• Build time
• Defect density
• Code churn
• Code ownership
• Code complexity
• Coding standards adherence
• Crash rate / time to restore service
• Build breaks / change fail rate
• Technical drag
• Ratio of fixing work vs feature work
70

Test automation dashboards
71
getzephyr.com

People & teams
Reveal issues that impact a team’s sustainable pace and level of engagement:
• Team health and well-being
• Team / manager / org eNPS
• % of time w/o interruptions (fire fights)
• Trust between leadership and team
• Learning log
• Overtime hours
• New employee setup time
• Team tenure / manager half-life
• Mean time to reorg
• Whole team contribution
• Transparency (access to data,
customers, shared learning)
• Employee silence vs voice
• Number of experiments / changes
72

Why team health and well being matters
A strong constitution is necessary for teams to
overcome adversity, maintain resilience and
nurture a culture rich in collaboration and
innovation.
73

Resilience
Dictionary: The capacity to adapt to stress or loss healthfully
• Positivity
• Focus
• Flexibility
• Organization
• Proactivity
74
Shout out to:
Daryl Conner

Visualizing team health:
andycleff.com/teamhealth
Gallup Q12
Mood App + Health Monitor
TeamMood
eNPS
Comparative Agility Survey
Crisp Happiness Index
Team Barometer
Spotify Team Health Check
Happiness Wall / Window
Lean Agile Intelligence
Team Morale Metric
H.I.P. Survey
OfficeVibe
15Five
7Geese
75

Team mood
76
teammood.com
Team members will receive a daily
email and/or slack prompt, asking for
their mood of the day. One click and
done!

Team mood
77
teammood.com

Measuring Team Culture
78
Forsgren PhD, Nicole. Accelerate . IT Revolution Press. Kindle Edition.
On my team….

Measuring Trust and Ownership
79

80

81

82

Discussion / Q & A
far?
What are you doing
you?
83

Optional
✦ When you return from the
break, think about what
you’ve learned so far and
make a list of the names of
colleagues with whom you
could share some or all of
the information you’ve
picked up so far.
Break Over In:
Sponge Activity

Care and usage As George Box said, “All models
are wrong, but some are useful.”
85

Integral lenses
I / Personal:
This is about “Me” –
my psychological,
inner world; how I see
things, what I
experience, my
mindset.
86
We / Relational:
A shared vision and
interpersonal
dynamics among
peers; how we
interact, what we
value together, the
team micro-culture.
Its / Environmental:
Things external to the
team systems,
including other value
streams, handoffs,
rules, corporate
culture, and policies.
It / Practices:
Refers to the team
structure, methods,
metrics, processes,
decision making
patterns, leadership
and team practices.

Who a metrics model is for
There are two groups of
stakeholders:
• The individuals & teams
(I + We)
• The people & systems
supporting the teams
(It + Its)
I
Personal
Subjective
IT
Practices
Objective
WE
Relational
Intersubjective
ITS
Environment
Interobjective
Individual / Singular
Collective / Multiples
Interior /
Invisible
Exterior /
Visible
87

From the team perspective
• Builds up awareness
• Reveals working and what’s not
• Expands perspectives
• Provides a balanced view
88

From the supporting role perspective
• Provides high level summary
• Reveals patterns across
teams
• Shows opportunities to
change the system
89

What this is not
• A competition
• A comparison
• A weapon
90

What this is not
• An incentivisation model
91

What this is not
• Done
92

How to choose?
93

Review the options….
Why “this metric?” – Why does it
matter? To whom does it matter?
What insights might we gain from it?
What is expected to change? Are we
looking for variability, consistency,
trends, values, direction?
94

How might a metric be gamed,
misused (or abused)?
What are some trade offs (collateral
damage) as well as the costs of
getting “better data”?
95

How often would we like to “take a
data point”?
How long will we run the
experiment?
How when we know when we’re
“done” with this metric?
96

How will we make this measurement
transparent?
Is this metric a leading or lagging
indicator?
97

If you’re really in a hurry to get started…
1 = Choose from People & Teams
2 = People & Teams + track
average age of items in process
3 = People & Teams + any two from
Little’s Law (CT = WIP / TH) and then
you’ll get one bonus for free!
5 = People & Teams + Little’s Law +
Customer NPS + Time Since Last
Release
98

Summary so far…
• Use predictability to delight
• Abandon old concepts
• Measure many things (just not too many)
• Learn rapidly
• More geek joy
99

Discussion / Q & A
far?
What are you doing
you?
100

Optional
✦ When you’ve returned from
the break, and before you sit
down, take a couple of really
deep breaths. Maybe stretch
your arms overhead too.
✦ Then add a comment about
what you like best about
the course so far.
Break Over In:
Sponge Activity Print out worksheets from:

Putting Theory
into Practice
102

1. Explore what’s going well along with your main pain
points in the context of your desired business outcome
2. What items are a priority to experiment with?
3. Brainstorm what metrics you might try. Come up with
some good ones, and some bad ones too…
4. Work thru the “when selecting metrics” questions…
Workshop: Overview
103

Workshop: Paint your canvas…
1.
104

Example: What’s going well / main pain points
105
+ Team does lunch and learns
! Too much “must fix now”
! Sprints over capacity
! No ability to A/B test ideas
! Infrequent releases

1.
106

107

Example: Priority to experiment with
0% slack time 20% slack time 6 wks 2x

109

110

Example: What metrics you might try
Customer NPS
DAU/MAU
eNPS / Team Mood
Story points per sprint
DAU (alone)

112

113

Assumptions and corresponding tests:
• We believe that [persona type] has [trouble/need/desire]
doing [problem/opportunity].
• We will know we have succeeded when [qualitative
outcome] and [quantitative outcome]. This will improve
[kpi / measurable signal].
114

Example: Assumptions and corresponding tests
We believe that our active users
has a desire for more frequent
releases that include innovative
social sharing features.
We know we have succeeded when:
• DAU/MAU reaches 50% within 3
months…
• Dev-Team mood on average is above
7/10…
• Velocity decreases 20%…

Assumptions and corresponding tests:
• We believe that [persona type] has [trouble/need/desire]
doing [problem/opportunity].
• We will know we have succeeded when [qualitative
outcome] and [quantitative outcome]. This will improve
[kpi / measurable signal].
116

Sharing
•What are the things you’ll measure?
•How will you learn rapidly?
•Will this move you towards predictability?
•How will you visualize things?
•How will your information help you make better decisions?
117Metrics for Agile Teams • Andy Cleff • AgileVelocity.Com

If you get stuck…
•How much? As in, how many “widgets”
are you producing each week?
•How fast? How long does it take to create that item
from the second the conversation starts to when it’s finished?
•How good? What’s the quality?
•How predictable? If you tell a customer to expect an item within a certain
number of days, how frequently do you meet that deadline?
•How happy? Is your team happy and is your culture healthy?

If you get stuck…
Business
Outcome Example Measures
Employee
Engagement
• eNPS
• Bad turnover
• Employee referrals
• Offer acceptance
• Feedback surveys
• Team learning logs
Continuous
Improvement
• Value stream flow efficiency
• Reduction in recurring impediments
• Cumulative flow

Business
Innovation • Market Share
• Number of validated business-level hypotheses developed
and tested
• Slack time
Customer
Satisfaction
• Number of times a week team members talk to an actual
customer
• NPS
• Retention (DAU/MAU)
• Referrals

Business
Market
Responsiveness
• AARRR (Acquisition, Activation, Retention, Referral,
Revenue)
• HEART, (Happiness, engagement, adoption, retention, task
success)
Predictability • % complete of Sprint plan
• Velocity variability
• Say-Do ratio
• Unplanned work items
Speed • Cycle time
• Lead time
• Deployment frequency
• Meantime to restore (MTTR)

Business
Quality • Defects in production
• Change fail percentage
• Automated test coverage
• Ratio of fixing work vs feature work
Productivity • Value delivered
• ROI
• KPIs
• Thieves of time

Let’s keep the conversation going…
If you need to work on your foundational practices...
If you need to re-focus...
If you feel stuck mid-transformation...
If you need ongoing support during your transformation…
Please contact us, we oﬀer:
• Private Agile Training
• Agile Assessments
• Agility Tune-ups
• Agile Transformation Services
123

Metrics for Agile Teams Forget Velocity: 42 Other Things to Ponder

Metrics for Agile Teams Forget Velocity: 42 Other Things to Ponder

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Metrics for Agile Teams Forget Velocity: 42 Other Things to Ponder

Similar to Metrics for Agile Teams Forget Velocity: 42 Other Things to Ponder(20)

Recently uploaded

Recently uploaded(20)

Metrics for Agile Teams Forget Velocity: 42 Other Things to Ponder