Measure Twice, Cut Once: Using Team Operation Metrics to Optimize a Scaling Software Shop

Conﬁdential │ ©2021 VMware, Inc.
Krista Luna & Daniel Witenberg
Measure Twice, Cut
Once: Using Team
Operation Metrics to
Optimize a Scaling
Software Shop
VMware Tanzu Labs
2021-09-02

Conﬁdential │ ©2021 VMware, Inc. 2
Who Might Find This Valuable?
Core audience, but not limited to:
Studio Leads Practice
Leads/Coaches
Business Owners

Program Health is an assessment
of the collective health of the set
of teams in a lab, studio or larger
grouping
What is Program Health?
By Program, we mean any collection of teams—Portfolio, Lab, Studio

4
A common challenge we ﬁnd when
joining clients on site, is the client’s
desire to scale and mature their
Agile software org, without scaling
the human capital investment to
managing that growth.

Organization are hungry for an
approach that allows for a large
and growing number of teams to
be managed and optimized by a
small team of program-oriented
practitioners—which grows
comparatively slowly.

With healthy quantitative
measurement of program
optimization, practice leadership at
large enterprises can manage
increasing scale more eﬀectively
than with qualitative tactics alone.

Practice Coaches can manage far
more team analytics than they
can team-level qualitative
intervention, leading to cost
savings around Practice
Management.
Teams’ progress on their Agile
maturity path will reﬂect naturally
and organically in their metrics
Coaching teams scale slower
than application teams
Beneﬁts of Quantitative Approach
Measuring Team Health Maps to Program Health
Agile Maturity can be clearly
measured

Creating a common language
A common and credible language in which
program health can be described and
discussed with stakeholders who don’t
come into regular contact with the program
teams is crucial.
That common language needs to be
meaningful to the business, consistent
across the organization, and provide
objective veriﬁability.

Manage by Exception as a Scaling Growth Model
Respond Only Where
there is need
Using this approach, we
‘don’t ﬁx what ain’t broke’
Scale Exponentially,
but Sustainably
Fewer Practices Coaches
needed for more teams
A quantitative approach allows us to Manage by Exception, which is to say, focus
on only those things that fall outside of our accepted standards.

What this isn’t!
Our approach will enhance other tactics, not replace them
DORA
DORA measures DevOps
lifecycle health.
Team Health
Checks
Provides valuable leading
and lagging feedback
indicators.
OKR/KPI/JTBD
Our approach doesn’t
replace team’s own work
to track their optimization
and set goals for
improvement.

11
Dashboarding a Program
Team performance transparency is the key to
sustainable scaled Program optimization

Anyone with an interest in the
Program should have a clear line
of sight into the metric and
understand that it’s credibly
connected to the thing it’s
supposed to measure.
To be useful, program metrics are just like the metrics we teach our PMs about for
product dev. They need to be:
A change in a metric means a
change in how optimized the
program is, so any metric we use
needs to be something we can
act against when things are
oﬀ-kilter
It needs to be easy to
understand, track and synthesize
meaning from a metric. They
shouldn’t create more work for a
practice lead to explain, than
they are saving in other areas.
How Do We Measure?
Actionable Accessible Auditable

13
To keep it simple, actionable and
understandable, we like to use RAG!
Red/Amber/Green is a quick visual
way to demarcate health/severity of
an exception, so it’s highly eﬀective
for quickly triaging response actions.

14
Which core metrics inform us
about Program Health?
Don’t ﬂood the zone—we want a few
measurements to tell us a lot

● Generating about as much
product value as they are able
to, with little resource waste on
non-value generating tasks
like fixing bugs or maintaining
environments and pipelines
● Routinely delivering about the
same increments of value over
a given regular iteration
period, making forecasting
future delivery volume more
precise.
Defining Optimization
We think of a program being optimized when it is highly efficient and highly reliable. In practice
this means that nearly all teams are:

16
Efficiency Metrics are used to
determine how much of a team or
a program’s overall available effort
is being expended on value
generating activity, versus other
activities.
Statistical Category: Efficiency

Mean Percentage of Accepted Stories that were defect corrections
Defect correction does not deliver new
value, so although ﬁxing bugs can be
valuable insofar as it enables
delivery/deployment of defective code
associated with previously delivered value,
the preference is for a team to spend its
capacity on delivering high quality,
defect-free code associated with novel
and new value, that won’t require defect
correction.
Eﬃciency Metric

When an engineer or pair delivers a story ready to be accepted or rejected, they
pick up another story and move on.
Acceptance criteria tests should be run as soon as possible, because:
● Newer stories may depend on successful delivery of earlier ones;
● Leaving acceptance to the end of an iteration of work may lead to rejection of
multiple stories, strongly aﬀecting velocity.
This metric may also indicate chronic unresolved bottlenecks or
blockers—removing those can greatly increase throughput.
Mean Time in Delivery State in days
Eﬃciency Metric
Mean Time in Delivery is a
Cycle Time segment!

19
Reliability tells us how well we can
forecast future delivery outcomes
for the program—or a given team.
Teams with regular and disciplined
work habits are more likely to
deliver a predictable volume of
new-value stories.
Statistical Category: Reliability

Volatility is simply the measure of how
diﬀerent a team or program’s velocity is,
iteration to iteration. High volatility
indicates that a team or program is
unpredictable and will only allow for
imprecise forecasting.
Low volatility leads directly to high
reliability, by narrowing the band between
best and worst case scenarios when
forecasting future work completion
windows.
Typically, a team with high volatility is
either facing challenges related to story
writing discipline, or is facing delivery
impediments related to organizational
complications not entirely within the
control of the team.
Mean Volatility
Reliability Metric

Visualizing a Forecast
● A low volatility program has a
relatively narrow band of
output scenarios even
projecting out fairly far into the
future, making forecasting the
business outcomes more
precise and more strategically
useful

22
Although not relevant at the team
level, spreads at the program level
help us to determine whether
lower-health indicators are
associated with the program as a
whole, or with outlier teams that
are performing diﬀerently to the
rest of the program.
Statistical Category: Spread Measurements

Tight spreads are a sign that whatever is
going on, it’s something that is being widely
practiced across the program. Coaching
strategies for non-optimal indicators in this
case, should revolve around
mass-intervention at the program level.
Wide spreads, on the other hand, would
suggest that a small number of teams are
outliers from the program. Examining the
team-level metrics will show which teams
deviate from the mean.
A strategy of specific team-level interventions
will likely be the more effective direction to
take. It is often the case that a team has
unique parameters to cope with, so, action
specific to that uniqueness is required to
unblock or re-orient the team.
Spread Measurements

Put it all together and you have a concise program dashboard

25
Practical Application
Making measurement meaningful and useful

When to start
It’s best to start at the beginning of an
engagement or as soon as possible
You will need:
● Planning tools that can surface the relevant raw data
● The data are obtainable in a sustainable,
easy-to-access manner

Set achievable goals for a Program
Gather your baseline
Now
Apply a liberal initial
baseline, drive the
Program toward realistic
and achievable
expectations
Next
As teams go more and
more green, we raise the
baselines to determine our
ceiling
Later
Eventually, we reach a
point where raising a
baseline turns nearly all
teams from green to red,
indicating peak
optimization

Balancing Quantitative with Qualitative
There’s value in learning...
●The local culture
●The operational model
...so that you can create the playbook
of tactics used to manage by
exception and enabling the
community of practice

Maturity Path
What metrics will do at each step
Now
Enable teams to gather &
understand their metrics
Next
Teams begin to understand
and self correct based on
their metrics
Later
Teams improve themselves
and the program around
them with little outside
enablement

Thank You
What questions are there?

Measure Twice, Cut Once: Using Team Operation Metrics to Optimize a Scaling Software Shop

Measure Twice, Cut Once: Using Team Operation Metrics to Optimize a Scaling Software Shop

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Measure Twice, Cut Once: Using Team Operation Metrics to Optimize a Scaling Software Shop

Similar to Measure Twice, Cut Once: Using Team Operation Metrics to Optimize a Scaling Software Shop(20)

More from VMware Tanzu

More from VMware Tanzu(20)

Recently uploaded

Recently uploaded(20)

Measure Twice, Cut Once: Using Team Operation Metrics to Optimize a Scaling Software Shop