Metrics should be used to learn and enable change in complex systems, not to control or constrain. Short-term metrics can provide information but longer-term perspectives are needed to understand emergent behaviors. It is important to consider unintended consequences and how metrics may influence behaviors in unexpected ways.
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Learn and Change with Metrics as a Complex System Agent
1. metrics
as a
learn and change
agent
Gaetano Mazzanti
@mgaewsj
Gama-Tech
2. as an agile coach
I am often asked to
define/introduce KPIs
“why?”
3. Case 1
“uhm, ehm, we
need to measure code & fix
if and how much
we are improving” no process
chaos
4. *typicallyfor
performance
appraisals
Case 2
“we
need to measure
traditional if and how much
we are improving;
rigid process
top-down
company policies
enforce using
KPIs*”
6. systems: some definitions
system
any network that has coherence,
it may be fuzzy, it may have purpose
agent
anything which acts within the system:
individual, group, idea, etc.
ordered complex chaotic
7. complex
system lightly constrains agents
agents modify system by their interaction
with it and each other, they co-evolve
ordered
system constrains agents
chaotic
agents underconstrained &
independent of each other
ordered complex chaotic
8. no metrics
code & fix
no process
chaos
ordered complex chaotic
9. metrics for a linear,
deterministic world
no metrics
traditional
code & fix
rigid process
top-down
no process
chaos
ordered complex chaotic
10. ???
agile
metrics for a linear,
deterministic world
no metrics
traditional
code & fix
rigid process
top-down
no process
chaos
ordered complex chaotic
11. complex systems
exhibit one or more
properties (behavior)
not obvious from the
properties of
the individual
interconnected parts
they are composed of
12. product development is
complex
“self-organizing, non-linear,
feedback systems are
inherently unpredictable,
they are not controllable“
D.Meadows
20. LeanStartup Loop
Ideas
Learn
Build
Data
Product
are we building
the right
product?
Measure
E.Ries
21. when looping
minimize total time through the
loop
learn fast
fail fast!
“biggest failure is failure to
learn” (M.Poppendieck)
22. single loop learning
lead to
actions results
how
which shape future
efficiency
doing things right
incremental change
23. single loop drawbacks
uncertainty
emerging information
unexpected impediments
delays
plan
defects
more detailed planning
rigid change management
overburden
(waste)
24. double loop learning
Chris Argyris
guide
values,
actions results
assumptions
how
why
step
outside
the
system!
lead to new/improved
effectiveness efficiency
doing the right things doing things right
question assumptions incremental change
radical change
25. timebound metrics
learn, change, move on
define metric*
set expiration date
goal ok or
values, expiration
actions results
assumptions date passed?
*actionable, accessible, auditable, time-bound
26. do not focus on the metric itself, it
is just a mean to understand/learn
and change
measures have value only if they
inform decisions or motivate action
27. be careful
how do you know that the results
you are seeing are related to
the changes you have made?
how do you know you are learning
correctly from those changes?
29. “bugs show lack of quality not its
presence” Gojko Adzic
removing what you don’t want (i.e. bugs)
does not imply getting what you do want
30. some hints
measure what scares you
measure outcome not output
i.e.
how much value created vs how
many stories done
defects trending down vs how many
unit tests created
31. but still
organizations want to measure individuals
=> people game metrics
⇒ “if you do X I will give you Y”
reduces motivation
1) no autonomy: feeling controlled by who
gives rewards
2) being payed for doing something: may
imply it’s not worth doing for its own
sake
33. how to avoid this?
no bonus, appraisals, etc.
shared goals (let the team find
how to reach them)
visualize feedback as
informational and not
controlling
give unexpected rewards
if you really have, let the team
do individual appraisals
35. inward & outward
looking metrics
inward outward
looking loooking
feedback
Business &
R&D Other
Stakeholders
boundary
objects
36. boundary objects
R&D business
metric
boundary object [sociology]: something that helps
different communities exchange ideas and information.
could mean different things to different people
but allows coordination and alignment
37. metrics quadrants
Business
outward
looking
&
feedback
Product
Process
inward
looking
Team Maturity
38. metrics quadrants
Business
boundary
objects
Product
Process
Team Maturity
39. metrics quadrants
Business
boundary
objects
Product
Process
agile
fragile
Team Maturity
41. metrics quadrants
Business
Lead
Time
Revenues
Cycle
Time
Quality
of
Service
(SLA)
ROI
Customer
SaHsfacHon
Throughput
Business
Value
Product
Process
Bugs?
WIP
Cadence
CI
Failures
Code
Quality
Rework
Technical
Debt
Impediments
Test
Coverage
RetrospecHves
Morale
Team Maturity
42. metrics quadrants
Business
what!?
no Lead
Time
Revenues
velocity? Cycle
Time
Quality
of
Service
(SLA)
ROI
Customer
SaHsfacHon
Throughput
Business
Value
Product
Process
Bugs?
WIP
Cadence
CI
Failures
Code
Quality
Rework
Technical
Debt
Impediments
Test
Coverage
RetrospecHves
Morale
Team Maturity
46. agility
being agile is not the goal,
it’s a mean
if you are really interested there are
plenty of agility tests on the Internet:
Nokia Test
Scrum Open Assessment - ScrumAlliance
Agile Maturity Model
Agile Evaluation Framework
Comparative Agility Assessment
etc.
47. impediments,
retrospectives, reviews
# of questions answered
?
# of questions asked
# action items addressed
# action items assigned (at previous meetings)
# of WTFs
WTF!?
WTF!?
57. flow related metrics
active WIP - buffered WIP
tasks that are really in progress – task
waiting to be handed-off (#,%,% of time
spent)
process efficiency
active time / cycle time
technical debt WIP / standard WIP
# of projects a person works in parallel
(should be 1!!!)
59. cumulative flow diagram
35
not so helpful?
30
25
backlog
Backlog
20
# user stories
to do To
Do
In
Progress
15
in progress
cycle time Done
WIP
10
throughput
done
5
0
time
66. how long since?
you talked to a customer
last useful retrospective
you learned something at work
your boss last freaked out
last critical bug
6
weeks
2
days
3
days
1 52
week days
67. and don’t forget
bus factor
# of key developers that need to be hit by a
bus to kill a project
68. “for every true one “per una vera
thousands are fake” mille sono finte”
F. De André