Metrics should be used to learn and enable change in complex systems, not to control or constrain. Short-term metrics can provide information but longer-term perspectives are needed to understand emergent behaviors. It is important to consider unintended consequences and how metrics may influence behaviors in unexpected ways.

1. metrics
as a
learn and change
agent
Gaetano Mazzanti
@mgaewsj
Gama-Tech

2. as an agile coach
I am often asked to
define/introduce KPIs
“why?”

3. Case 1
“uhm, ehm, we
need to measure code & fix
if and how much
we are improving” no process
chaos

4. *typicallyfor
performance
appraisals
Case 2
“we
need to measure
traditional if and how much
we are improving;
rigid process
top-down
company policies
enforce using
KPIs*”

5. unfortunately things are
not so simple and linear

6. systems: some definitions
system
any network that has coherence,
it may be fuzzy, it may have purpose
agent
anything which acts within the system:
individual, group, idea, etc.
ordered complex chaotic

7. complex
system lightly constrains agents
agents modify system by their interaction
with it and each other, they co-evolve
ordered
system constrains agents
chaotic
agents underconstrained &
independent of each other
ordered complex chaotic

8. no metrics
code & fix
no process
chaos
ordered complex chaotic

9. metrics for a linear,
deterministic world
no metrics
traditional
code & fix
rigid process
top-down
no process
chaos
ordered complex chaotic

10. ???
agile
metrics for a linear,
deterministic world
no metrics
traditional
code & fix
rigid process
top-down
no process
chaos
ordered complex chaotic

11. complex systems
exhibit one or more
properties (behavior)
not obvious from the
properties of
the individual
interconnected parts
they are composed of

12. product development is
complex
“self-organizing, non-linear,
feedback systems are
inherently unpredictable,
they are not controllable“
D.Meadows

13. so what are we
supposed to do?

14. we can
watch, learn and work
with the system

15. metrics
learn & change

16. how?

17. loop!

18. PDCA Loop
plan
act do
check
Shewhart & Deming

19. OODA Loop
observe orient
act decide
J.Boyd

20. LeanStartup Loop
Ideas
Learn
Build
Data
Product
are we building
the right
product?
Measure
E.Ries

21. when looping
minimize total time through the
loop
learn fast
fail fast!
“biggest failure is failure to
learn” (M.Poppendieck)

22. single loop learning
lead to
actions results
how
which shape future
efficiency
doing things right
incremental change

23. single loop drawbacks
uncertainty
emerging information
unexpected impediments
delays
plan
defects
more detailed planning
rigid change management
overburden
(waste)

24. double loop learning
Chris Argyris
guide
values,
actions results
assumptions
how
why
step
outside
the
system!
lead to new/improved
effectiveness efficiency
doing the right things doing things right
question assumptions incremental change
radical change

25. timebound metrics
learn, change, move on
define metric*
set expiration date
goal ok or
values, expiration
actions results
assumptions date passed?
*actionable, accessible, auditable, time-bound

26. do not focus on the metric itself, it
is just a mean to understand/learn
and change
measures have value only if they
inform decisions or motivate action

27. be careful
how do you know that the results
you are seeing are related to
the changes you have made?
how do you know you are learning
correctly from those changes?

28. what’s going on?
open
bugs
time

29. “bugs show lack of quality not its
presence” Gojko Adzic
removing what you don’t want (i.e. bugs)
does not imply getting what you do want

30. some hints
measure what scares you
measure outcome not output
i.e.
how much value created vs how
many stories done
defects trending down vs how many
unit tests created

31. but still
organizations want to measure individuals
=> people game metrics
⇒ “if you do X I will give you Y”
reduces motivation
1) no autonomy: feeling controlled by who
gives rewards
2) being payed for doing something: may
imply it’s not worth doing for its own
sake

32. Hawthorne Effect

33. how to avoid this?
no bonus, appraisals, etc.
shared goals (let the team find
how to reach them)
visualize feedback as
informational and not
controlling
give unexpected rewards
if you really have, let the team
do individual appraisals

34. metrics
quadrants

35. inward & outward
looking metrics
inward outward
looking loooking
feedback
Business &
R&D Other
Stakeholders
boundary
objects

36. boundary objects
R&D business
metric
boundary object [sociology]: something that helps
different communities exchange ideas and information.
could mean different things to different people
but allows coordination and alignment

37. metrics quadrants
Business
outward
looking
&
feedback
Product
Process
inward
looking
Team Maturity

38. metrics quadrants
Business
boundary
objects
Product
Process
Team Maturity

39. metrics quadrants
Business
boundary
objects
Product
Process
agile
fragile
Team Maturity

40. metrics quadrants
Business
Product
Process
Team Maturity

41. metrics quadrants
Business
Lead
Time
Revenues
Cycle
Time
Quality
of
Service
(SLA)
ROI
Customer
SaHsfacHon
Throughput
Business
Value
Product
Process
Bugs?
WIP
Cadence
CI
Failures
Code
Quality
Rework
Technical
Debt
Impediments
Test
Coverage
RetrospecHves
Morale
Team Maturity

42. metrics quadrants
Business
what!?
no Lead
Time
Revenues
velocity? Cycle
Time
Quality
of
Service
(SLA)
ROI
Customer
SaHsfacHon
Throughput
Business
Value
Product
Process
Bugs?
WIP
Cadence
CI
Failures
Code
Quality
Rework
Technical
Debt
Impediments
Test
Coverage
RetrospecHves
Morale
Team Maturity

43. fragility
code quality
reduce
technical debt
lack of advanced engineering practices
(i.e. TDD, CI) => rework

44. code quality evolution

45. code quality evolution

46. agility
being agile is not the goal,
it’s a mean
if you are really interested there are
plenty of agility tests on the Internet:
Nokia Test
Scrum Open Assessment - ScrumAlliance
Agile Maturity Model
Agile Evaluation Framework
Comparative Agility Assessment
etc.

47. impediments,
retrospectives, reviews
# of questions answered
?
# of questions asked
# action items addressed
# action items assigned (at previous meetings)
# of WTFs
WTF!?
WTF!?

48. metrics
queues

49. queues are bad
increase reduce
cycle time quality
risk motivation
variability
overhead
stop starting start finishing

50. cumulative flow diagram
arrivals
queue size
cumulative (WIP)
quantity
time in queue departures
(cycle time) (throughput)
time
source: Donald Reinertsen

51. cumulative flow diagram
WIP is a leading indicator
cycle time
cumulative WIP
quantity
time

52. cumulative flow diagram
large batches large queues
cumulative
quantity
time

53. cumulative flow diagram
small batches small queues
cumulative
quantity
time

54. Kanban board
if you can’t see it
you can’t manage it
backlog to do in progress done
2 3
WIP
cycle time =
throughput
cycle time

55. no WIP limit -> queue!
backlog
to do
ready
in progress
done
2
3

56. slack (%)
optimize flow
absorb variation

57. flow related metrics
active WIP - buffered WIP
tasks that are really in progress – task
waiting to be handed-off (#,%,% of time
spent)
process efficiency
active time / cycle time
technical debt WIP / standard WIP
# of projects a person works in parallel
(should be 1!!!)

58. visualizing tasks dynamics
backlog to do in progress done
2 4
1
2
3
4
days inactive task

59. cumulative flow diagram
35
not so helpful?
30
25
backlog
Backlog
20
# user stories
to do To
Do
In
Progress
15
in progress
cycle time Done
WIP
10
throughput
done
5
0
time

60. single column dynamics
In
Progress
6
5
4
3
WIP
2
1
0
4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
3
2
1
in
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
out
-­‐1
-­‐2
-­‐3
-­‐4

61. Kanban board dynamics

62. control charts
source: Samuli Heljo

63. metrics
easy but powerful
42

64. Happiness Index
leading or lagging?
feedback board
Mon Tue Wed Thu Fri
Tom J K J L J
Anne J J J J J
Paul L J K J J
Joe J J J J K
Eva J J J L J
niko-niko calendar

65. Pizza Index
Pizza = Overtime => not good
Steve Denning

66. how long since?
you talked to a customer
last useful retrospective
you learned something at work
your boss last freaked out
last critical bug
6
weeks
2
days
3
days
1 52
week days

67. and don’t forget
bus factor
# of key developers that need to be hit by a
bus to kill a project

68. “for every true one “per una vera
thousands are fake” mille sono finte”
F. De André

70. Gaetano Mazzanti
Gama-Tech
@mgaewsj
info@gama-tech.net