This document discusses key performance indicators (KPIs) for measuring agile projects. It begins by defining metrics and KPIs, noting that KPIs should be tied to strategic objectives and have defined targets. It then discusses characteristics of good KPIs and provides examples of both traditional and agile KPIs related to time, effort, scope, and quality. The document cautions that too many KPIs can be useless and advocates keeping metrics simple. It also discusses challenges like cheating on metrics and provides tips for using tools and dashboards to effectively measure agile performance.

1. Agile KPIs
Gaetano Mazzanti
Gama-Tech

2. are you measuring your performances ?

3. a metric is a measure or a combination of measures for
quantitatively assessing, controlling or improving
a process, a product, a team
a KPI, Key Perfomance Indicator, is a (aggregate)
metric that:
is tied to a strategic objective;
have at least one defined time-bound target value
(number, range, limit, percentage, trend, variation)
Metric vs. KPI

4. self-induced vs. enforced
internal vs. external
snapshot vs. forward looking
different perspectives

5. actionable understandable
accessible
Many “recipes” and
characteristics, i.e.
SMART
Specific
Measurable
Achievable
Relevant
Timely
INVEST
Immediately actionable
Negotiable
Valuable
Estimable
Sized to fit
Testable
a good KPI
3 are enough

6. Agile vs. Traditional KPIs

7. classic KPIs
efforttimescope quality

8. some examples (time, effort, scope)
Is planning accurate ?
How long does it take for a
requirement to be delivered
to customers ?
Are effort and cost estimates
accurate ?
How is effort split between
design, coding and testing ?
Are requirements satisfied ?
Are requirements changing ?
How often ?
How “big” is the software ?
Schedule Adherence &
Variance
Lead Time, Cycle Time
Slip Charts
Effort & Cost Adherence
Cost per Phase
Amount of Rework
Requirements adherence
Requirements volatility (churn)
Code size (KLOCs :( ,
#modules, #classes, …)

9. a typical problem
time
Is planning accurate ?
slip chart
33%
200%
66%
300%

10. Defects
Code
Architecture
Usability
Documentation
Installation
Support
etc.
metrics explosion
Quality

11. exponential metric growth: i.e. Defects
total # defects
# defects by category (i.e. critical, major, minor)
# non-functional defects (usability, performance)
# new defects / time
# defects fixed / time
# critical defects / time
# re-opened defects (regression)
# tests / defect
time required to fix a defect
# defects found in-house / total # defects (DRE)
# defects found / # test hours spent
etc.

12. too many KPIs are useless

13. keep it simple, one step at a time

14. self-observation
problem #1
neglect shortcomings
problem #2
excuses
benchmarking
problem #3
who do you compare to?
evaluating: easier said than done

15. Metrics should not scare or threaten people
Enforced metrics are often cheated or ignored

16. Cheating
copy & paste some code,
copy & paste unit tests making slight modifications
result: increased code coverage
write some buggy code and
quickly fix it
result: increased number of fixed bugs
(maybe you also get credit for additional LOCs!)

17. Sex, Lies & Statistics
(beware of wrong/biased numbers)
1920: “most criminals are farmers”
actually most people where farmers at that time
1940: “twins more likely if mothers are in their 25s-30s”
most mothers gave birth in their 25s-30s at that time
growth much easier when
starting from a small base

18. Code Quality Metrics
KLOCs
code reviews & specific analysis tools needed
how fragile you are
Complexity
Duplication
Smells
Churn

19. Code Analysis Tools are Fun!
source:
source:

20. some Agile principles:
Create Value for the customer as early as
possible
Eliminate Waste (WIP, YAGNI)
Drive and Respond to Change, quickly
Time/Capacity Boxing (see Scrum and Kanban)
Provide Visibility into project progress
Enter Agile

21. productivity defects efficiency
effort customer satisfaction
velocity time requirements
size trend quality burndown roi
schedule business value capacity
complexity cumulative flow effectiveness
Googling with KPIs

22. Story Point
an arbitrary value to express effort, complexity and risk
associated to a user story, feature, task
Velocity
S
Velocity
# story points / interval (where
interval is usually a sprint or more
in general time)
Velocity is a team specific metric, it
cannot be used to compare
different teams
Velocity Improvement is a key goal
for Agile teams

23. 0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11
ideal SP left
Release Burndown Chart
sprint
Average velocity =
50 Story Points /
Sprint
500 Story Points (SP)
=> 10 Sprints

24. -100
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14
added SP
SP carried over from
previous sprint
SP todo from initial list
ideal trend
velocity
added SP trend
actual trend
Release Burndown Chart
sprint
velocity
delay
new expected
completion date
SPs have to be dropped
to get back on track

25. 0
50
100
150
200
250
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
real SP left
real Xtra time
left
ideal burndown
ideal Xtra time
left
Sprint Burndown Chart
bad
baddays
delay
look, ma
instant feedback!
Xtra time = time reserved
for other tasks (emergencies,
bug fixing, reviews, etc.)

26. each story has one or more automated tests
when tests pass the story has been completed
=> you are forced to become agile:
cannot do BDUF*, must do automated testing/
continuous integration
must deliver early value
Compile
Build
Test
Deploy
Running Tested FeaturesRon Jeffries
C.I. Quality Feedback Loop*Big Design Up Front

27. Backlog
Story
Points
Value For
Customer
Value
Delivered
(*100)
story 1 13 21 162
story 2 21 13 62
story 3 34 21 62
story 4 5 8 160
story 5 5 5 100
story 6 13 3 23
story 7 8 8 100
story 8 3 1 33
story 9 13 5 38
story 10 21 5 24
Early Value Delivery (again)
Business Value Points
Desire of Customer
Value =
Cost of providing functionality
68.3% of value
already delivered
here
same velocity
greater value!
story points

28. Iteration Related Metrics
(%) Stories & Story Points Completed
(%) Stories added/removed (Sprint & Release)
Stories unfinished/moved to Next Sprint
Sprints Moved to Next Release
Lead & Cycle Time (Stories & Defects)
Average Age of Stories and Defects
(%) Failed Builds, (%) Failed Tests
Defects Added & Fixed (absolute & trend)

29. WIP
Cumulative Flow Diagram
to do
doing
done
cycle time
cycle time
WIP
WIP is
increasing
done
doing
to do
Work In Process (WIP) and Cycle Time
should be minimized

30. Some Kanban Specific Metrics
Cycle Time =
Number of Things in Process/
Average Completion Rate
Little’s Law
time spent in each lane ?
bottlenecks ?
cycle time
lead time
Flow = Speed * Density,
Density Speed
=> Traffic Jam
40 60 25
ouch!
ouch!

31. first thing in the morning: dashboard!
Short Term vs. Long Term Dashboard.
Short Term: build ok, automated tests ok, value earned, schedule
(burndown), critical bugs entered, critical bugs still there, impediments, ...
Long Term: trends: failed builds, failed tests, # of tests, velocity, earned
value per month, # and type of bugs, …
i.e. increase test coverage

32. Essential Tools
a big board and lots of cards (hard to extract metrics though)
source code management (obvious)
automated build & test suites (CI)
issue tracking*
some/many of these tools may be
part of integrated product suites
*unless you belong to a small elite
code analysis
agile management system
dashboard

33. Take-Home Points

34. measure, evaluate, improve

35. communicate clearly
@you: are you getting this ?

36. transparency: KPIs visible to everyone

37. communicate visually

38. use simple tools (but use them!)

39. build a simple but effective dashboard

40. aim higher

41. Gaetano Mazzanti
info@gama-tech.net
Gama-Tech
Photo Credit:
iStockphoto.com S.Bouchard
Bruce McBroom/@Apple Corps Ltd