Agile KC Presentation on 9/26/13

1. Agile Metrics
Craig Rudman
VP Services, Agile Coach

2. Purpose
• Greater alignment of strategy to execution
• Improve the flow of work
• Improve product quality
• Support continuous improvement efforts
• Improve the quality of life
• Increase trust
Discuss ways to go about measuring work and workflow

3. Roadmap
Context
• What can we
measure?
• What should
we measure?
• What
principles
inform agile
metrics?
Systems
Thinking
• Who is your
customer?
• The nature of
demand
• Value
creation
Agile at Scale
• Portfolio
• Program
• Team

4. Why Metrics?
• Increase visibility of work
• Increase alignment of strategy to
execution
• Improve trust
• Improved product quality
• Improved workflow
• Improved performance
• Identify and solve problems
Metrics give us
insight into the
health of our
organization,
our people,
processes, and
tools.
We use metrics to help us adapt and improve

5. We can quantify those things we can see
We can qualify the things we can't see
What can we measure?
Quantify
• Effort
• Time
• System Components
• Tests
• Events
• Data
Qualify
• Quality
• Perceptions
• Feelings
• Attitudes

6. We should measure the things we value
What should we measure?
If your goal is…
• Improving visibility of work
• Better alignment
• Increasing trust
• Improving quality
• Improving process
• Solving problems
then you should measure…
→ work-in-process
→ context
→ commitments
→ breakage and rework
→ flow
→ root causes

7. The Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping
others do it. Through this work we have come to value:
• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on
the left more.

8. The Twelve Principles of Agile Software Development
• Our highest priority is to satisfy the customer through early and continuous delivery of
valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change
for the customer's competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a
preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they
need, and trust them to get the job done.
• The most efficient and effective method of conveying information to and within a
development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users
should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity--the art of maximizing the amount of work not done--is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and
adjusts its behavior accordingly.

9. Agile Metrics Support Agile Principles
If motivated by these principles… Measure for these outcomes…
Our highest priority is to satisfy the customer through early and continuous delivery of valuable
software.
Customer satisfaction
Cycle time and throughput
Product value realized
Welcome changing requirements, even late in development. Agile processes harness change for the
customer's competitive advantage.
Changeability of requirements
Value of change
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference
to the shorter timescale.
Frequency of delivery
Product stability
Business people and developers must work together daily throughout the project. Frequency/quality of collaboration
Delays
Build projects around motivated individuals. Give them the environment and support they need, and
trust them to get the job done.
Alignment of work to motivated people
Quality of team support
Release predictability
The most efficient and effective method of conveying information to and within a development team
is face-to-face conversation.
Efficiency of communication
Effectiveness of communication
Working software is the primary measure of progress. Rate of product adoption
Frequency of product usage
Failure demand
Breakage
Agile processes promote sustainable development. The sponsors, developers, and users should be
able to maintain a constant pace indefinitely.
Burnout, attrition, turnover, morale
Stability of cadence and velocity
Stability of team organization
Continuous attention to technical excellence and good design enhances agility. Skills growth
Increasing rates of reuse
Design/code consistency and quality
Test coverage/automation
Simplicity--the art of maximizing the amount of work not done--is essential. Low complexity
Short work queues
The best architectures, requirements, and designs emerge from self-organizing teams. Internal locus of control
High feature/story acceptance rates
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its
behavior accordingly.
Consistency of cadence
Effectiveness of retrospectives
Progress towards improvement goals

10. Exercise: Five Votes
• Read through the agile principles
• Think about where you would like to see
improvement
• Cast your five votes accordingly
– You can put them all on one item
– You can distribute them across up to five items
• Working as a group, we will discuss a set of
metrics to support the highest-priority item

11. Customers Suppliers
Assets/Expenses
Profit/Loss
Cash Flow
Market Share
Customer Satisfaction
Employee Satisfaction
Shareholder Benefit
Society Benefit
Knowledge Creation
Product/Service Quality
Time to Market
Demand
Revenue
Resources
Take a Systems View
Demand
Value
Revenue
Metrics help us
understand and
manage these
things
The primary metric for agile is whether or not value is being created.
This is determined empirically, by demonstration, at the end of every single iteration.
Your business is a value-creation system

12. The Big Blue Company
Information Services
Data
Management
21
Application Development
218
Infrastructure/Ops
77
Services
94
Claims (206)
Prov. & Gov't Relations (162)
Sales & Marketing (146)
Finance (106)
All Others (155)
Customer Service (227)
Membership (129)
Who is your customer?
Demand
Value
Revenue

13. Mandates
Preventive
Maintenance
How much work can you manage?
Initiatives
Production Support
Enhancement Requests
Defects
Rework
Stuff and Nonsense

14. 74%
18%
8%
How much work can you manage?
Production Support
Enhancement Requests
Defects
Rework
Stuff and Nonsense
Mandates
Preventive
Maintenance
Initiatives
209,320
49,788
22,150
Total capacity = 281,258 hrs

15. Types of Demand
• What is the nature of customer demand?
– How much of it is failure demand?
– How much of it is mandates?
– How much of it is value demand?
• If demand exceeds capacity, what are your
options?
“All time spent handling failure demand is waste.”*
*Mary Poppendieck

16. Portfolio Management
• Failure Demand (from 74% to 50%)
– Defects
– Rework
– Missing Features
• Value Demand (from 8% to 20%)
– Preventive Maintenance
– New Features or Services
– New Capabilities
• Mandates (from 18% to 30%)
– Regulatory compliance
50%
30%
20%
Failure Demand
Mandates
Value Demand
Constrain effort to align with strategy

17. It’s All About Choices
• Failure Demand
– Take a long-term view
– Perform root-cause analysis to find the source of greatest demand
– Fix it
– Repeat the process
• Value Demand
– Rank initiatives according to highest value
– Estimate the size of the effort
– Find the highest value with the least effort
– Build that thing with the least delay
– Repeat the process
• Mandates
– Fix the due date
– Prioritize features using a MoSCoW model (Must, Should, Could, Won't)
– Evaluate design options and favor the simplest approach, even if sub-optimal
– Optimize workflow to maximize throughput

18. How we build itWhat we build
Things We Can Measure
Performance
• Throughput
• Cycle Time
• Effort
• Schedule
• Inventory
– Work Not Started
– Work In Progress
Value
• ROI
• Revenue
• Cost
• Cash Flow
• Market Share
• Growth
• Quality

19. Value Goals
• Increase revenue
• Increase market share
• Increase customer satisfaction
• Reduce cost
• Manage risk

20. Features Deliver Value…
• Improve customer
satisfaction for both
members and providers
• Increase the number of
claims processed per
employee
• Reduce administrative costs
as a percentage of revenue
Add claim status inquiry to web
Add additional conditions to
claims adjudication so that
fewer claims "PEND"
Reduce dependencies between
systems by introducing service
layer to architecture

21. … But Only When In Production
• How long will you have
to wait before the
feature makes it into
production?
• How long will the
feature have to be in
use before it pays for
development and
begins to add value?
Ideation
Design
Planning
Coding
Testing
Staging
Release
Effort incurs
development
costs
Time waiting for new features represents
lost opportunity costs

22. Cost/Benefit Analysis of Features
• Begin by establishing the value proposition
• Evaluate your solution options
• Calculate development effort and cost of each
• Calculate the cost of delay
• Track adoption and use of the feature to
measure value

23. Example: Claim Status on the Web
Every day, Customer Service handles some number of phone calls with
certain percentage of those calls are inquiring into claim status. That
leads us to the following analysis:
• Assume 2400 phone calls each day on average
• Assume 20% of them on average (480) are claim status inquiries
• Assume 10 minutes per call on average
• Assume $50 per hour is the average fully-loaded cost of a CSR
• 4800 minutes or 80 hours per day on claim status inquiry equates to
$4,000 per day
• If there are 253 work days in a year, that equals $1,012,000 spent
responding to claim status inquiry each year
Note: the example shown is purely fictional and for illustration purposes only.

24. Evaluating Options
Three solutions are considered:
1. Do nothing
2. Develop a feature with fully-committed
teams in four months at a cost of $255,000
3. Develop the same feature with partially-
committed teams in sixteen months at a cost
of $338,000

25. $-
$20,000
$40,000
$60,000
$80,000
$100,000
$120,000
$140,000
$160,000
$180,000
Option 1
Option 2
Option 3
Cost of Delay
Total 3-year cost:
1. Do Nothing: $3,040,000
2. Dedicated team: $977,000
3. Shared team: $2,070,000
$1,093,400

26. Performance Goals
• Increase throughput
• Reduce cycle time
• Improve quality
• Eliminate waste
• Improve predictability

27. Better Processes Improve Performance
• Increase throughput
• Decrease schedule
variance
• Shorten feature cycle
time
Increase frequency of releases
Negotiate scope
Reduce WIP to move smaller
batches through development

28. Goal: Improve Throughput
• Features per quarter
• Stories per sprint
• Issues per day
• Value per month
Throughput is the rate at which a system achieves its goal in units of
output over time

29. High Variability Reduces Flow
Symptoms
– Flow becomes
unpredictable
– Bottlenecks occur
– Cycle time increases
– Throughput goes down
Response
– Reduce WIP
– Increase slack
– Swarm around bottlenecks
Knowledge work inherently has a high degree of variability

30. Flow
Design Capacity: the
maximum amount of work
that a system is capable of
completing over time
Cycle Time: the amount of
time that work spends in
the system
Utilization: the amount of
capacity being used at a
given time (%)
Effective Capacity: the
actual amount of work
that a system is capable of
completing over time
WIP: the amount of work
in that is in flight at a
given time
Slack: the amount of
capacity not being used at
a given time (%)
Throughput: the average
rate at which completed
work exits the system
Constraint: anything that
limits throughput

31. Collecting Flow Data
• Model value stream
• Record transitions
• Calculate cycle time
• Calculate WIP
Feature Ready Dev/Test Add'l Test Deploy
Cycle Time
(days)
1 2/1/13 2/2/13 2/2/13 2/3/13 2
2 2/1/13 2/2/13 2/3/13 2/3/13 2
3 2/1/13 2/3/13 2/3/13 2/5/13 4
4 2/1/13 2/3/13 2/4/13 2/5/13 4
5 2/1/13 2/3/13 2/6/13 2/7/13 6
6 2/1/13 2/4/13 2/9/13 2/9/13 8
7 2/1/13 2/4/13 2/9/13 2/9/13 8
8 2/1/13 2/6/13 2/9/13 2/10/13 9
Date Ready Dev/Test Add'l Test Deployed Total WIP
2/1/13 10 0 0 0 10
2/2/13 8 1 1 0 10
2/3/13 5 2 1 2 8
2/4/13 7 3 2 2 12
2/5/13 7 3 0 4 10
2/6/13 6 3 1 4 10
2/7/13 8 3 0 5 11
2/8/13 9 4 0 5 13
2/9/13 10 2 2 7 14
2/10/13 9 1 3 8 13

32. Cumulative Flow Diagram
0
2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Ready
Dev
Test
Deployed
WIP (9 items)
Throughput (1.25 items/day)
Cycle Time (10 days)

33. Agile Teams
• Team: a group of people linked together with common
purpose
• Backlog: Work identified, but not yet started
• Sprint: a time-boxed interval of time during which
work is completed
• Story: a narrative about the feature to be developed
with clear completion criteria
• Story Points: an estimate of the size of a story
compared to other stories
• Velocity: team throughput, as measured by story
points per sprint

34. Velocity Measures
• Burndown: total work
remaining
• Burnup: total work
completed
• Scope Change: additions
or subtractions of work
from the backlog
• Glidepath: estimated
time remaining until the
backlog is depleted
based on available data
Sprint
Pts
Completed Burnup Burndown
Scope
Changes Glidepath
0 750 900
1 40 40 790 80 915
2 90 130 700 850
3 10 140 790 100 784
4 75 215 765 50 719
5 120 335 565 -80 654
6 32 367 533 589
7 60 427 473 523
8 95 522 378 458
9 393
10 328
11 262
12 197
13 132
14 67
15 1
16 0

35. 0
100
200
300
400
500
600
700
800
900
1000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Burnup
Burndown
Glidepath
Velocity Reporting
Velocity = 65 pts/sprint

36. Velocity Pros and Cons
Used Well
• Helps teams make commitments
that they can keep
• Helps estimate the depth of a
backlog
• Helps product owners know how
much runway to prepare
• Helps teams reflect on their
efficiency and identify ways to
improve
• Accurately reflects the
sustainable pace of a team
• Emphasizes the importance of
“done”
Used Poorly
• Misunderstood as a measure of
team effectiveness
• Misused as a measure of
individual performance
• Invites meaningless or damaging
comparisons between teams
• Leads to Inflated Velocity
Syndrome (IVS), a misguided
attempt to improve throughput
by focusing on utilization rather
than outcomes.

37. Commitment
• Scope is negotiable
• Teams are self-governing
• Deadlines matter
• Done means done
Agility is rooted in commitment

38. Commitment Variance
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9
Plan
Actual
Range (-20%)
Range (+20%)

39. A Good Metric
• Supports your goals
• Fits your process
• Is fact-based
• Difficult to
manipulate

40. Inspect and Adapt
Portfolio
• Inspect: Demand (planned and unplanned)
• Adapt: Control allocation of effort to demand categories
Program
• Inspect: Workflow; feature acceptance rates; value attainment
• Adapt: Reduce WIP; negotiate feature scope and priority; remove
bottlenecks; increase slack; smaller, more frequent releases
Team
• Inspect: Team throughput, commitment variance, quality
• Adapt: Negotiate story scope and priority, remove
impediments, regular retrospectives, continuous improvement