How do we know we’re
delivering value or having
the intended impact?
Presented by
Kevin Burns
1-27-2017
kburns@sagesw.com, @kevinbburns

Open Discussion
Who’s measuring value, outcomes, and/or impacts today?
How are you measuring them?
If you’re not measuring them, why not?
Who’s measuring cost?
How are you measuring cost?
kburns@sagesw.com, @kevinbburns 2

The 1st Agile Principle
Our highest priority is to satisfy the customer through early and
continuous delivery of valuable software.
(should we change valuable to beneficial impact?)
How do we define value (impact) and how do we measure it?
Not all Projects (or Features) are created equal.
kburns@sagesw.com, @kevinbburns 3

Is value determined by delivery on time, on budget, and on scope?
Is the scope delighting the customer?
Are they using everything we delivered?
kburns@sagesw.com, @kevinbburns 4

In a survey of 4 products, 65% of the features were rarely or never used.
How much money could have been
saved if we never built them?
In the Waterfall project world, we have to ask
for everything we can think of because capital
will end at the end of the project. Instead we
should be asking what has the most value in
terms of the business outcome and/or impact
and how are we going to measure it.
kburns@sagesw.com, @kevinbburns 5

kburns@sagesw.com, @kevinbburns 6

kburns@sagesw.com, @kevinbburns 7

kburns@sagesw.com, @kevinbburns 8

Assumptions Challenged
• 3 things we wish were true
• Customer knows what they want
• Developers know how to build it
• Nothing will change along the way
• 3 things we have to live with
• Impact isn’t known until software is used in production
• Developers discover how to build it
• Many things change along the way
kburns@sagesw.com, @kevinbburns 9

Marty Cagan Quotes
• Customers don’t know what they
want. It’s very hard to envision the
solution you want without actually
seeing it.
• At least 2/3 of our ideas are never
going to work. The other 1/3 will take
3-4 iterations to get right.
• The role of the product manager is to
discover a product that is valuable,
usable, and feasible. Product, design,
and engineering work together to
arrive at optimal solution.
kburns@sagesw.com, @kevinbburns 10

kburns@sagesw.com, @kevinbburns 11

What we measure is changing
Business Customer
PO, SM, BL
Software Engineering
AD, DD, DA
User
UX, BA, QA, SME
Business
Valuable
Design
Usable
Technically
Feasible
INNOVATIVE
SOLUTION
kburns@sagesw.com, @kevinbburns 12

Lean Startup
• Are we asking what are Minimum
Viable (Valuable) Product and how do
we know when we’ve delivered it?
• Use a scientific method to measure,
learn and pivot or preserver.
• Use meaningful quantitative
objective measure to evaluate impact.
• Can you use A/B testing?
kburns@sagesw.com, @kevinbburns 13

MVP
Innovation
User
UX, BA, QA, SME
Business
Valuable
Design Usable
Software Engineering
AD, DD, DA
Business Customer
PO, SM, BL
Use scientific method
(measurable) to learn
and discovery your
Minimum Viable
(Valuable) Product
(MVP)
Technically
Feasible
MVP innovations emerge
from Conversations kburns@sagesw.com, @kevinbburns 14

Impact-Drive Development
kburns@sagesw.com, @kevinbburns 15

kburns@sagesw.com, @kevinbburns 16

Value and/or Impact driven culture
• Are we measuring the Cost vs Benefit at all levels of our work items?
• Portfolio
• Program
• Project
• Feature/Capability
• Story/Requirement
• Tasks/Test
• Are we measuring the Impact our features have on our customers?
• The act of sizing helps us define done
and what the really valuable work is
• Use story telling and test statements
create understanding of value and DoD
kburns@sagesw.com, @kevinbburns 17

Multi-month
Monthly
2-weeks
Leadership
T-Shirt Sizing
X-S 1 Sprint
S <1 month
M 1-3 months
L 3-9 months
X-L >9 months
Team Planning-Poker
Fibonacci Sizing
(1,2,3,5,8,13,20,40,100)
Team task hours to capacity (2,4,6)
Solution Decomposition Sizing Pattern
Sizing our Cost
The act of sizing helps us
understand what’s
valuable to deliver.
Scope doesn’t grow, our
understanding does.
kburns@sagesw.com, @kevinbburns
18

Deliver 100% of 10% of Project
• Can we incrementally deliver value and test it’s
impact?
• Can we create incremental release plans to
deliver 100% of 10% of project?
• What constraints do we have in working this
way?
• Can we overcome or work within these
constraints and still deliver incrementally?
• What/who is preventing this approach?
kburns@sagesw.com, @kevinbburns 19

Create Faster Feedback
• When queues and batch sizes are large
feedback is slow
• Slow feedback hurts quality, efficiency, and
cycle time
• Feedback speed has enormous economic
leverage in product development, but it is
rarely explicitly managed
kburns@sagesw.com, @kevinbburns 20

The Front-Loaded Lottery
• A lottery ticket pays $3000 to winning three digit number
• You can pick the number in two ways:
• Pay $3 to select all three digits at once
• Pay $1 for the first digit, find out if it is correct, then choose if you wish to pay
$1 for the second digit, and then choose if you wish to pay $1 for the third
digit.
kburns@sagesw.com, @kevinbburns 21

Value of Feedback
100%
Spend $1
Savings = $0.90
Savings = $0.99
10%
1%
0 $1 $2 $3
Probability
of
Occurrence
Cumulative Investmentkburns@sagesw.com, @kevinbburns 22

Sequence Work Correctly (Cost of Delay)
• The sequence in which work is processed is called the queuing
discipline
• By changing the queuing discipline we can reduce the cost of a queue
without decreasing the size of the queue
• Since manufacturing has homogeneous flows it always uses FIFO
(First-In-First-Out)
• For the non-homogeneous flows of product development other
approaches have better economics
kburns@sagesw.com, @kevinbburns 23

Use FIFO for Homogeneous Flow
First-In First-Out
Cost
of
Delay
1
2
3
A
B
Time
Cost
Delay Cost
Last-In First-Out
Cost
of
Delay 1
2
3
A
B
Time
Cost
Project Duration Cost of Delay
1 3 3
2 3 3
3 3 3
kburns@sagesw.com, @kevinbburns 24

Weighted Shortest Job First (WSJF) for
Non-homogenous flow
High Weight First
Cost
of
Delay
1
2
3
A
B
Time
Cost
Delay Cost
Low Weight First
Cost
of
Delay
A
B
Time
Cost
Project Duration Cost of
Delay
Weight =
COD/Duration
1 1 10 10
2 3 3 1
3 10 1 0.1
1
2
3
160 7
96 % Reduction in COD
kburns@sagesw.com, @kevinbburns 25

Paul Ellarby example
1. Create Method for measuring Value
2. Understand the value and cost of each
portfolio down to the feature level
3. Allocate Value Points across
feature/capabilities
4. Track Value vs Cost for each iteration
https://www.thoughtworks.com/insights/blog/how-do-you-measure-value
kburns@sagesw.com, @kevinbburns 26

How to measure anything – Douglas Hubbard
http://www.howtomeasureanything.com/
kburns@sagesw.com, @kevinbburns 27

Measurement Basics
• A measurement is an observation that quantitatively reduces uncertainty.
Measurements might not yield precise, certain judgments, but they do reduce
your uncertainty.
• A good object of measurement is something that is clearly defined and it’s
observable.
• Uncertainty is the lack of certainty: the true outcome/state/value is not known.
• Risk is a state of uncertainty in which some of the possibilities involve a loss.
• Much pessimism about measurement comes from a lack of experience making
measurements. Hubbard, who is far more experienced with measurement than
his readers, says:
• Your problem is not as unique as you think.
• You have more data than you think.
• You need less data than you think.
• An adequate amount of new data is more accessible than you think.
kburns@sagesw.com, @kevinbburns 28

Apply Information Economics for Decision-making
1. Define a decision problem and the relevant variables.
2. Determine what you know.
3. Pick a variable, and compute the value of additional information for
that variable.
4. Apply the relevant measurement instrument(s) to the high-
information-value variable.
5. Make a decision and act on it.
kburns@sagesw.com, @kevinbburns 29

Selecting a measurement method
To figure out which category of measurement methods are appropriate
for a particular case, we must ask several questions:
1. Decomposition: Which parts of the thing are we uncertain about?
2. Secondary research: How has the thing (or its parts) been measured by
others?
3. Observation: How do the identified observables lend themselves to
measurement?
4. Measure just enough: How much do we need to measure it?
5. Consider the error: How might our observations be misleading?
kburns@sagesw.com, @kevinbburns 30

Tools
http://www.howtomeasureanything.com/
kburns@sagesw.com, @kevinbburns 31

Questions & Next Steps
• How many of us know what business/user outcomes and impacts
we’re trying to achieve on our projects?
• Do you have metrics in place to evaluation our progress/success
outcomes and impacts?
• Who want’s help creating some objective measures?
• Where do we go from here?
kburns@sagesw.com, @kevinbburns 32