1.
Evidence-based
Decision-making
for Lean Product
Development
presented by
Kevin Burns
@ MN AEG
Jan. 17, 2017
kburns@sagesw.com, @kevinbburns

2.
Kevin Burns
Coach
Org Change Agent
kburns@sagesw.com, @kevinbburns 2

3.
My work history and experience
kburns@sagesw.com, @kevinbburns 3

4.
kburns@sagesw.com, @kevinbburns 4

5.
Peace Corp Recruitment and Public Affairs
Story of how we used technology to
improve Peace Corp recruitment.
Switch from USPS to email
Switch from manual data entry to wild-card
search in gopher email system and screen
scraping results, an early for of ETL.
Conduct direct email campaigns when spam
still meant ‘meat in a can’
kburns@sagesw.com, @kevinbburns 5

6.
kburns@sagesw.com, @kevinbburns 6

7.
Open Discussion
Who’s measuring value, outcomes, and/or impacts today?
How are you measuring them?
If you’re not measuring them, why not?
Who’s measuring cost?
How are you measuring cost?
kburns@sagesw.com, @kevinbburns 7

8.
The 1st Agile Principle
Our highest priority is to satisfy the customer through early and
continuous delivery of valuable software.
(should we change valuable to beneficial impact?)
How do we define value (impact) and how do we measure it?
Not all Projects (or Features) are created equal.
kburns@sagesw.com, @kevinbburns 8

9.
Is value determined by delivery on time, on budget, and on scope?
Is the scope delighting the customer?
Are they using everything we delivered?
kburns@sagesw.com, @kevinbburns 9

10.
In a survey of 4 products, 65% of the features were rarely or never used.
How much money could have been
saved if we never built them?
In the Waterfall project world, we have to ask
for everything we can think of because capital
will end at the end of the project. Instead we
should be asking what has the most value in
terms of the business outcome and/or impact
and how are we going to measure it.
kburns@sagesw.com, @kevinbburns 10

11.
kburns@sagesw.com, @kevinbburns 11

12.
kburns@sagesw.com, @kevinbburns 12

13.
Marty Cagan Quotes
• Customers don’t know what they
want. It’s very hard to envision the
solution you want without actually
seeing it.
• At least 2/3 of our ideas are never
going to work. The other 1/3 will take
3-4 iterations to get right.
• The role of the product manager is to
discover a product that is valuable,
usable, and feasible. Product, design,
and engineering work together to
arrive at optimal solution.
kburns@sagesw.com, @kevinbburns 13

14.
kburns@sagesw.com, @kevinbburns 14

15.
What we measure is changing
Business Customer
PO, SM, BL
Software Engineering
AD, DD, DA
User
UX, BA, QA, SME
Business
Valuable
Design
Usable
Technically
Feasible
INNOVATIVE
SOLUTION
kburns@sagesw.com, @kevinbburns 15

16.
Lean Startup
• Are we asking what are Minimum
Viable (Valuable) Product and how do
we know when we’ve delivered it?
• Use a scientific method to measure,
learn and pivot or preserver.
• Use meaningful quantitative
objective measure to evaluate impact.
• Can you use A/B testing?
kburns@sagesw.com, @kevinbburns 16

17.
MVP
Innovation
User
UX, BA, QA, SME
Business
Valuable
Design Usable
Software Engineering
AD, DD, DA
Business Customer
PO, SM, BL
Use scientific method
(measurable) to learn
and discovery your
Minimum Viable
(Valuable) Product
(MVP)
Technically
Feasible
MVP innovations emerge
from Conversations kburns@sagesw.com, @kevinbburns 17

18.
Value and/or Impact driven culture
• Are we measuring the Cost vs Benefit at all levels of our work items?
• Portfolio
• Program
• Project
• Feature/Capability
• Story/Requirement
• Tasks/Test
• Are we measuring the Impact our features have on our customers?
• The act of sizing helps us define done
and value
• Use story telling and test statements
create understanding of value and DoD
kburns@sagesw.com, @kevinbburns 18

19.
Don Reinertsen’s Seven Big Ideas
2nd Generation Lean Product Development
1. Understand your economics
2. Manage your queues
3. Exploit variability
4. Enable smaller batches
5. Control WIP and start rates
6. Prioritize based on economics
7. Accelerate feedback
kburns@sagesw.com, @kevinbburns 19

20.
1. Understand Your Economics
• In product development all difficult decisions involve multiple
variables.
• Making decisions that affect multiple variables requires
quantification.
• Doing quantification, isn’t as hard as you might think.
kburns@sagesw.com, @kevinbburns 20

21.
A Typical Question
Should we operate our testing process at 80%
utilization with a 2-week queue, or 90% utilization
with a 4 week queue?
kburns@sagesw.com, @kevinbburns 21

22.
Making Economic Decisions
• Waste
• Cycle time
• Variability
• Efficiency
• Revenue
• Unit Cost
• Value-Added
Proxy Variable Space
Transformations Life Cycle Profits
Economic Space
kburns@sagesw.com, @kevinbburns 22

23.
Modeling Process
Model
Expense
Overrun
Model
Cost
Overrun
Model
Value
Shortfall
Model
Schedule
Delay
Model
Risk
Change
Create Baseline Model
Determine Total Profit Impact of Missing a MOP
Calculate Sensitivity Factors
kburns@sagesw.com, @kevinbburns 23

24.
Guidelines for decision-making
• Set strategic goals/guidelines for decision-making so low-level tactical
decision can be decentralized while still being aligned within strategic
goals/objectives.
• While good intentioned, centralized decision-making is often slow
and suboptimal because it lacks the context of all the variable at play
at tackle level
kburns@sagesw.com, @kevinbburns 24

25.
2. Manage Your Queues
• Many product developers assume higher utilization leads to faster
development.
• They neither measure nor manage the invisible queues in their
process.
• Consequently, they underestimate the true cost of overloading their
processes.
• Such overloads severely hurt all aspects of development
performance.
kburns@sagesw.com, @kevinbburns 25

26.
Effect of Capacity Utilization
10 20 30 40 50 60 70 80 90 100
05101520
QueueSize
% Capacity Utilization
kburns@sagesw.com, @kevinbburns 26

27.
Managing Queues
Cost of Excess Delay
Total Cost
Dollars
Excess Product Development Resources
Minimize Total Cost
to Maximize Profits
kburns@sagesw.com, @kevinbburns 27

28.
Why Queues Matter
• Queues Create…
• Longer cycle time
• Lower Quality
• More variability
• Increased risk
• More overhead
• Less motivation
Managing queues is the key to improving product development economics
kburns@sagesw.com, @kevinbburns 28

29.
3. Exploit Variability
• In manufacturing it is always desirable to reduce variability
• In product development eliminating variability eliminates innovation
• We must understand the specific conditions that make variability
valuable and manage our process to create these conditions
• We need development process that function in the presence of
variability
kburns@sagesw.com, @kevinbburns 29

30.
Asymmetric Payoff and Option Pricing
Expected Price Payoff vs Price
Expected Payoff
ExpectedPayoff
Price
PricePrice
Probability
Payoff
Strike Price
Strike Price
kburns@sagesw.com, @kevinbburns 30

31.
Higher Variability Raises the PayoffExpectedPayoff
Price
Strike Price
Payoff SD = 15
Payoff SD = 5
Option Price = 2, Strike Price = 50,
Mean Price – 50, Standard Deviation = 5 and 15
kburns@sagesw.com, @kevinbburns 31

32.
4. Enable Smaller Batches
• When work products are invisible, batch sizes are invisible
• When batch sizes are invisible, product developers pay little attention
to them
• Many companies institutionalize large batch sizes
• Batch size reduction is attractive because it is fast, easy, cheap,
granular, leveraged, and reversible
• It is a great starting point for LPD
Batch Size Queues Cycle Time
X 0.5 X 0.5 X 0.5
kburns@sagesw.com, @kevinbburns
32

33.
Drawing Review Process
200
10 Weeks
20
1 Week
Unreviewed Drawings
Large Batch Small Batch
kburns@sagesw.com, @kevinbburns
33

34.
Benefits of Small Batch Testing
Higher ValidityFewer Open Bugs
Faster Cycle Time
Early Feedback
Less Debug Complexity
More Efficient Debugging
More Uptime
Smaller Change
Fewer Status Reports
Less Requirement Changes
Faster Learning
Lower Cost Changes
Cheaper Debugging
Cheaper Testing
Less Non-Value-Added
Better Code
Cheaper Correction
Better
Economics
kburns@sagesw.com, @kevinbburns 34

35.
Setting Batch Size
Transaction Cost
Cost
Items per Batch
Economic Batch Size
1 2 3 4 5 6 7 8 9 10
05101520
Total Cost
kburns@sagesw.com, @kevinbburns 35

36.
5. Control WIP and Start Rates
• Many developers incorrectly assume that the sooner they start work,
the sooner they will finish it
• They are constantly tempted to start too much work
• This dilutes resources and causes long transit time through their
processes
• A long transit time hurts efficiency, quality and responsiveness
kburns@sagesw.com, @kevinbburns 36

37.
Little’s Formula
• By constraining WIP in development processes we can control cycle
time
• This approach, which is known as Lean Kanban, is currently growing
rapidly in software development
https://en.wikipedia.org/wiki/Little%27s_law
MeanResponseTime = MeanNumberInSystem / MeanThroughput
kburns@sagesw.com, @kevinbburns 37

38.
Control Number of Active Projects
1
2
3
4
1
2
3
4
COD Savings of Project 1 and 2 Late Start Advantages for Project 3 and 4
Time to Deliver
Time to Deliver Time to Deliver
kburns@sagesw.com, @kevinbburns 38

39.
Avoid Long Planning Horizons
• The further out you plan, the less likely your forecast will be accurate
• Don’t do detailed analysis on things beyond a quarter
• Market conditions change everyday, this can change requirements
• Changing requirements cause churn (waste)
kburns@sagesw.com, @kevinbburns 39

40.
Visual WIP Control Boards
Ready Queue Coding Ready to Test Testing Done
WIP constraints = 10
13
14
15
16
11
10
9
8
7
6
5
4
3
2
112
?
kburns@sagesw.com, @kevinbburns 40

41.
6. Sequence Work Correctly
• The sequence in which work is processed is called the queuing
discipline
• By changing the queuing discipline we can reduce the cost of a queue
without decreasing the size of the queue
• Since manufacturing has homogeneous flows it always uses FIFO
(First-In-First-Out)
• For the non-homogeneous flows of product development other
approaches have better economics
kburns@sagesw.com, @kevinbburns 41

42.
Use FIFO for Homogeneous Flow
First-In First-Out
Cost
of
Delay
1
2
3
A
B
Time
Cost
Delay Cost
Last-In First-Out
Cost
of
Delay 1
2
3
A
B
Time
Cost
Project Duration Cost of Delay
1 3 3
2 3 3
3 3 3
kburns@sagesw.com, @kevinbburns 42

43.
Weighted Shortest Job First (WSJF) for
Non-homogenous flow
High Weight First
Cost
of
Delay
1
2
3
A
B
Time
Cost
Delay Cost
Low Weight First
Cost
of
Delay
A
B
Time
Cost
Project Duration Cost of
Delay
Weight =
COD/Duration
1 1 10 10
2 3 3 1
3 10 1 0.1
1
2
3
160 7
96 % Reduction in COD
kburns@sagesw.com, @kevinbburns 43

44.
7. Create Faster Feedback
• When queues and batch sizes are large feedback is slow
• Slow feedback hurts quality, efficiency, and cycle time
• Feedback speed has enormous economic leverage in product
development, but it is rarely explicitly managed
kburns@sagesw.com, @kevinbburns 44

45.
The Front-Loaded Lottery
• A lottery ticket pays $3000 to winning three digit number
• You can pick the number in two ways:
• Pay $3 to select all three digits at once
• Pay $1 for the first digit, find out if it is correct, then choose if you wish to pay
$1 for the second digit, and then choose if you wish to pay $1 for the third
digit.
kburns@sagesw.com, @kevinbburns 45

46.
Value of Feedback
100%
Spend $1
Savings = $0.90
Savings = $0.99
10%
1%
0 $1 $2 $3
Probability
of
Occurrence
Cumulative Investmentkburns@sagesw.com, @kevinbburns 46

47.
?
Kburns@sagesw.com
@kevinbburns
612-396-7724
kburns@sagesw.com, @kevinbburns 47

48.
References
• Don Reinertsen
• https://www.youtube.com/watch?v=L6v6W7jkwok&spfreload=1#t=16.242347
• https://www.amazon.com/gp/product/1935401009/ref=as_li_tl?ie=UTF8&camp
=1789&creative=9325&creativeASIN=1935401009&linkCode=as2&tag=reinertas
soci-20&linkId=U56AHKXXQVN4VZFY
• https://en.wikipedia.org/wiki/Little%27s_law
kburns@sagesw.com, @kevinbburns 48