EuroSP3 conference Dec 2004, Köln. #2 in what I pretentiously call my Goldratt Trilogy.

1. ROI at the Bug Factory:
Goldratt and Throughput
1st European Conference on Software Projects, Process & People
01 - 03 December 2004 – Köln, Germany
Software Projects Track session WP1
Neil Thompson
Thompson information Systems Consulting Limited
www.TiSCL.com
© Thompson
information
Systems
Consulting Limited
1 www.qualtechconferences.com

2. 1. Introduction
• Primary objective of presentation – to explore some analogies between
processes for software development and implementation and those for
manufacture (of physical things):
– to see whether recent (and not-so-recent) revolutionary changes in
manufacturing have relevance for software
– to relate such analogies to existing work in agile methods
– to relate this thinking to the software development lifecycle whether agile
or not, and to verification, validation & testing (VV&T)
– to help you in planning and managing projects & programmes
• Secondary objectives:
– to use the concept of Return On Investment (ROI) as an introduction to
thinking about process effectiveness, efficiency and improvement
– to use Goldratt’s Theory Of Constraints (TOC) as the basis of considering
manufacturing processes, and for its general thinking tools
© Thompson
information
Systems
Consulting Limited
2 www.qualtechconferences.com

3. What is ROI,
why measure it?
• ROI: return / investment (usually expressed as a percentage):
– “return” could be profit, (value – cost), (net benefit)…
– “investment” may be expressed as net cost
• I’m a “crossover” from EuroSTAR (but wider background)
• So I’ll start with the example of ROI for VV&T:
– to decide how many test environments to buy, and how life-like
– to decide how many testers & QA people to hire
– to estimate duration of testing
– to justify budgets
– to decide how to divide effort between testing & reviews/inspections
– to focus training
– to justify automation
– to motivate staff?
– to make managers look good?
© Thompson
information
Systems
Consulting Limited
3 www.qualtechconferences.com

4. Why ROI difficult for IT
systems
• Although costs relatively easy to measure, not trivial
• Benefits no longer about replacing manual systems; often are
enhancing market position, or opening new market, and those can be
very difficult to forecast
• Misleading / unbalanced precision between costs & benefits
• Misguided cost-cutting (eg through inadequate understanding of
benefits)
• Excuses for not calculating ROI “properly”:
– “we’ve simply got to have it”
– project / system too small to bother
– it’s too difficult
– wouldn’t be believed anyway
• Problem of “construct validity”: you may not really be measuring what
you think you’re measuring Cem Kaner
© Thompson
information
Systems
Consulting Limited
4 www.qualtechconferences.com

5. ROI even more difficult
for VV&T
• Partitioning difficulties:
– how to distinguish as activities, when much is part of develop’t work
– whether / how / why to distinguish by job title
• Biggest problem is quantifying benefits in terms of costs
saved by preventing failures later:
– Boehm’s exponential-type cost rise questioned for modern methods
– isn’t as simple as “early=good”, because some bugs not (easily)
visible until later (though continuous integration helps)
– how can we say fault A would have caused failure B at date X if it
had not been found by test level C?
– even where live failure costs calculated, validity is questionable
– how distinguish poor VV&T from “good VV&T where there are very
few bugs there to find”?
© Thompson
information
Systems
Consulting Limited
5 www.qualtechconferences.com

6. Also other complications,
but let’s persist still
• Additional complications to using ROI for VV&T:
– “Tyranny of numbers” two different books
– whether testing actually reduces risk or is merely (a
misinterpretation of?) “Project Intelligence” Paul Gerrard
– is effectiveness / efficiency distinction feasible / useful?
• But ROI thinking should still be helpful:
– still need to answer those questions, don’t we?
• which VV&T activities to do, which not, which partially?
• (coarse-tune) which people to use?
• (fine-tune) how to maximise effectiveness & efficiency?
– so Agenda…
© Thompson
information
Systems
Consulting Limited
6 www.qualtechconferences.com

7. Agenda
• 2. Goldratt’s Theory Of Constraints
• 3. The code factory and the bug factory
• 4. Beyond the factory: the supply chain, and live systems
• 5. The Systems Development LifeCycle as an
inventory network
• 6. Conclusions & way forward
© Thompson
information
Systems
Consulting Limited
7 www.qualtechconferences.com

8. 2. Goldratt’s Theory Of
Constraints: what it is
• Now, “multifaceted management philosophy”, but
originally emerged in early 1980s from new approach to
materials scheduling in factories
• I already applied Goldratt’s thinking tools to process
definition in software testing STAREast 2003
• Here looking wider, at whole SDLC, going back to
Goldratt’s original arguments in manufacturing The Goal, The Race
• Evidence of TOC success in reducing inventory, lead times
& cycle times; improving due-date performance & revenue
• Associated with Japanese improvements (eg Just In Time,
Lean), but JIT came first, Goldratt was a response and a
claimed improvement; Lean followed a few years later
© Thompson
information
Systems
Consulting Limited
8 www.qualtechconferences.com

9. Parts of TOC using here
• Goal definition, eg make money:
– measures cash flow (survival), profit (absolute) & ROI (relative)
• Throughput better for these measures than cost management
• Adverse effect of inventory on competitive edge:
– it damages not only ROI & cash flow but also profit
• Relationship of throughput to inventory & operating expense:
– throughput (T) = money in (should maximise)
– inventory = money tied up (should minimise, but not damage T)
– operating expense = money out (should minimise, but not damage T)
• Drum-Buffer-Rope as a technique to optimise inventory
• Relationship to Synchronous Flow Manufacturing, Japanese
methods and other quality techniques
© Thompson
information
Systems
Consulting Limited
9 www.qualtechconferences.com

10. Applicability of TOC
beyond manufacturing
• Military logistics
• Marketing, sales & distribution
• Project management
• Measurements, human relationships, medicine etc
• Using technology, eg assessing benefits of functionality
• IT systems development:
– focussing of Goldratt’s Critical Chain on hi-tech projects Robert C Newbold
– methodology design Alistair Cockburn
– “Lean software development” Mary & Tom Poppendieck
– Agile management using Feature Driven Development David J Anderson
© Thompson
information
Systems
Consulting Limited
10 www.qualtechconferences.com

11. But software development
isn’t like manufacturing?
• Software isn’t like hardware
• Intellect adds value less predictably than machines
• The manufacturing part of software development is disk
duplication: “development” is really a design activity
• People are more important than processes
• Software development doesn’t repeat exactly, people are
always tinkering with the processes
• Development involves discovery, production involves
reducing variation
• But I say: does that make all the analogies worthless, or
do they just need interpreting? I suggest the latter…
© Thompson
information
Systems
Consulting Limited
11 www.qualtechconferences.com

12. 3. The code factory and
the bug factory
• No-waste factory
a b c Stated Demonstrations &
requirements acceptance tests
Programming
• Now here’s some waste: meeting/escalation (and loaded
personal memories), or inventory?
a b c a b d Implicit
’ requirements
Documented
? Acceptance tests
I
I
requirements
?
Meeting / escalation to agree
Programming
© Thompson
information
Systems
Consulting Limited
12 www.qualtechconferences.com

13. Specs & unfinished
software are inventory
• Specifications are generally not needed after go-live (I will come later to
exceptions) so they are not end-product, they are work-in-progress
(especially intermediate levels like functional & non-functional specs)
• Untested software, and even finished software not paid for, is also
inventory
• Iterative lifecycles help if “adaptive” (product-based) rather than
“transformational” (where specifications multiply!)
a b Revised & new requirements
a b c
’ ’
I
May include
Programming Programming
redesign
© Thompson
information
Systems
Consulting Limited
13 www.qualtechconferences.com

14. Full W-model bulges with
Business
objectives
inventory
Test against Post-implement-
Make
into Verify ation review
Requirements Verify & Validate Acceptance Acc. retest,
Statement RS, + spec. AT test fix & reg. test
Validate
(incl.
“QA”) Functional Verify & Validate System Sys. retest,
Spec. FS, + spec. AT test fix & reg. test
Retest lower
Technical V&V TD, Integrat- Int. retest, levels
+ spec. IT fix & reg. test where
Design ion test necessary
Module V&V MS, + Unit Unit retest,
Specs. spec. UT test fix & reg. test
Code
© Thompson Static-check
information
Systems
Consulting Limited
14 www.qualtechconferences.com

15. In a factory, small
batches reduce inventory
Stages &
units / hour (i) 1.3
(ii) 10.0
Multi-batch
1000 (iii) 1.0 200 200 200 200 200
(ie “iterative”)
Single-batch
(ie “waterfall”) (iv) 10.0
(v) 2.0
0 1 2 3 4 0 1 2 3 4
months months
Inventory Inventory
© Thompson Based on Goldratt, The race (North River Press 1986)
information
Systems
Consulting Limited
15 www.qualtechconferences.com

16. Why inventory damages
competitiveness (i)
Goldratt’s objectives for My IT analogies
competitiveness
Product: 1. quality Quality & scope: 1. few defects now,
reliable in future
2. engineering 2. amount & sophistication of funct-
ionality, non-functional attributes
Price: 3. higher margins Low-cost: 3. economical development
& testing so flexible pricing
4. lower investment per 4. no environments panic, lower break-
unit even point & flexibility to compete
Responsiveness: Timely delivery: 5. rarely / never late
5. due-date performance
6. shorter quoted lead- 6. rapid application development!
time
© Thompson
information
Systems
Consulting Limited
16 www.qualtechconferences.com

17. Why inventory damages
competitiveness (ii)
Objectives for Advantages of low inventory
competitiveness (and/or iterative lifecycles!)
Product: 1. quality Defect detection & fixing sooner after
their introduction, fewer recurrences
2. engineering Quicker realisation of improvements
Price: 3. higher margins Lower probability of needing overtime
4. lower investment per Lower probability of needing additional
unit machines (test environments)
Responsiveness: More reliable materials forecasting
5. due-date performance (smoother staff recruitment)
6. shorter quoted lead- Shorter overall time (already shown on
time slide 15)
© Thompson
information
Systems
Consulting Limited
17 www.qualtechconferences.com

18. Throughput relationships,
why good
Net profit ROI Cash flow
Operating
Throughput Inventory expense
High quality
Good scope
Low people costs
Low machine costs
Rarely late
Rapid
Competitive edge Extended from Goldratt, The race (North River Press 1986)
© Thompson
information
Systems
Consulting Limited
18 www.qualtechconferences.com

19. Drum-buffer-rope
• Optimise throughput by:
– (1) drumbeat based on constraining stage (a) & orders (b)
– (2) buffer to protect constraining stage from upstream disruptions
– (3) rope to prevent leader extending gap on constraining stage
• For subassemblies feeding in, have additional buffers
“troops marching” materials flow
buffer
raw materials (1)
in constraining
”leader” stage (a) assembly
(2)
(3) buffer
subassembly orders (b)
© Thompson
information
Systems
Consulting Limited
19 www.qualtechconferences.com

20. Buffer management
too
cautious
premature
interruption release
of inventory healthy of inventory
in in
y hours
x days
too
lean
© Thompson
information
Systems
Consulting Limited
20 www.qualtechconferences.com

21. Tracing buffer holes back
to process steps to fix
3 5
2 1
4
© Thompson
information
Systems
Consulting Limited
21 www.qualtechconferences.com

22. Relationship to other
quality approaches
• To recap: Goldratt argues why DBR is better than JIT
• In addition to TOC, use lean techniques to minimise waste
• TOC project management (critical chain, see Newbold):
– deprecates multi-tasking
– translates buffer management to contingency placement
– recommends also time management & Pareto techniques
– uses risk management to calculate buffer sizes
• Goldratt is more process-oriented than product- (as was
Deming)
© Thompson
information
Systems
Consulting Limited
22 www.qualtechconferences.com

23. Is VV&T a factory?
• VV&T produces (depending on your viewpoint):
– “project intelligence”;
– risk reduction;
– a mountain of test plans, scripts, logs, reports etc etc;
– bug reports
• Customer does not want to pay for bugs
• Bugs are imperfections on inventory rather than inventory
themselves; may not be many in the first place
• “Removal of bugs” is not an easy end product to grasp, and
the developers do it anyway
• So the code factory & the bug factory are two aspects of
the same thing; VV&T can help identify constraints
© Thompson
information
Systems
Consulting Limited
23 www.qualtechconferences.com

24. 4. Beyond the factory:
supply chain &
live systems
• TOC/JIT says that functionality should be pulled by
customers rather than pushed by analysts / vendors (but
why does this often not happen?)
• Software passing testing but not yet paid for is still WIP
• Software development has its value chain
• Poppendieck principles include “decide as late as
possible”: contradiction with Newbold?
• Biggest point here is that most systems go on for years,
and need some documentation or staff continuity (this
point receives insufficient attention often)
© Thompson
information
Systems
Consulting Limited
24 www.qualtechconferences.com

25. 5. SDLC as an
inventory network
• TOC suggests choosing SDLC with a view to what else is wanted
other than working software, eg training material, maintenance
documentation, regulatory audit
• This affects value of inventory
• Cockburn uses TOC principles, of which one consequence is effect on
method
• Another contradiction? Advice on handling slack
• TOC process improvement can monitor buffers, and is not just for
agile methods
• Can still think of VV&T ROI as tangibles & intangibles, even if we
can’t (yet?) calculate
• But via the throughput route to ROI we can still seek to optimise
© Thompson
information
Systems
Consulting Limited
25 www.qualtechconferences.com

26. Inventory moving through
SDLC
Amount of
functionality Requirements Inventory in
this stage Lead time for
Specification of process
this stage
Design of process
Inventory in
process
Programming & Within each stage
overall
unit testing of testing, can
Integration subdivide by
testing pass/fail,
bug states etc
System
testing
Acceptance
testing
Live and
paid-for
Date
If lines not approx parallel, inventory is growing
© Thompson Based on Anderson, Agile management for software engineering (Prentice Hall 2004)
information
Systems
Consulting Limited
26 www.qualtechconferences.com

27. SDLC as an
inventory network (cont’d)
raw materials
in Where is/are the
Requirements
constraining stage(s)?
Where should buffers
Specification
be / not be? Acceptance
testing
order(s)
System
testing
Design assembly
Integration
testing sub-assemblies
Programming Unit
testing
© Thompson
information
Systems
Consulting Limited
27 www.qualtechconferences.com

28. 6. Conclusions and
way forward
• Although original Goldratt ideas don’t translate directly to
software development, after so much success and so many
analogies, can’t ignore (and thinking tools useful for
“anything”)
• Several of the later interpretations have plausible messages
• Although a few authors have already applied TOC to agile
methods, I hope that by going back to the original
principles in some detail I have widened consideration to
potentially any SDLC, and provided a platform for me and
others to develop this thinking further…
© Thompson
information
Systems
Consulting Limited
28 www.qualtechconferences.com

29. Way forward
project-informing
Published Constraint-led
methodologies thinking process definition
(“Tailorable”?) tools & improvement
Context → “Methodology Maintenance regime
per project”
Circumstance-driven
© Thompson
information
Systems
Consulting Limited
29 www.qualtechconferences.com

30. Summary
Heavy industry, waste, burnout Light industry, lean, cool fumes
“How clean is your factory?”
© Thompson
information
Systems
Consulting Limited
30 www.qualtechconferences.com

31. References &
acknowledgements
• Main references:
– Goldratt, Eli: The race & Necessary but not sufficient
– Newbold, Robert: Project management in the fast lane
– Mellis, Werner: Process & product orientation
– Cockburn, Alistair: Agile software development
– Poppendieck, Mary & Tom: Lean software development
– DeMarco & Lister: Waltzing with bears
– Anderson, David: Agile management for s’ware engineering
– Boehm & Turner: Balancing agility & discipline
• Acknowledgements:
– Jens Pas EuroSTAR 1998, metrics
– Greg Daich STAREast 2002, documentation superstitions
© Thompson
information
Systems
Consulting Limited
31 www.qualtechconferences.com

32. Contact details
Neil Thompson
NeilT@TiSCL.com
Questions?
www.TiSCL.com
23 Oast House Crescent
Farnham, Surrey, England
GU9 0NP, United Kingdom
phone +44 (0)7000 NeilTh (634584) or
+44 (0)7710 305907
© Thompson
information
Systems
Consulting Limited
32 www.qualtechconferences.com