So WHY invest in better project management and process improvement? This paper points to the answer. The lessons are very powerful. Cut defects and costs by 60%. Read on ... The business case for software process improvement may also be the business case for project management - in as much as CMMi process improvement implements the fundamentals of project management. This paper leverages work done by Larry Putnam, Stan Rifkin and Joe Kolinger in applying metrics to understand productivity and cost improvements in technology projects.

1. Is Your Investment in Software Process Improvement Paying Off?
By
Joe Kolinger
Introduction
The Software Engineering Institute (SEI) Software Development Maturity Assessment
Methodology is used to assess the software development capability of organizations. Research
by Lawrence Putnam of Quantitative Software Management (QSM) demonstrates a strong
relationship between Capability Maturity Model (CMM) maturity level and the QSM ‘Productivity
Index’ (PI). Specifically, rising CMM levels result in higher Productivity Indices, which result in
lower development costs.
In a nutshell, higher Productivity Index values are associated with projects that cost less, finish
faster, and have fewer defects. Ideally the CMM process improvements should be associated
with more efficient projects and better quality. What’s covered in this article is that the QSM
methodology, benchmark database, and tool set measure of the benefits of CMM improvements.
This article points to the economic benefit of effective software process improvement, and the
role that measurement plays in proving it.
Background
Many companies have undertaken software process improvement (i.e., Software Engineering
Institute’s CMM/I) with the hope that better process will somehow produce better results. For
example by moving to CMMI level 3 they would expect to experience projects with shorter
schedules, reduced costs, improved reliability, fewer emergencies, etc. However, without a
metrics plan and a quantitative toolset in place – apart from “anecdotal evidence” – they will never
really know. Even worse, process focus, without the right measures encourages individuals to
default to rote compliance with ‘process.’ Eventually, lacking any believable measures of
improvement the organization abandons the improvement effort altogether. After all, process
improvement requires discipline and continuous investment.
If you’re going to pursue software process improvement, protect the investment from the outset
with a solid metrics plan and benchmark database.
Start a Basic Metrics Plan
Rather than trying to measure too much, organizations need a basic ‘starter set’ of metrics for
basics such as duration, effort, size, and defects. Furthermore these metrics should not be used
to measure individuals, but rather to better understand the software development process,
making continuous improvement for repeatable success. Metrics misapplied will submarine data
quality and put a halt to improvement. Fear drives out learning. Without learning there is no
improvement.
Apply a Measurement Framework
Software projects are ‘different’ from other projects, such as construction. Software does not
obey the laws of physics and science. Rather it requires human learning, discovery, problem
solving and communication, which makes schedule prediction more difficult. (When asked to
estimate “how long” to complete an unfamiliar programming task the developer responds, “Don’t
know. How long does it take to catch a fish?”)
69 Sandy Creek Way – Novato, CA 94947
(415)898-2300 joe@kolinger.net

2. Software development projects – or most design-type problem solving projects follow a non-linear
resource staffing pattern. The sample Raleigh Curve below (Figure 2) shows the common
pattern. At the beginning of the project there is a staffing ramp-up (Physical Design), the project
peaks at the conclusion of programming and the beginning of testing, and finally the long tail
(Testing and Debugging) represents the effort to find and remove bugs over a considerable time
frame.
Software development and circuit design projects tend to follow this effort/staffing distribution.
[Figure 2] – The entire lifecycle of a software project follows a curve of rising and then falling
manpower. The long tail of the curve represents the many years of so-called software
maintenance.
Also included in the QSM Measurement Framework are these measures:
 Effort – Person hours of work
 Duration – Elapsed days
 MBI (Manpower Buildup Index) – Rate at which people are added to the project
 Defect Density – the number of bugs to be removed
 Size – Some characterization of what is delivered.
 Productivity Index – a macro measure of the organization’s development efficiency
Understanding the Productivity Index at a Glance
The software process Productivity Index (or PI) is a QSM metric, representing the level of an
organization's software development efficiency. The PI is a macro measure of the total
development environment. PI values from 1 to 40 have been adequate to describe the full range
of projects seen so far.
Low PI values typically are associated with poorer working environments, poor tools and/or high
product complexity. Higher values are associated with good environments, tools and
management and well-understood, straightforward projects [Ref. 1, 5].
"Productivity" encompasses many important factors in software development, including:
69 Sandy Creek Way – Novato, CA 94947
(415)898-2300 joe@kolinger.net

3.  Management influence
 Software development process and methods
 Software development tools, techniques and aids
 Skills and experience of development team members
 Availability of development computer(s)
 Complexity of application type
Note that the PI is calculated from the size, schedule and effort that were applied to a completed
project. This means that the PI is objective, measurable and capable of being compared on a
numeric scale.
Projects normalized around the PI can be meaningfully compared to one another. Without this
normalization projects’ performance cannot meaningfully be compared. For example, project A
took 6 months to complete, and project B took 4 months to complete. What conclusion can be
drawn? None. But if both projects have PI’s then we might see that one had greater size, or
greater complexity – or had a team that ramped too slowly. In any event, an organization that
gets a bead on its PI has incredibly valuable information to help estimate future projects.
CMM Transition Breakpoints
Research conducted by QSM on the relationship between CMM Level and PI shows in table 3.
The benchmark database consisted mostly of Level 1 and Level 2 projects (which characterize
the world), therefore the relationships are statistically strongest at these levels. In the Business
Systems column we can see that average PI improves from a 12 to a 17 as the organization
graduates to Level 2.
CMM Business Systems Engineering Systems Real-Time Systems
Level PI Value PI Value PI Value
I 12 10* 6*
II 17 15 9
III 19.5 18 11.5
IV 22 20.5 14
V 25 23 16.5
[Table 3] Transition Breakpoints for Three Application Types - Note: * Estimated
PI Improvement Reveals Economic Savings
The graph in Table 4 shows results for three Business Systems projects of similar size and
complexity. Process improvements – which are related to PI improvements - have lead to more
efficient development capability, and a much lower cost. The transition from CMM Level 1 to 2
shows a 50% cost reduction. The transition from Level 1 to Level 3 shows a 75% reduction in
costs and a 250% improvement in reliability.
But if we’re to realize a 50% savings, how could it be reasonably spent? How about these
options:
 Finish faster
 Use fewer people
 Deliver more scope
 Use the saved money on other projects
 Give back to the Business
69 Sandy Creek Way – Novato, CA 94947
(415)898-2300 joe@kolinger.net

4. CMM PI Duration Effort Peak Staff Mean Time to Cost
Level Defect
1 12 15 123 PM 12 1.76 Days $1.2 M
2 17 12 67 PM 8 2.65 Days $0.67 M
3 19.5 10 31 PM 5 4.85 Days $0.31 M
[Table 4] – The Economic Value of Software Process Improvement for Business Systems
[NOTICE the cost at level 1 vs. cost at level 2]
The related graph in Figure 5 displays the Raleigh curve effort distributions for the three Business
Systems projects. Note that project duration and peak staffing decrease with CMM level
improvements. This is good news.
[Figure 5] – The Economic Value of Process Improvement (Courtesy of QSM)
Is your software process improvement effort paying off?”
Companies rightly undertake process improvement initiatives looking for some kind of
improvement in delivery and cost. To some the method of choice is the CMM, to some it’s Agile
methods, to others it’s custom processes and project management offices (PMO). But a word of
caution: the project and organizational processes that can be implemented have the possibility –
but not guarantee - of improving software delivery. Organizations frequently become totally lost
in process, and confuse ‘process sophistication’ with ‘real maturity’.
Using the right measurement framework we can actually tell the difference between process
improvement efforts that are paying off and those that are not. Are we finding more bugs earlier
69 Sandy Creek Way – Novato, CA 94947
(415)898-2300 joe@kolinger.net

5. in the lifecycle? Are the schedules becoming shorter? Is the Productivity Index increasing? Are
costs dropping? Do we require fewer people to get the project completed? Is the user finding
fewer bugs? Which techniques are giving us the best return on investment? Are project
estimates improving? Are fewer dates slipping? What are the priority areas we should next focus
on? The point here is that the right measurement framework helps us to know and intelligently
manage the investment in process improvements to yield the best return on that investment.
You may have process improvement efforts from which you are already seeing benefit. It’s
possible that what you are seeing is only a fraction of what is possible. Without a benchmark
database you will not know the extent to which you are performing beneath your capability.
Conclusion -
QSM’s research suggests a strong relationship between CMM level and the QSM Productivity
Index. These improvements lead to significantly reduced software development costs. With a
metrics plan and the right measurement tool set, meaningful measures position an organization to
manage significant economic benefit.
Considering that the QSM tool set has a framework including industry data from over 8,000
projects, and a method to normalize project experiences, it is suitable as a software project
management estimation and analysis tool.
This article points to the business case for software improvement. If we understand the nature of
CMM level 2 and 3 improvements, they are primarily focused on project management. For this
reason this article also suggests the business case for investing in project management.
References
1. Putnam, Lawrence H. and Ware Myers, Measures for Excellence: Reliable
Software On Time Within Budget, Prentice Hall, Englewood Cliffs, NJ, 1992, pp. 32-36.
2. Humphrey, Watts, David H. Kitson and Tim C. Kasse, The State of Software Engineering
Practice: A Preliminary Report, CMU/SEI-89-TR-1, ESD-TR-89-01, Software Engineering
Institute, Carnegie-Mellon University, Pittsburgh, Feb. 1989, 27 p.
3. Putnam, Lawrence H., “The Economic Value of Moving up the SEI Scale”,
Managing System Development, Applied Computer Research, Inc., July 1994
4. Putnam, Lawrence H., Arlyn D. Schumaker and Paul E. Hughes, Economic
Analysis of Re-Use and Software Engineering Process, (Final Draft Report) TR-
9265/11-2, prepared for Standard Systems Center, Air Force Communication
Command, Maxwell Air Force Base, Alabama 36114, under contract FO1620-90-D-0007,
February 1993.
5. Putnam, Lawrence H. and Ware Myers, Executive Briefing: Managing
Software Development, IEEE Computer Society Press, Los Alamitos, CA, 1996, 79 p. Linking the
QSM Productivity Index with the SEI Maturity Level,
Version 6, 2000
6. Putnam, Lawrence H, Linking the QSM Productivity Index with the SEI Maturity Level. July,
2000
7. Kolinger, Joe., Seven Signs You Have a Bad Project Estimate, Project Management Institute
SF-Bay Area Chapter, January 2010
69 Sandy Creek Way – Novato, CA 94947
(415)898-2300 joe@kolinger.net

6. 8. Kolinger, Joe, "Practical Lessons in Software Estimation" – Fireman's Fund Insurance
Company, 2002, Novato, CA
9. Kolinger, Joe. "Building the Business Case for Software Project Management" -
Quantitative Software Management SLIM User Conference, Washington, D.C.
STC'96 - (U.S. Air Force - Software Technology Conference), Salt Lake City, Utah
About Us
Kolinger Associates provides solutions and advice for better estimating and managing large
projects. With our help you will …
Spend your money a little differently … and get a much better result.
Also see, “7 Signs You Have a Bad Project Estimate … and What to Do About It”:
http://www.kolinger.net/2010/02/03/7-signs-you-have-a-bad-project-estimate-and-what-you-can-
do-about-it/
Go to www.kolinger.net for more details or email joe@kolinger.net
69 Sandy Creek Way – Novato, CA 94947
(415)898-2300 joe@kolinger.net