Process and measurment

1. 942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 22
Software Project and Process Measurement
Dr. Christof Ebert
Vector Consulting Services
Software measurement is the discipline that ensures that we stay in control and can replicate successful processes. It applies to
products (e.g., ensuring performance and quality), processes (e.g., improving efficiency), projects (e.g., delivering committed
results), and people (e.g., evolving competence). This article discusses software measurement and provides practical guidance for
project and process measurement.
G oals that are measured will be
achieved. In almost every area of life,
setting goals and monitoring achievement
tions, this is demanded by laws (such as
the Sarbanes-Oxley Act) and liability regu-
lations. For all organizations, it is a simple
reduce the differences between status
and objectives.
The E-4 measurement process is
are the foundations for success. Take a question of accounting and finance con- based on the Deming Cycle (Plan, Do,
relay team in track or swimming. What dri- trol. The measurement process is an Check, Act) and extends classic measure-
ves these athletes to continuously inherent part of a business process (see ment paradigms, such as the goal-ques-
improve? There are always competitors to Figure 1). Therefore, software engineer- tion-metric approach, by adding an imme-
beat, a number 1 ranking to achieve, or a ing—as part of the product development diate action-focus. It follows the observa-
record to break. Many of the same drivers business process—also needs controlling tion that to effectively and continuously
apply to the software industry. Software and measurement. Software measure- improve a process, it is important to first
development is a human activity and, as ments include performance measurement, stabilize it, keep it in its specification lim-
such, demands that organizations continu- project control, and process efficiency and its, and continuously improve it [2]. The
ously improve their performance in order effectiveness. major difference is E-4’s clear focus on
to stay in business. Software measurement Measurements are management tools. goal-oriented measurement and execution
is a primary approach to control and man- Consequently, measurements must be of decisions at the end of the four steps.
age projects and processes and to track governed by goals such as reducing pro- The E-4 measurement process is gov-
and improve performance. ject risks or improving test efficiency, oth- erned by ISO/IEC 15939 [3], a standard
The way software measurement is used erwise they simply build up into data ceme- that summarizes how to plan and imple-
in an organization determines how much teries. Figure 1 shows this objective-driven ment a measurement process. It consists
business value that organization actually generic measurement process, known as of two parts: The first part establishes the
realizes. Software measurements are used E-4 [1]: measurement program and prepares it
to: 1. Establish concrete improvement or within the project or organization. The
• Understand and communicate. control objectives and the necessary second part is about execution: You
• Specify and achieve objectives. measurement activities. extract data, evaluate it, and execute cor-
• Identify and resolve problems. 2. Extract measurements for the estab- rective actions based on the outcome of
• Decide and improve. lished performance control needs. the measurement analyses. The second
Today, improving a business (and its 3. Evaluate this information in view of a part—driven by the first part—sets the
underlying processes) is impossible with- specific background of actual status standards and defines what to do with the
out continuous measurements and con- and objectives. measurements. It is not enough to simply
trol. For some industries and organiza- 4. Execute decisions and actions to collect numbers and report them.
Measurements are only effective if they
Figure 1: A Generic Measurement Process are embedded as tools to support deci-
sion-making and to drive the implementa-
tion of decisions. For the fundamentals of
software measurement and practical solu-
tions, I refer to the book “Software
Measurement” [1].
Project Measurement
Project measurement is the major applica-
tion of software measurement. Software
measurement is an absolutely necessary
precondition for project success, but only
one-third of all software engineering com-
panies systematically utilize techniques to
measure and control their product releas-
es and development projects [4].
The goal of project measurement is to
master the project and finish it according
to commitments. What is needed is a way
to determine if a project is on track or
22 CrossTalk The Journal of Defense Software Engineering July/August 2009

2. 942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 23
Software Project and Process Measurement
not. Whether it is embedded software
engineering, the development of a soft-
ware application, or the introduction of a
managed IT-service, demand outstrips the
capacity of an organization. As a result,
I’ve observed the acceptance of impossi-
ble constraints in time, content, and cost;
as a direct consequence of acceptance
comes an increase in churn and turmoil—
as well as budget overruns, canceled pro-
jects, and delays. Still, far too many pro-
jects fail to deliver according to initial
commitments simply because of insuffi-
cient or no adequate measurement.
While many organizations claim that
project work is difficult due to changing
customer needs and high technology
demands, the simple truth is that they do
not understand the basics: estimation,
planning, and determining progress dur- Figure 2: Measurements Provide a Closed Feedback Loop for Effective Corrective Action
ing the project. Consequently, they fully input to the project beyond the original is exactly this pattern of exciting technolo-
lose control once customer requirements project targets. Making the actions effec- gy and skills—combined with fast growth
change. tive is the role of the project manager. but insufficient engineering and manage-
Project control can help in avoiding A client recently asked us for help in ment processes—that eventually hits many
these problems by answering a few simple improving engineering efficiency at their companies in embedded software develop-
questions derived from the following organically grown embedded software ment. Finally, the client’s customers con-
management activities: business. Software costs initially did not cluded that despite all of the technical
• Decision-making. What should I do? matter when there were only a few engi- advantages, processes and practices were
• Attention directing. What should I neers. As things progressed, software below current professional software engi-
look at? dominated hardware and cost increases neering expectations. A natural first step
• Performance evaluation. Are we were out of control. Customers liked the on our side was to introduce a lean yet
doing a good or bad job? technology but increasingly found quality effective measurement program.
• Improvement tracking. Are we issues and, when the customers made To get started without much overhead,
doing better or worse than in the last audits, did not really find a decent engi- our team recommended a lean set of pro-
period? neering process. Our Q&A with the client ject indicators [1, 5, 6]. They simplified the
• Target setting. What can we realisti- went like this: selection by reducing the focus on project
cally achieve in a given period? Us: “How do you set up a software pro- tracking, contractor oversight, and pro-
• Planning. What is the best way to gram management perspective. Here is
ject?”
achieve our targets?
The Client: “Well, we take the require- our short list of absolutely necessary pro-
Project monitoring and control is defined as
ments, build a team of engineers, and ject measurements:
a control activity concerned with identify-
follow our V-shaped master plan.” • Requirements status and volatility.
ing, measuring, accumulating, analyzing,
and interpreting project information for Us: “Is the planning typically accurate?” Requirements status is a basic ingredi-
strategy formulation, planning, and track- The Client: “No, but we don’t have any ent to tracking progress based on
ing activities, decision-making, and cost better basis.” externally perceived value. Always
accounting. As such, it is the basic method Us: “How do you make sure you have the remember that you are paid for imple-
for gaining insight into project perfor- right quality level?” menting requirements, not just gener-
mance and is more than just ensuring the The Client: “Well, we do a unit test, inte- ating code.
overall technical correctness of a project. gration test, and system test. But, in • Product size and complexity. Size
The most important elements are the exis- fact, we test too much and too late.” can be measured as either functional
tence of a closed loop between the object Us: “Is your cost in line with expectations size in function points or code size in
being controlled, the actual performance or could you do better?” lines of code or statements. Be pre-
measurements, and a comparison of tar- The Client: “We really don’t know about pared to distinguish according to your
gets against actuals. Figure 2 shows this how this all is measured and what to measurement goals for code size
control loop. The project with its underly- learn from measurements. That’s why between what is new and what is
ing engineering processes delivers results, we invited you.” reused or automatically generated
such as work products. It is influenced and In terms of software and technology, code.
steered by the project goals. Project con- this client was way above average. • Effort. This is a basic monitoring
trol captures the observed measurements However, it was unclear to management parameter to assure that you stay with-
and risks and relates them to the goals. By and engineers how to assess status, miti- in budget. Effort is estimated up-front
analyzing these differences, specific gate risks, and improve performance. The for the project and its activities.
actions can be taken to get back on track project managers had no history database Afterwards, these effort elements are
or to ensure that the project remains on to decide on release criteria. Requirements tracked.
track. These actions serve as an additional were changing, but without any control. It • Schedule and time. The next basic
July/August 2009 www.stsc.hill.af.mil 23

3. 942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 24
Process Replication
monitoring measurement is ensuring key measurements to indicate quality. (e.g., cost to complete). Such dashboards
that you can keep the scheduled deliv- Reliability models are established to provide information in a uniform way
ery time. Similar to effort, time is bro- forecast how many defects still need to across all projects, thus not overloading
ken down into increments or phases be found. Note that quality attributes the user with different representations and
that are tracked based on what is deliv- are not only functional but also relate semantics to wade through. All projects
ered so far. Note that milestone com- to performance, security, safety, and within the organization must share the
pletion must be lined up with defined maintainability. same set of consistent measurements pre-
quality criteria to avoid poor quality These measurements must be actively sented in a unique dashboard. A lot of
being detected too late. used for the weekly tracking of status, time is actually wasted by reinventing
• Project progress. This is the key mea- risks, and progress (e.g., increment avail- spreadsheets and reporting formats when
surement during the entire project exe- ability, requirements progress, code deliv- the project team should be focused on
cution. Progress has many facets: ery, defect detection), while others are creating value.
Simply look to deliverables and how used to build up a history database (e.g., Measurements—such as schedule and
they contribute to achieving a project’s size, effort). Most of these measurements, budget adherence, earned value, or quali-
goals. Typically, there are milestones such as defect tracking, are actually by- ty level—are typical performance indica-
for the big steps, and earned value and products from operational databases. This tors that serve as traffic lights on the status
increments for the day-to-day opera- ensures sufficient data quality to compare of the individual project. Only projects in
tional tracking. Earned value tech- project status across all projects of a port- amber and red status that run out of agreed
niques evaluate how results (such as folio. External benchmarks (such as pro- variance (which, of course, depends on
implemented and tested requirements vided in [1,6]) help in bootstrapping a the maturity of the organization) would
or closed work packages) relate to the measurement program as they immediate- be investigated. The same dashboard is
effort spent and elapsed time. This ly point towards inefficiency and below- then allowed to drill down to more
then allows for estimating the cost to average performance. detailed measurements to identify root
complete and remaining time to com- To ease monitoring and avoid getting causes of problems. When all projects
plete the project. lost in the fog of numbers, projects follow a defined process and utilize the
• Quality. This is the most difficult should aggregate the relevant information same type of reporting and performance
measurement, as it is hardly possible to in a standardized dashboard. Figure 3 tracking, it is easy to determine status,
accurately forecast whether the prod- shows a simplified dashboard of how we identify risks, and resolve issues—without
uct has already achieved the right qual- have utilized it with many clients. It getting buried in the details of micro-
ity level that is expected for opera- includes milestone tracking, cost evolu- managing the project.
tional usage. Quality measurements tion, a selection of process measurements,
need to predict quality levels and track work product deliveries, and faults with Process Measurement
how many defects are found compared status information. There can be both Software engineering processes determine
to estimated defects. Reviews, unit test, direct measurements (e.g., cost) as well as the success of organizations as well as
and test progress and coverage are the indirect measurements and predictions how they are perceived in the marketplace.
Figure 3: Measurement Dashboard Overview
24 CrossTalk The Journal of Defense Software Engineering July/August 2009

4. 942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 25
Project and Process Measurement
Purchasing organizations worldwide are
using tools such as CMMI to evaluate
their suppliers, be it in automotive, infor-
mation/communication technologies, or
governmental projects [1, 7]. Markets have
recognized that continuous process
improvement contributes substantially to
cost reductions and quality improvement.
An increasing number of companies are
aware of these challenges and are proac-
tively looking at ways to improve their
development processes [8]. Suppose a
competitor systematically improves pro-
ductivity at a relatively modest annual rate
of 10 percent (as we can see in many soft-
ware and IT companies). After only three
years, the productivity difference is (1.13)
= 1.33, or a 33 percent advantage. This
directly translates into more capacity for
innovation, higher margins, and improved
market positioning.
Software measurement is a necessary Figure 4: Applying the E-4 Measurement Process to Achieve Improvements
precondition to performance improve- decision-making. They must be used for added or changed following a customer
ment. The concept of objective-driven effectively communicating status and question—rather than including the cus-
process improvement will focus processes progress against the objectives set for the tomer in the trade-off and impact analysis.
on the objectives they must achieve. business, process, or project. Measure- When speaking to client-customers, this
Processes are a means to an end and need ments, both direct and indirect, should be became even more evident: Customers
to be lean, pragmatic, efficient, and effec- periodically evaluated in conjunction with perceived such ad-hoc changes as a weak-
tive—or they will ultimately fail, despite all the driving objectives and to identify ness and missed clear guidance and leader-
the push one can imagine. Figure 4 shows problems or derive decisions. Example: ship on feature content.
this goal-driven relationship from business The selected performance indicator was
objectives to concrete annual performance schedule predictability, measured as the Step 4: Commit to concrete
objectives (on an operational level) to spe- normalized delays compared to the origi- improvement objectives
cific process performance measurements. nally agreed deadline. Schedule changes These improvement objectives should
What follows are eight integral soft- (after a project had started) were not con- directly address the mentioned weaknesses
ware measurement steps, showing how sidered in this measurement. This avoided and simultaneously support the overarch-
objective-driven process improvement is the argument that a specific delay was jus- ing business objective. We use the acronym
translated into concrete actions, and how tified due to changing requirements. Once SMART to outline good objectives:
software measurements are used to such an excuse is accepted, delays typical- • Specific (precise).
improve processes. Each step includes an ly increase—as do costs (due to the • Measurable (tangible).
example—all taken from the same medi- changes). In this case, we found almost • Accountable (in-line with individual
um-sized company that Vector recently unanimous agreement from product man- responsibilities).
consulted—showing how we put each step agers and business owners who found that • Realistic (achievable).
into action. the lack of schedule changes helped avoid • Timely (suitable for current needs).
the downward spiral of requirements These objectives are reviewed and
Step 1: Identify the organization’s changes, delays, and cost overruns. approved by upper-level managers before
improvement needs from its business proceeding further. This ensures that pri-
goals Step 3: Identify the organization’s orities are appropriate and that nothing
These business goals provide the guidance hot spots, such as areas where they relevant has been overlooked or misun-
for setting concrete engineering perfor- are feeling the most pain internally derstood. Example: Two improvement
mance improvement objectives on a or from customers objectives, improving estimations and
short-term basis. Example: The business Typical techniques include root cause improving requirements development,
goal with our client was to improve rev- analysis of defects and customer surveys. were agreed upon. The respective perfor-
enues and cash flows and to reduce cost of Example: Average schedules in the com- mance targets were agreed to in a manage-
non-performance from schedule delays. pany were 45 percent behind initial com- ment seminar to achieve buy-in. Each pro-
mitments. A root cause analysis of delays ject was required to have two estimates
Step 2: Define and agree on the was performed: 40 percent of delays came where the first was allowed to deviate by
organization’s key performance from insufficient project management; 30 20 percent and the second to deviate by 10
indicators (KPIs) percent from changing requirements; 20 percent. After the project started, the
Such KPIs are standardized across the percent from supplier delays; and 10 per- requirements change rate was required to
organization and ensure visibility, account- cent from other causes. Looking to the be under 20 percent—except that cus-
ability, and comparability. Naturally, KPIs highest-ranking root causes, we found tomers would, after a clear decision-mak-
must relate to the business goals. insufficient planning and control within ing process, agree on trade-offs and pay
Measurements should drive informed the project. Often, requirements were for the changes. Time-boxing and incre-
July/August 2009 www.stsc.hill.af.mil 25

5. 942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 26
Process Replication
mental development with prioritized the leadership of product management. hide useful and necessary information.
requirements was then introduced to We have seen reports on software pro-
achieve improvement objectives. Require- Step 8: Implement improvements grams with more than 50 pages full of
ments priorities were agreed upon across and deliver tangible results graphs, tables, and numbers. When asked
impacted stakeholders from product Implement the agreed-upon changes to about topics such as cost to complete,
management, engineering, and market- operational projects and measure expected cost of non-quality after han-
ing/sales before each new project started. progress against the committed improve- dover, or time-to-profit, the organization
ment objectives. Example: Performance did not have a single number. Sometimes
Step 5: Identify specific levers1 to measurements were collected from all numbers are even created to hide reality
start improvements and connect ongoing projects. There weren’t repri- and attract attention to what looks good.
them to ROI planning mands for insufficient performance, but Be aware that measurements are some-
This is typically best done when using a it was carefully analyzed. We found with times abused to obscure and confuse
process improvement framework such as this client that too many requirements reality.
CMMI [6]. This framework provides the changes lacked a clear specification, The primary question with software
necessary guidance regarding which best analysis, and commitment by the product measurement is not “What measure-
practices to apply and how processes manager. Consequently, a strong focus ments should I use?” but rather “What
relate to each other. Without the right was given towards change management do I need to improve?” It is not about
levers, chances are high that objectives and change review boards2. A weekly pro- having many numbers but rather about
will not be reached. Example: Focus was ject review was introduced after a few having access to the exact information
on requirements development, require- weeks, enhanced with daily Scrum meet- needed to understand, manage, and
ments management, technical solutions, ings of development teams. Require- improve your business. This holds true
project planning, and project monitoring ments changes passing the change review for both project and process measure-
and control. Project managers were edu- board had to have their proper business ment.
cated in project management techniques case or were not accepted. Although the As a professional in today’s fast-paced
and negotiation skills. results proved valid, marketing and sales and ever-changing business environment,
were unhappy because they wanted flexi- you need to understand how to manage
Step 6: Perform a brief gap analysis bility and no accountability for the projects on the basis of measurements
of the selected process areas to changes. This improvement reduced and forecasts. You need to know which
identify strengths and weaknesses requirements changes (within the first measurements are important and how to
This systematic look at weaknesses helps three months) substantially. The first few use them effectively. Here are some suc-
to focus limited engineering resources projects had 20 percent schedule overrun cess factors to pragmatically use measure-
where it matters most. Example: (compared to the previous average of 45 ments:
Requirements were collected rather than percent) and two of them were even • Estimate project time and effort and
developed; requirements management close to 10 percent. These two were fur- realistically set deadlines.
satisfied the basic need for change man- ther evaluated to identify best practices. • Check feasibility on the basis of given
agement; engineering was too technolo- As it turned out, the project managers requirements, needs, and past project
gy-driven and was extended to capture demanded requirements reviews by prod- performance (productivity, quality,
business reasoning; project planning uct managers and testers before accept- schedule, and so on). Do not routine-
showed severe weaknesses in estimation ing requirements to change review ly overcommit.
and feasibility analysis; and project moni- • Manage your requirements and keep
boards. The quality of requirements sub-
toring and control showed weaknesses in track of changes. Measure require-
stantially improved. This change was
getting stakeholder agreement on ments changes and set thresholds of
immediately pushed forward by senior
changes. what is allowed.
management for all projects. As it turns
• Know what value means to your client
out, testers were unhappy because they
Step 7: Develop a concrete action or customer. Track the earned value
didn’t have the time scheduled for doing
plan for the identified weaknesses of your project.
Avoid trying to change all weaknesses at the reviews. Rather than demanding over- • Understand what is causing delays
the same time. Use increments to subdi- time, senior management asked for five and defects. Do not let delays accu-
vide bigger changes. Consider available percent slots (two hours) each week; this mulate. Remove the root causes and
resources and skills and get external sup- proved to be sufficient time to review do not simply treat the symptoms.
port if you lack competencies, such as one to three requirements with the neces- • Be decisive and communicate with a
change management. Example: The most sary depth. fact-based approach. Avoid disputes
urgent need was project planning. A ded- with management, clients, and users.
icated one-month initiative was launched Getting More From Your • Deliver what matters. Use measure-
right away to install a suitable estimation Measurements ments to keep commitments.
method and to train people on it. A tool Measurement programs mostly fail • Continuously improve by analyzing
for feasibility analysis was introduced in because they are disconnected from actu- your measurements and then imple-
parallel because not much of the organi- al business. Too often, things are mea- menting respective objective-driven
zation’s own historical data was available. sured that do not matter at all and there changes. Follow through until results
A historical database was installed for a is usually no clear improvement objective are delivered.
set of key project measurements. In a behind what is measured. Often the col- Practitioners should help manage-
second phase, earned value analysis was lected measurements and resulting ment so that they make decisions on the
introduced. After three months, require- reports are useless, only creating addi- basis of measurements. This will give
ments development was launched under tional overhead. In the worst cases, they management the necessary information
26 CrossTalk The Journal of Defense Software Engineering July/August 2009

6. 942134_Text:Aug2004.qxd 6/12/09 3:21 PM Page 27
Project and Process Measurement
before and after the decision so that they “Embedded Software: Facts, Figures
can follow the effects and compare them and Future.” IEEE Computer 42.4: About the Author
to the goals. Managers should ensure that 42-52, Apr. 2009.
decisions are made based on facts and Christof Ebert, Ph.D.,
7. Chrissis, Mary Beth, Mike Konrad, and is managing director and
analyses. They should always consider
Sandy Shrum. CMMI: Guidelines for partner at Vector Con-
number 4 in the E-4 process (executing
decisions and actions), and make sure Process Integration and Product sulting Services. He is
that decisions move the organization Improvement. 2nd ed. Boston: helping clients worldwide
towards agreed-upon objectives. Addison-Wesley Professional, 2006. to improve technical
Accountability means setting realistic 8. Gibson, Diane L., Dennis R. Gold- product development and to manage
and measurable objectives. Objectives enson, and Keith Kost. “Performance organizational changes. Prior to working
such as reduce errors or improve quality by 50 Results of CMMI-Based Process
percent are not useful. The objective at Vector, he held engineering and man-
Improvement.” Technical Report agement positions for more than a
should be clear and tangible: Reduce the CMU/SEI-2006-TR-004. Aug. 2006
number of late projects by 50 percent for this decade in telecommunication, IT, aero-
year compared to the previous year. These <www.sei.cmu.edu/pub/documents/ space, and transportation. A measure-
objectives must end up in a manager’s key 06.reports/pdf/06tr004.pdf>. ment practitioner who has worked for
performance indicators. Projects using Fortune 500 companies, Ebert authored
unclear oral, memo, or slide reports will Notes “Software Measurement,” published by
most certainly fail to deliver according to 1. “Levers” in this case mean to set up Springer, now in its third fully revised
commitments. Projects using a standard the improvement project in a way that
spreadsheet-based progress reporting— edition.
the different changes follow some
related to requirements, test cases, and order, won’t come ad-hoc and isolated,
defects versus plans—will immediately Vector Consulting Services
and would thus meet objectives.
receive the necessary attention if they Ingersheimer Straße 24
deviate. With this attention, corrective 2. Change review boards are staffed with D-70499 Stuttgart
action will follow, and the project has a engineering and product managers and Germany
good chance to recuperate because of ensure that both technical and busi- Phone: +49-711-80670-0
sufficiently early timing. ness rationale and impacts of changes E-mail: christof.ebert@
Goals that are measured will be are considered. They then make vector-consulting.de
achieved. The reason is very simple: informed and firm decisions.
Once you set a SMART objective and fol-
low it up, you make a commitment to
both yourself and your team or manage-
ment group. As humans—and specifical-
ly as engineers—we want to deliver
results. Measurement provides the stimu-
lus and direction to reach our goals.N
References
1. Ebert, Christof, and Reiner Dumke.
Software Measurement. New York/
Heidelberg: Springer-Verlag, 2007.
2. Juran, Joseph M., and A. Blanton
Godfrey. Juran’s Quality Handbook.
New York: McGraw-Hill Professional,
2000.
3. ISO/IEC. ISO/IEC 15939:2002. Soft-
ware Engineering – Software Measure-
ment Process. 2002.
4. Kraft, Theresa A. “Systematic and
Holistic IT Project Management Ap-
proach for Commercial Software With
Case Studies.” Information System
Education Journal 63.6: 1-15. 23 Dec.
2008 <http://isedj.org/6/63/>.
5. Carleton, Anita D., et al. “Software
Measurement for DoD Systems:
Recommendations for Initial Core
Measures.” Technical Report CMU/
SEI-92-TR-19. Sept. 1992 <www.sei.
cmu.edu/pub/documents/92.reports
/pdf/tr19.92.pdf>.
6. Ebert, Christof, and Capers Jones.
July/August 2009 www.stsc.hill.af.mil 27