The document discusses common pitfalls that organizations face when starting a software measurement program. It recommends setting realistic goals focused on understanding rather than control or improvement. Programs should choose measures that directly answer goals and questions, are easy to collect, and provide good lifecycle coverage. Definitions of measures must be unambiguous and results should be presented using clear indicators. The ultimate pitfall is delaying the start of the program while trying to design the perfect measurement approach.

1. Avoiding measurement programme
pitfalls
aas
Robert van Lieshout
Alert Automation Services
Pink Roccade

2. Introduction
aas
! Robert
van Lieshout - robert.van.lieshout@aas.nl
– SPI consultant & trainer
! Alert
Automation Services - www.aas.nl
– Pink Roccade competence center
– Product, Process & Organization
– Consultancy, training and project services
! Pink
Roccade - www.pinkroccade.nl
– Dutch top 3 ICT company, 7700+ employees
© Alert Automation Services
2

3. Contents
aas
! Once
upon a time
! Snakes and Ladders
! The ultimate measurement programme pit
! Summary
! Questions
© Alert Automation Services
3

4. Once upon a time
aas
Special Products International Ltd.
Develops complex electronic gadgets with
embedded software.
Software department has started a measurement
programme.
© Alert Automation Services
4

5. Snakes and Ladders
aas
! We
will examine some common pitfalls and
ways to avoid them
! There
are many more pits on the way to
successful software measurement, but they have
less dangerous snakes
© Alert Automation Services
5

6. The start of a programme
aas
! Select
a pilot project
! Get commitment
– from the project team
– from management
! Communicate
© Alert Automation Services
6

7. First attempt
aas
Measures:
! planned and actual effort
! planned and actual duration
Result:
! a difference of 15% between planned and actual
data
© Alert Automation Services
7

8. Why measure?
aas
! Measurement
goals relate to business goals
! Measurement programme answers questions
Goal 1
Goal 2
Hypotheses
Q1
Source: G/Q/M Method
M1
Q2
M2
Q3
M3
M4
M5
© Alert Automation Services
Q4
M6
M7
8

9. Second attempt
aas
Goal:
! Improve the accuracy of estimations of effort
and duration to within a 10% margin.
Questions:
! What is the average estimation accuracy?
Result:
! Estimation accuracy unchanged
© Alert Automation Services
9

10. What is realistic?
aas
Improve
Control
Understand
Source: G/Q/M Method
© Alert Automation Services
10

11. Third attempt
aas
Goal:
! Understand what influences the accuracy of
estimations of effort and duration
Hypothesis:
! Estimation method and staff experience are the
key issues
© Alert Automation Services
11

12. Third attempt
aas
Measures:
! Estimation accuracy, estimation method, staff
experience
Result:
! Inconclusive data
© Alert Automation Services
12

13. Which measures?
aas
! Easy
to collect
! Provide answers to the questions
! Give good life cycle coverage
! Best
practices are available:
– Practical Software & Systems Measurement
– Putnam & Myers (QSM)
© Alert Automation Services
13

14. Fourth attempt
Goal:
! Understand what influences the accuracy of
estimations of effort and duration
Measures:
! Planned & Actual effort, duration, size, defects
Result:
! Disagreement about defect measurement
© Alert Automation Services
aas
14

15. What does it mean?
aas
! Definition
must be unambiguous
! Describe the what, when & how
Source: CMU/SEI -92-TR-19
© Alert Automation Services
15

16. How to present results?
! Use
indicators
! Be consistent, simple and clear
! One measure by itself is not the whole story
Gantt Chart
S
S
24 7
24
Aggregate Staffing Rate
S
S
7
24 7
24
Total Cum Effort
S
S
7
50
Maint
Jan
'96
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
*
Jan
'96
Total Defect Rate
S
S
24 7
24
S
S
27
Jul
0
*
Jan
'96
9
Jan
'97
15
Jul
21
Jan
'98
3
Jul
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
27
Jul
Current Plan
Actual
Interpolated
S = Start, 2 = DDES, 4 = CUT, 7 = DEL
0
*
S
S
24 7
24
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
0
*
24 7
24
7
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
0
*
Date 12/6/97 (19.18 mos)
7
Green Control Bound
21
Jan
'98
40
Jan
'96
60
20
Jan
'96
15
Jul
80
0
*
40
Current Forecast
9
Jan
'97
Total MTTD
S
S
7
27
Jul
0
*
$ (millions)
600
200
3
Jul
24 7
24
Total Cum Cost
7
400
Jan
'96
1000
Jan
'96
1000
ESLOC (thousands)
24 7
24
2000
0
*
2000
Size
S
S
27
Jul
Days
Jan
'96
21
Jan
'98
21
Jan
'98
7
Defects
100
15
Jul
15
Jul
Defects
200
9
Jan
'97
9
Jan
'97
Total Cum Normalized Defects
7
300
3
Jul
3
Jul
24 7
24
PM
100
People
150
MB
aas
Elapsed Months
Agg. Staff
Total Cum Effort (PM)
Total Defect Rate
Total Cum Normal Defects
Total MTTD (Days)
Size (ESLOC(K))
Total Cum Cost ($ M)
PI
Yellow Control Bound
© Alert Automation Services
Plan
18.52
24.17
1338.99
4
1622
5.06
398.54
30
18.6
Actual/
Forecast
18.52
81.45
1746.07
9
1044
2.53
443.40
39
16.8
%Diff
0.0
237.0
30.4
96.4
-35.6
-50.0
11.3
30.4
-9.7
Source: QSM
Life Cycle includes MB, Maint
16

17. The ultimate pit
aas
! ...
is sitting around for 6 months trying to think
up the perfect measurement programme
! Take
a little time to avoid the worst mistakes,
but you will make some mistakes, so best get
started and get the mistakes over with
© Alert Automation Services
17

18. Summary
aas
1.Know why you are measuring
2.Know what results you are expecting
3.Set a realistic goal, understand first
4.Choose the right measures
5.Define the measures unambiguously
6.Discuss and analyze the results
7.Get started today
© Alert Automation Services
18

19. References
aas
! The Goal/Question/Metric
Berghout, ISBN 0-07-709553-7
! Practical Software
http://www.psmsc.com/
Method, v.Solingen &
& Systems Measurement,
! Software
Measurement for DoD Systems:
Recommendations for Initial Core Measures,
Carleton et al, CMU/SEI-92-TR-19
! Measures
for Excellence: Reliable Software On
Time Within Budget, Putnam & Myers, ISBN 0-13567694-0
© Alert Automation Services
19

20. Questions?
aas