Presentation for the European SEPG conference in 2002.

1. Avoiding measurement programme
pitfalls
aas
Robert van Lieshout
Alert Automation Services
Pink Roccade

2. Introduction
aas
! Robert
van Lieshout - robert.van.lieshout@aas.nl
– SPI consultant & trainer
! Alert
Automation Services - www.aas.nl
– Pink Roccade competence center
– Product, Process & Organization
– Consultancy, training and project services
! Pink
Roccade - www.pinkroccade.nl
– Dutch top 3 ICT company, 7700+ employees
© Alert Automation Services
2

3. Contents
aas
! Once
upon a time
! Snakes and Ladders
! The ultimate measurement programme pit
! Summary
! Questions
© Alert Automation Services
3

4. Once upon a time
aas
Special Products International Ltd.
Develops complex electronic gadgets with
embedded software.
Software department has started a measurement
programme.
© Alert Automation Services
4

5. Snakes and Ladders
aas
! We
will examine some common pitfalls and
ways to avoid them
! There
are many more pits on the way to
successful software measurement, but they have
less dangerous snakes
© Alert Automation Services
5

6. The start of a programme
aas
! Select
a pilot project
! Get commitment
– from the project team
– from management
! Communicate
© Alert Automation Services
6

7. First attempt
aas
Measures:
! planned and actual effort
! planned and actual duration
Result:
! a difference of 15% between planned and actual
data
© Alert Automation Services
7

8. Why measure?
aas
! Measurement
goals relate to business goals
! Measurement programme answers questions
Goal 1
Goal 2
Hypotheses
Q1
Source: G/Q/M Method
M1
Q2
M2
Q3
M3
M4
M5
© Alert Automation Services
Q4
M6
M7
8

9. Second attempt
aas
Goal:
! Improve the accuracy of estimations of effort
and duration to within a 10% margin.
Questions:
! What is the average estimation accuracy?
Result:
! Estimation accuracy unchanged
© Alert Automation Services
9

10. What is realistic?
aas
Improve
Control
Understand
Source: G/Q/M Method
© Alert Automation Services
10

11. Third attempt
aas
Goal:
! Understand what influences the accuracy of
estimations of effort and duration
Hypothesis:
! Estimation method and staff experience are the
key issues
© Alert Automation Services
11

12. Third attempt
aas
Measures:
! Estimation accuracy, estimation method, staff
experience
Result:
! Inconclusive data
© Alert Automation Services
12

13. Which measures?
aas
! Easy
to collect
! Provide answers to the questions
! Give good life cycle coverage
! Best
practices are available:
– Practical Software & Systems Measurement
– Putnam & Myers (QSM)
© Alert Automation Services
13

14. Fourth attempt
Goal:
! Understand what influences the accuracy of
estimations of effort and duration
Measures:
! Planned & Actual effort, duration, size, defects
Result:
! Disagreement about defect measurement
© Alert Automation Services
aas
14

15. What does it mean?
aas
! Definition
must be unambiguous
! Describe the what, when & how
Source: CMU/SEI -92-TR-19
© Alert Automation Services
15

16. How to present results?
! Use
indicators
! Be consistent, simple and clear
! One measure by itself is not the whole story
Gantt Chart
S
S
24 7
24
Aggregate Staffing Rate
S
S
7
24 7
24
Total Cum Effort
S
S
7
50
Maint
Jan
'96
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
*
Jan
'96
Total Defect Rate
S
S
24 7
24
S
S
27
Jul
0
*
Jan
'96
9
Jan
'97
15
Jul
21
Jan
'98
3
Jul
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
27
Jul
Current Plan
Actual
Interpolated
S = Start, 2 = DDES, 4 = CUT, 7 = DEL
0
*
S
S
24 7
24
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
0
*
24 7
24
7
3
Jul
9
Jan
'97
15
Jul
21
Jan
'98
27
Jul
0
*
Date 12/6/97 (19.18 mos)
7
Green Control Bound
21
Jan
'98
40
Jan
'96
60
20
Jan
'96
15
Jul
80
0
*
40
Current Forecast
9
Jan
'97
Total MTTD
S
S
7
27
Jul
0
*
$ (millions)
600
200
3
Jul
24 7
24
Total Cum Cost
7
400
Jan
'96
1000
Jan
'96
1000
ESLOC (thousands)
24 7
24
2000
0
*
2000
Size
S
S
27
Jul
Days
Jan
'96
21
Jan
'98
21
Jan
'98
7
Defects
100
15
Jul
15
Jul
Defects
200
9
Jan
'97
9
Jan
'97
Total Cum Normalized Defects
7
300
3
Jul
3
Jul
24 7
24
PM
100
People
150
MB
aas
Elapsed Months
Agg. Staff
Total Cum Effort (PM)
Total Defect Rate
Total Cum Normal Defects
Total MTTD (Days)
Size (ESLOC(K))
Total Cum Cost ($ M)
PI
Yellow Control Bound
© Alert Automation Services
Plan
18.52
24.17
1338.99
4
1622
5.06
398.54
30
18.6
Actual/
Forecast
18.52
81.45
1746.07
9
1044
2.53
443.40
39
16.8
%Diff
0.0
237.0
30.4
96.4
-35.6
-50.0
11.3
30.4
-9.7
Source: QSM
Life Cycle includes MB, Maint
16

17. The ultimate pit
aas
! ...
is sitting around for 6 months trying to think
up the perfect measurement programme
! Take
a little time to avoid the worst mistakes,
but you will make some mistakes, so best get
started and get the mistakes over with
© Alert Automation Services
17

18. Summary
aas
1.Know why you are measuring
2.Know what results you are expecting
3.Set a realistic goal, understand first
4.Choose the right measures
5.Define the measures unambiguously
6.Discuss and analyze the results
7.Get started today
© Alert Automation Services
18

19. References
aas
! The Goal/Question/Metric
Berghout, ISBN 0-07-709553-7
! Practical Software
http://www.psmsc.com/
Method, v.Solingen &
& Systems Measurement,
! Software
Measurement for DoD Systems:
Recommendations for Initial Core Measures,
Carleton et al, CMU/SEI-92-TR-19
! Measures
for Excellence: Reliable Software On
Time Within Budget, Putnam & Myers, ISBN 0-13567694-0
© Alert Automation Services
19

20. Questions?
aas