IWSM Presentation

1. Putnam’s Effort-Duration Trade-Off Law: Is the
Software Estimation Problem Really Solved?
Han Suelmann
October 7th, 2014

2. Putnam’s study – reference
L.H. Putnam,
“A generic empirical solution to the macro software sizing
and estimating problem,”
IEEE Transactions on Software Engineering,
vol. 4, pages 345 ─ 361,
July 1978.
2

3. 3

4. Agenda
• Putnam’s study: results and influence
• Putnam’s approach
• Intermezzo – A statistical pitfall
• Critical evaluation:
o dataset is very limited
o model and assumptions are unclear
o analysis is incorrect
• Other studies provide no corroboration
• Simulation study demonstrate incorrectness
5

5. 6

6. Putnam’s study: results and influence
Claims:
• Generic empirical equations that describe size – effort –
duration relationships.
• Method will produce accurate estimates.
• Only a few quick reference tables and a pocket calculator
needed.
• Trade-off law: K ~ 1 / T4.
7

7. Putnam’s study is very influential
Influence:
• incorporated in estimation software
• many references
• sometimes cited as authoritative
8

8. Putnam’s approach
1) Gather data on effort (K), duration (T) and size (S).
2) Define difficulty: D = K / T 2
3) Define productivity: P = S / K
4) Find relationship between D and P.
Result: P ~ D -0.67
5) Perform basic algebraic manipulations to find
relationships between S, T, and K.
Result: S = C ∙ K1/3 ∙ T4/3,
3
S
and therefore: .
9
4
T
K 

9. The crucial relationship…
10
K
2 T
D 
S
K
P 
difficulty
productivity

10. Putnam’s approach
1) Gather data on effort (K), duration (T) and size (S).
2) Define difficulty: D = K / T 2
3) Define productivity: P = S / K
4) Find relationship between D and P.
Result: P ~ D -0.67
5) Perform basic algebraic manipulations to find
relationships between S, T, and K.
Result: S = C ∙ K1/3 ∙ T4/3,
3
S
and therefore: .
11
4
T
K 

11. Intermezzo – A statistical pitfall
• Two researchers examine relationship between S and K.
• Both assume linear relationship.
• Researcher 1 writes K = aS + b
• Researcher 2 writes S = a’K + b’
12
S
K
S
K

12. Researcher 2’s does his linear fit

13. Intermezzo – The results are quite different
• Researcher 1 writes K = aS + b and finds
(1) K = 1.01 S −0.02.
• Researcher 2 writes S = a’K + b’ and finds
(2) S = 0.50 K + 3.0.
• Researcher 2 then derives
(3) K = 2.02 S – 6.2.
14

14. Researcher 1: does her fit

15. Intermezzo – A statistical pitfall
• Researcher 1 writes K = aS + b and finds
(1) K = 1.01 S −0.02.
• Researcher 2 writes S = a’K + b’ and finds
(2) S = 0.50 K + 3.0.
•
Researcher 2 then derives
(3) K = 2.02 S – 6.2.
16
^
^
^

16. Linear fit: minimising least squares
S
K

17. Critical evaluation (1) – dataset is very limited
• only 13 projects
• all US Military
• 4 are left out => 9 projects remaining
18

18. Critical evaluation (2) – model is unclear
19
size
duration
effort
 Putnam does not make clear and consistent
choices regarding model structure.
 Only one parameter to capture effort-duration
interaction

19. Critical evaluation (3) – analysis
20

20. Critical evaluation (3) – analysis
21

21. Critical evaluation (3) – analysis
22

22. Critical evaluation (3) – analysis
23

23. Critical evaluation (3) – analysis
24

24. Critical evaluation (4):
Difficulty – Productivity relationship
Putnam’s reasoning:
More precisely notated:
25
2/3 P D



K
S
1/3 4/3 S  K T
ˆ 2/3 P  D
ˆ ˆ 2/3 4/3 ??    S K K T
2/3
2





T
K

25. Other studies − Corroboration by Putnam et al.
Putnam & Putnam, “A data verification of the software fourth
power trade-off law,” (Proc. of the Int. Soc. of Parametric Analysts – 6th Annu.
Conf., vol. III(I), pp. 443–471, 1984.)
Putnam & Myers, “Measures for excellence – Reliable
software on time, within budget”, (Englewood Cliffs: Yourdon, 1992.)
Confirmed that K ~ 1 / T4, but…
Found (Dunsmore et al., 1986) and admitted (Putnam &
Myers, n.d.) to be based on circular reasoning.
26

26. Other studies – No corroboration from Jeffery
Jeffery (1987):
• 47 MIS in 4 large organisations
• Find P as a function of K and T.
Result:
• P ~ K−0.47T −0.05
• essentially no productivity – duration relationship
• comparison with Putnam’s P ~ K−0.67T 1.33
• no confirmation
• strictly speaking: no refutation either
27

27. Other studies – No corroboration from
Barry, Mukhopadhyay, and Slaughter
Barry, Mukhopadhyay, and Slaughter (2002):
Ansatz: ln K = … + β1 T
Result: β1 = 0.000677 ± 0.000103, p = .031.
So – larger duration predicts larger effort.
28

28. Other studies – Team size affects effort, so…?
Putnam & Myers (n.d.): larger team size predicts larger
effort:
Teams of 5 or less have better productivity than teams of 20
or more.
Supported by other studies. Example (Rodríguez et al.):
PDR ~ (average team size)^0.57
But…
• translation to effort-duration trade-off unclear
• interpretation in terms of causation dubious
29

29. Several interpretations are possible…
30
larger
team size
more
effort

30. Simulation (1)
Goal: check whether the analysis issues really lead to
incorrect results.
Method:
• generate simulated data with known structure
• analyze simulated data, following Putnam’s approach
• check whether results are consistent with assumptions
31

31. Simulation (2)
Model assumptions:
• Size, effort, and duration are unrelated random numbers.
• Log-normal distributions.
• 1000 projects.
32

32. Simulation (3) – analysis
33

33. 34
K
2 T
D 
S
K
P 

34. Simulation (4) – result
Fit yields:
ln P  0.67ln Dconstant
After transformation:
After some manipulations (same as Putnam’s):
K Yet, no
35
0.670.02 P  D
1
 
4.1 0.4
T
relationship
actually exists!

35. Simulation (5) – coincidence?
For convenience, write s = ln S, k = ln K, and t = ln T.
Difficulty and productivity:
• ln D = k – 2t
• ln P = s – k
Derive the slope of P against D:
D P
cov(ln , ln )
2


k

(ln | ln ) 2 2
Follow Putnam closely, finding K ~ T u , with

  which yields u = − 4 if  8
36
.
var(ln ) 4
k t
D
B P D
 
 
2
2

u
1
k 2

t
k

t

36. Simulation (6) – result
37
 S K 

37. Conclusions
38
Claims:
• Generic equations that
describe size – effort –
duration relationships.
• Method will produce
accurate estimates.
• Trade-off law: K ~ 1 / T4.
Limited dataset,
no corroboration
Not addressed
Faulty analysis,
no corroboration

38. Conclusion
39
No credibility for
Putnam’s result
Putnam’s
original study
was wrong
No
corroboration

39. The bad news
• Handling statistical
relationships as if exact.
• Interpreting statistical
relationship as causal
relationships without
sufficient support.
40
Both issues are
rather common in
the estimation /
metrics literature.

40. Question time