This document discusses using the Analytic Hierarchy Process (AHP) and Saaty method to choose the best decision alternative. It provides examples of using AHP to estimate software sizes by comparing to a known example, and prioritize system requirements by comparing costs and benefits. Measurement data can support AHP by providing attributes and sizes of past projects to estimate new projects. AHP is useful for ranking choices based on criteria in a relatively short time and detects inconsistencies in rankings.

1. Peter Baxter, DISTRIBUTIVE MANAGEMENT
Hampton Roads INCOSE Decision Analysis Conference - Nov 2009
USING MEASUREMENT AND THE SAATY
METHOD TO CHOOSE THE BEST
DECISION ALTERNATIVE

2. OBJECTIVES
Learn:
 How to construct and weight evaluation
criteria.
 How to make pair-wise comparisons of
alternatives.
 How to expand the example for more
complex, nested types of criteria.
 How a measurement process can support
the Saaty method.
D I S T R I B U T I V E
M A N A G E M E N T
2

3. WHAT IS THE AHP/SAATY
METHOD?

4. SIMPLE DEFINITION
Compute the matrix values for a and … voila!
The Analytic Hierarchy Process (AHP)
D I S T R I B U T I V E
M A N A G E M E N T
4

5. COMPARE THE SIZES
OF THESE CIRCLES
A B C D E
A B C D E
A 1 7 9 5 3
B 1/7 1 3 1/5 1/7
C 1/9 1/3 1 1/7 1/9
D 1/5 5 7 1 1/3
E 1/3 7 9 3 1D I S T R I B U T I V E
M A N A G E M E N T
5

6. SCALE OF COMPARISON
 Don’t need an absolute scale
(if you already know the size of all but one)
 Relative scale approximates difference
D I S T R I B U T I V E
M A N A G E M E N T
6

7. SIMPLE EXPERIMENT
One test is worth 1000 expert opinionsD I S T R I B U T I V E
M A N A G E M E N T
7

8. WEBER’S EXPERIMENT
In 1846 Weber found, for example, that people while holding in
their hand different weights, could distinguish between a
weight of 20 g and a weight of 21 g, but could not if the
second weight is only 20.5 g. On the other hand, while they
could not distinguish between 40 g and 41 g, they could
between 40 g and 42 g, and so on at higher levels. We need
to increase a stimulus s by a minimum amount Δs to reach a
point where our senses can first discriminate between s and
s+Δs. Δs is called the just noticeable difference (jnd). The
ratio r = Δs/s does not depend on s.
Weber’s law states that change in sensation is noticed when the stimulus
is increased by a constant percentage of the stimulus itself. This law
holds in ranges where Δs is small when compared with s, and hence
in practice it fails to hold when s is either too small or too large.
D I S T R I B U T I V E
M A N A G E M E N T
8

9. WHY IT WORKS
 People are inconsistent at providing an absolute
scale to evaluate objects.
How much does Rock A weigh?
How much does Rock B weigh?
 There are better at comparing pairs of objects.
Does Rock A weigh more than Rock B?
D I S T R I B U T I V E
M A N A G E M E N T
9

10. APPLYING THE TECHNIQUE
1. Select criteria(s) to evaluate
2. Define comparison scale
3. Perform pair-wise comparison
4. Check consistency
5. Calculate values
D I S T R I B U T I V E
M A N A G E M E N T
10

11. APPLYING THE METHOD

12. EXAMPLE #1
SOFTWARE ESTIMATION
 Technique: Compare sizes of 7 unknowns to one known
 A controlled test where existing software sizes are known.
 Ask 30 grad students to estimate the SLOC of common data
structures like stack, queue, list.
 Estimate SLOC three ways:
1. Guess a number.
2. Compare to one reference structure using numeric
scale.
3. Compare to one reference structure using relative scale.
From “Establishing Software Size Using Pair-wise Comparison Method”
by Eduardo Miranda
D I S T R I B U T I V E
M A N A G E M E N T
12

13. EXAMPLE #1
SOFTWARE ESTIMATION
D I S T R I B U T I V E
M A N A G E M E N T
13

14. EXAMPLE #1
OBSERVATIONS
Miranda’s Observations
 The high variability of the “finger in the wind approach”,
which is almost two to three times bigger than the
corresponding paired comparisons method.
 The high correlation, r = .979, existing between the
relative sizes of modules independent of the estimation
method employed. This seems to corroborate the
premise that the human mind is better at establishing
differences than at guessing absolute values.
D I S T R I B U T I V E
M A N A G E M E N T
14

15. EXAMPLE #2
SYSTEM REQUIREMENTS
 Technique: Compare cost and value of 9 requirements to each
other
 Given nine security requirements, decide which one(s) have
greatest cost-benefit.
 Benefits
“By using AHP, the requirements engineer can
also confirm the consistency of the result. AHP can
prevent subjective judgment errors and increase the
likelihood that the results are reliable.”
From an SEI study “Requirements Prioritization Case Study Using
AHP” by Nancy Mead.
D I S T R I B U T I V E
M A N A G E M E N T
15

16. EXAMPLE #2
APPROACH
Approach
1. Review requirements for completeness.
2. Apply pair-wise comparison for value.
3. Apply pair-wise comparison for implementation
cost.
4. Calculate AHP matrix and diagram for value & cost.
5. Use resulting diagram for analyzing requirements.
D I S T R I B U T I V E
M A N A G E M E N T
16

17. EXAMPLE #2
COMPARISON SCALE
D I S T R I B U T I V E
M A N A G E M E N T
17

18. EXAMPLE #2
COMPARISON
Compare one attribute of each requirement to another
requirement using a relative scale.
#1 is same
as #1
#1 is greater
than #2
#2 is much
less than #8
D I S T R I B U T I V E
M A N A G E M E N T
18

19. ABOUT CONSISTENCY
 If: A > B > C
 Then: C > A is wrong
AHP contains a technique to calculate the extent of
pair-wise consistency, which can then be
compared to a consistency tolerance.
AHP can also indicate which pair-wise comparison
(like the one above) is inconsistent.
D I S T R I B U T I V E
M A N A G E M E N T
19

20. EXAMPLE #2
COMPARISON SUMMARY
D I S T R I B U T I V E
M A N A G E M E N T
20

21. EXAMPLE #2
FINDINGS
Client feedback
 It may be beneficial to see the consistency matrix.
 Understand weight of cost and value.
 Difficult to understand the motivation of each
reviewer.
D I S T R I B U T I V E
M A N A G E M E N T
21

22. MEASUREMENT AND AHP

23. MEASUREMENT AND AHP
Goal is for Measurement to support AHP estimation
Estimate
Plan
Monitor
& Control
Capitalize
Measurement Process
Use AHP
D I S T R I B U T I V E
M A N A G E M E N T
23

24. TYPICAL MEASUREMENT
PROCESS
Evaluate
Establish
Capability
Technical
and Management
Processes
INFORMATION
NEEDS
ANALYSIS RESULTS
ANALYSIS
RESULTS AND
PERFORMANCE
MEASURESIMPROVEMENT
ACTIONS
Measurement Process
USER FEEDBACK
Plan Perform
Core Measurement Process
Experience
Base
MEAS-
UREMENT
PLAN
24

25. WHAT TO MEASURE?
Measurement “Requirements” = information needs
 Based on your business needs.
(not a pre-defined list)
 Information needs are:
 “Input” to the measurement process.
 Provided by the management and technical process that
need information to perform their jobs.
 Become the requirements for measurement process.
 Refined into measures and then resulting information
products are provided to the “users”.
D I S T R I B U T I V E
M A N A G E M E N T
25

26. COMMON PROJECT
MEASURES
 Cost
 Staffing , Staff Hours
 Functional / Requirements Size
 SLOC
 Defects
Estimation Project/Program Store
Factors estimated
using AHP
D I S T R I B U T I V E
M A N A G E M E N T
26
Usually there is opportunity for measurement to support AHP since
there is overlap between what is estimated and what is measured

27. MEASUREMENT GUIDE AHP
 Measurement contains project attributes:
 Lifecycle model
 Principle architecture
 Application domain
 AHP estimation of projects with similar attributes.
 Reference sizes should not differ by more than an
order of magnitude.
D I S T R I B U T I V E
M A N A G E M E N T
27

28. EXAMPLE #3
 Technique: Estimate staff of a new project based
on 5 known (completed) projects
 Estimate of total staff hours in person years
 Uses data available in a measurement process
from five completed projects
 Projects were selected because they share
similar attributes to project being estimated.
D I S T R I B U T I V E
M A N A G E M E N T
28

29. EXAMPLE #3
D I S T R I B U T I V E
M A N A G E M E N T
29

30. OTHER EXAMPLES
From Saaty:
 Evaluate the best city in China for Disney
to build a new theme park.
 Determine optimum foreign relations
policy for dealing with Iran.
 Estimate market share of “super” retail
stores.
 Selecting a school.
D I S T R I B U T I V E
M A N A G E M E N T
30

31. SUMMARY
 AHP benefits:
 Because it takes a relatively small amount of
time, it is very economical for ranking choices
based on your criteria
 Detects inconsistency in the ranking, allowing
you to understand and address it
 Integrates well with a measurement process
D I S T R I B U T I V E
M A N A G E M E N T
31

32. RESOURCES
“Requirements Prioritization Case Study Using AHP”
by Nancy Mead Software Engineering Institute
“Establishing Software Size Using the Paired
Comparisons Method” by Eduardo Miranda
“Relative Measurement and Its Generalization in
Decision Making Why Pairwise Comparisons are
Central in Mathematics for the Measurement of
Intangible Factors The Analytic
Hierarchy/Network Process” by Thomas Saaty
D I S T R I B U T I V E
M A N A G E M E N T
32

33. QUESTIONS
D I S T R I B U T I V E
M A N A G E M E N T
33

34. Peter Baxter
Distributive Management
www.distributive.com
pbaxter@distributive.com
D I S T R I B U T I V E
M A N A G E M E N T
34