1. Analysis by Competing
Hypotheses
A business tool
from The Psychology of Intelligence Analysis by
Richards J. Heuer, Jr., CIA, 1978-86
adapted for business by John Braren, Jr.
with example
2. Why this presentation?
To put some extra time to good use while
between contracts, I decided to create
several presentations of some tools I believe
could be useful for business.
This is the first presentation.
I have made this as brief as possible in an
attempt to not kill interest, but I have no
doubt some points deserve more
development. Please feel free to contact me
at JBraren@nc.rr.com if you want to discuss
this further, or to offer comments to improve
the presentation and the clarity.
3. BI is probably wrong / FBDM
TMI- Too much information. Fact Based Decision
Making can mean more “volume” and less
“quality”.
Too many biases built in to collection
Programmed BI causes pre-filtering and
predetermined hierarchies and answers
Limits data with collection mechanisms
4. Current Process is troublesome
Select solution and find “proof”
This can yield the wrong answer for all the
right reasons
50:50 chance to get right answer - for all
the wrong reasons
5. Two bad examples
Wrong decision, for right reasons
Buy a copier that is cheap, saves $$ and ink, has small foot
print
But – can‟t print from remote, can‟t queue job for off-
hours, and only holds 100 sheets so need to monitor all
jobs
Right decisions, for wrong reasons
Build a bridge:
1- let‟s make out of steel, it‟s nice and shiny
2- I don‟t like driving over water, so let‟s put at
narrow point of river
3- Make lowest point of bridge at least 76‟ over water;
the mast on my sailboat is 72‟
6. So why does „satisficing‟ and
FBDM persist?
Habit and comfort
◦ More comfortable with failure than change
We are surrounded by the practice of
deciding and then developing CYA support
◦ Rather than stretching to find the most possibilities
and then expending effort to disprove most of them
More lucrative to sell BI tools and code than
a decision making skill
7. What is ACH?
Analysis by comparative hypotheses
Developed for the CIA in 1978 – 1986
Based on:
◦ - finding most possible answers
◦ - applying ALL pro/con data against ALL
hypotheses
◦ - disproving possibilities, not „proving‟
selections
8. The steps 1 – 4 of 8
1. Identify all the possible hypotheses to be
considered.
2. List all significant evidence and arguments for
and against. Combine to one matrix – all
evidence for all hypotheses.
3. Identify the evidence and arguments that were
most diagnostic.
a) All + or all – of no decision making value
4. Refine the matrix definitions as needed-
Hypothesis, Evidence, Original Question.
9. The steps 5 – 8 0f 8
5. Evaluate each hypothesis. Disprove hypotheses
and eliminate, rather than prove them.
6. Find the lynchpin items of evidence. Scrutinize
these.
e) The conflicting + and – decision points
7. Report the conclusions. Discuss the relative
likelihood of all the hypotheses, not just the most
likely one.
8. Identify milestones for future observation, to
monitor and re-evaluate analysis conclusions.
11. Which is best vest to buy?
V1 V2 V3 V4 V5 V6
Color F Orange F Orange F Green F Green Black Tan
Size Small All All All All All
Cost $1 $2 $3 $4 $ .10 $ .05
V5 and V6 might seem best buys, but colors don‟t seem right.
Need to add element of “Visibility”.
12. Which is best vest to buy? - 2
V1 V2 V3 V4 V5 V6
Color F Orng F Orng F Green F Green Black Tan
Size Small All All All All All
Cost $1 $2 $3 $4 $ .10 $ .05
Visibility 2 mile ½ mile ¼ mile 1 mile 1/100 1/100
mile mile
**With Visibility added, we see color isn‟t a decision factor.
**Small only size won‟t work for all our users, so eliminate V1. V5
and V6 are unacceptable distances, so eliminate these.
**V3 is less visible for more money than V2; eliminate V3
**And back to Visibility, what do we need? 1 mile for hunting season
or ½ mile for traffic visibility? This is our Lynchpin data.
13. A Real Example
The worst answer for all the best reasons
This real example works through a brief
version of the steps that went into making
a less valuable decision.
99+% consensus was the first decision was
best.
14. Where do we start a new business
system?
The three options were:
US based established company plant which
produces for largest (80% revenue) customer
Foreign established plant, no large customer
production
Newly acquired US plant, no large customer
production
15. The Evidence for hypotheses - #1
This was the evidence used to make the actual decision. It seems
to lead to an obvious conclusion (“new facility”) based on the
positive data. This was the path that was followed.
ACH forces the search for the lynchpin evidence and guards against
finding support for the obvious, which is too often wrong.
80% facilities Non-80% facilities New facilities, non-
80%
1. Can‟t afford to trouble
80% customer
_ _ __ ++ ++
2. Most of team from 80%
facility, want to avoid stress
n/a _ +
3. Need to learn new
system, so might as well do
n/a n/a ++
once
15
16. The Evidence for hypotheses - #2
With the help of hindsight, I have added the last three elements
of evidence to the matrix. If the search for evidence had been an
active exercise to consider all stake-holders and the complete
„state‟ of the implementation, these elements would have been
discovered and considered from the start.
80% facilities Non-80% facilities New facilities, non-80%
1. Can‟t afford to trouble 80%
customer
____ ++ ++
2. Most of team from 80%
facility, want to avoid stress
n/a _ +
3. Need to learn new system,
so might as well do once
n/a n/a ++
4. Employees feel pain, want
new system
+ + n/a
5. Employees need to keep
some of old, don‟t want new
n/a n/a _
system
6. Unique unfamiliar
measurement system
n/a n/a _
16
17. Evaluating Evidence through #2
1. The first hypothesis of starting with the 80% customer
facility has a huge negative and can be eliminated with
confidence. (Note that significance of evidence mat be
very subjective. If there is any doubt, the hypothesis
probably should be kept in play.)
2. If we look at evidence element 2 (team from 80%
facility), we can see that the evidence might be re-stated
as “Team from outside facility”, in which case it carries
the same negative (or positive) weight for both
remaining facility types.
3. And now elements 5 &6 add two negatives to the „new
facility‟ hypothesis which makes this our next best choice
to cut as an option.
4. But now we want to add a 7th piece of evidence.
17
18. The Evidence for hypothesis - #3
80% facilities Non-80% facilities New facilities, non-80%
1. Can‟t afford to trouble
80% customer
___ ++ ++
2. Most of team from 80%
facility, want to avoid stress
n/a _ _
3. Need to learn new system,
so might as well do once
n/a n/a ++
4. Employees feel pain, want
new system
+ + n/a
5. Employees need to keep
some of old, don‟t want new
n/a n/a __
system
6. Unique unfamiliar
measurement system
n/a n/a _
7. Rationalizations usually
add to scope, but:
Bringing facility into 80% n/a + +
methods will build “corporate
team” perception
18
19. Review each hypothesis
Review validity of each hypothesis; eliminate as
possible
Evaluating Evidence through #3
◦ Evidence elements 2 & 7 have equal values so they can
be eliminated as not useful.
◦ Because we are trying to disprove hypotheses, the two
negatives (elements 5 & 6) for “new facility” become the
only two valuable elements. These are the lynchpins.
19
20. The Conclusion
80% facilities Non-80% facilities New facilities, non-80%
1. Can‟t afford to trouble
80% customer
___ ++ ++
2. Most of team from 80%
facility, want to avoid stress
n/a _ _
3. Need to learn new system,
so might as well do once
n/a n/a ++
4. Employees feel pain, want
new system
+ + n/a
5. Employees need to keep
some of old, don‟t want new
n/a n/a __
system
6. Unique unfamiliar
measurement system
n/a n/a _
7. Rationalizations usually
add to scope, but:
Bringing facility into 80% n/a + +
methods will build corporate
team sense
20
21. Conclusion note - Item 4
80% facilities Non-80% facilities New facilities, non-80%
4. Employees feel + + n/a
pain, want new
system
Note that while this point may seem important, the fact is that it
was a non-starter from the beginning.
** With 2 “reasons for” and an “n/a” it had no dis-prove value.
** And even after eliminating the “80% facility”, it still had no
dis-prove value for the last two choices.
** This is a clear example where bias for a solution can be seen
as contrary to making a best choice decision.
21
22. ACH Steps Review
1. Identify all the possible hypotheses to be considered.
2. List all significant evidence and arguments for and against.
Combine to one matrix – all evidence for all hypotheses.
3. Identify the evidence and arguments that were most
diagnostic.
4. Refine the matrix- Hypothesis, Evidence, Original Question.
5. Evaluate each hypothesis. Disprove hypotheses and
eliminate, rather than prove them.
6. Find the lynchpin items of evidence. Scrutinize these.
7. Report the conclusions. Discuss the relative likelihood of all
the hypotheses, not just the most likely one.
8. Identify milestones for future observation, to monitor and
re-evaluate analysis conclusions.
23. Exercise
As a group, or individually, apply the ACH process to a
decision you might make, or best, to one decision that
worked and one that did not work.
The comparison of the historical decisions might drive
home the value of the ACH approach.
The exercise can be done quickly and still show its
value:
Identify question, refine, and follow through rest of
steps quickly
Best done on flip chart or white board, or an Excel
grid. Whatever works.
24. The Evidence for using ACH
Select and then find ACH Process
support
1. Find most possible
solutions or responses - ++
2. Disqualify options that do
- ++
not work
3. Keep options until they
are disproved; scientific - ++
4. Refine the question, the
evidence, and the solution - ++
throughout the process
5. Avoid bias - ++
24
