How Four Statistical Rules Forecast Who Wins a Competitive Bid

The Intelligence Collaborative
http://IntelCollab.com #IntelCollab
How Four Statistical Rules
Forecast Who Wins a
Competitive Bid
Powered by
A Complimentary Webinar from Aurora WDC
12:00 Noon Eastern /// Wednesday 22 October 2014
~ featuring ~
Dr. Bill Zangwill Arik Johnson

Powered by
Bill Zangwill
Bill Zangwill is a Professor, Emeritus, from the University of Chicago,
Booth School of Business. He has authored four published books, one
of which was selected by the Library Journal as “One of the Best
Business Books of the Year,” and had over 50 papers in academic
journals. In addition, he has had three articles published in the Wall
Street Journal. His consulting engagements include top firms such as
IBM, AT&T, Motorola, many smaller firms and the US government. He
has also taught at the University of Illinois and the University of
California, Berkeley. He is considered one of the most innovative
thinkers in his field.
Email: willard.zangwill@chicagobooth.edu
The Intelligence Collaborative is the online learning and networking
community powered by Aurora WDC, our clients, partners and other friends
and dedicated to exploring how to apply intelligence methods to solve real-world
business problems.
Apply for a free 30-day trial membership at http://IntelCollab.com or learn
more about Aurora at http://AuroraWDC.com – see you next time!

Questions, Commentary & Content
α Use the Questions pane on your
GoToWebinar control panel and all questions
will be answered in the second half of the
hour.
α You are welcome to tweet any comments on
Twitter where we are monitoring the
hashtag #IntelCollab or eavesdrop via
http://tweetchat.com/room/IntelCollab
α Slides will be available after the webinar for
embedding and sharing via
http://slideshare.net/IntelCollab
α To view the recording and download the PPT
file, please register for a trial membership at
http://IntelCollab.com.
Powered by

Powered by
Agenda
► Introduction
► Risk, Bias, Errors: Caused by Usual
Procedures, but Usually Missed
► Application to Situations in Business
► Summary, Q&A and Discussion

Powered by
The Joy When We Win.

How do we avoid the grief and attain the joy?
Powered by

Novel Approaches. Two patents, third pending.
Powered by

Why are the patents and new approaches
needed?
Because usual methods typically have hidden bias, hidden risks
and hidden errors.
► MAJOR CONUNDRUM: We believe in the usual approaches
and keep using them because the errors are hidden so are
missed and not considered.
► E.g., Behavioral research. If we believe the decision will be 70-80%
Powered by
successful, likely it will be roughly 50%.
► First methodology that expressly helps warn of errors, bias,
issues missed
► Humans miss issues.
► 2 patents and third pending
► Follow the four rules

Powered by
Four Rules – Boost Win Rate
1. Stop using usual methodologies as they
likely have hidden risk, bias, errors.
They can cause losses despite our belief in them.
2. Do not assume. TEST.
3. Improve twice.
Best ideas come after reviewing situation a couple times.
4. Follow procedure, otherwise bias and risks
will slip in secretly, stealthily and
surreptitiously.

Powered by
Risk, Bias, Errors: Caused by Usual
Procedures, but Usually Missed
Math. Bias. Surprises.

Powered by
Does Arithmetic Have a Hidden Secret?
Example: SUBTRACTION
100 $100,000,000
-90 -$90,000,000
----- ---------------
10 $10,000,000
WHAT CAN POSSIBLY BE HIDDEN HERE? WHAT MIGHT BE WRONG?

In Business Decisions, Most Data Are Not
Known That Perfectly
Powered by
Example: SUBTRACTION ERROR
100
-90
-----
10
But error in value
100 +/_ 10 (90 – 110) ( 10% error)
-90
------
0 to 20 (100% error)
HUGE ERROR INCREASE FROM 10% to 100%
WORSE IF BOTH FACTORS HAVE ERRORS OR OVER TIME
Math causes error.
TYPICAL ALGORITHM: WE BELIEVE IT IS RIGHT YET INSTEAD PRODUCES
ERROR

Powered by
DCF and Other Financial Formula Can
Increase Error
Gordon growth model:
$10 million payout in one year, g = growth rate, r= rate of return
Value, DCF = $10/ (r – g) (in millions)
Example 1: r = 12% , g = 10%
Value = $10/ ( 0.12 - 0.10) = $10/(0.02) = $500 million
Example 2: r = 12% , g = 11% growth rate increased by 10%
Value = $10/ ( 0.12 - 0.11) = $10/(0.01) = $1 billion
10% error in input
100% error in answer.
Proprietary material. Do not disseminate. Patent pending and copyrights University of Chicago and Decision Command, Inc. SLAM(TM) is a registered
service mark of Decision Command.

Powered by
More Accurate Using Better Math
Two quantities, 40 and 10
What percentage is 40 of total?
40/( 40 + 10) = 40/50 80%
Suppose 25% error in 40. Could be 30 – 50.
What is percentage of the total now?
30/(30 +10) 75%
50/(50 + 10) 83.66%
Much reduced error relative to 80% base --– error less than 7%
ERROR REDUCED FROM 25% TO LESS THAN 7%.
THE MATH IN MOST ALGORITHMS IN COMMON USE CANNOT BE
TRUSTED.
Must use better approaches that reduce error and attack the hidden secret.

Powered by
Improves Accuracy, Reduces Risk
Pij = conditional probability in a cell, Pj = probability of column


(Uses Bayes and Laplace approaches)
Suppose there is risk in a specific cell entry Plm
Consider risk (percent error) in the output Pm over risk in the
input Plm.
The fractional risk in output is
The fractional risk in the input is
i ij
j
i ij
j
P
P
P


m
lm
m
dP
dP
P
1
P
lm
Proprietary Information, FOUO


Powered by
Error Percentage Always Decreases,
Proprietary Information, FOUO
    
i ij i l im i im i l im
j
m
m 2 lm
i ij
m lm j
lm lm
m i im
lm
i ij
j
i ij i l im i im i l im i ij i im i im i im
j j
lm
i im i im i ij i im
j j
i ij i im
j
i ij
( P ) P P P
dP [
dP dP ( P )
P dP
[ ](P ) ](P )
1 P P
P
P
( P ) P P P ( P ) P P P
(P )
( P ) P ( P ) P
( P ) P
( P
 
 

 


         
 
   
  





 
 

j
i im
i ij
j
)
P
1 1
( P )


  


Percentage error out less
than percentage error in

Serious Problems Often Cause Us to Lose: Most
Algorithms Increase Risk and Errors, and We Do Not
Know It
Bayesian Algorithm I Use Reduces Risk and Errors
1. “The model was previously shown to be surprisingly robust to obvious violations of
this independence assumption, yielding accurate classification models
even when there are clear conditional dependencies.”
 “Many researchers have noted the good performance of SBC, including Clark
and Niblett, Knononenko, Langley and Sage, and Domingos and Pazzani.”
2. Beyond Independence: ..is commonly thought to assume that attributes are independent
given the class, but this is apparently contradicted by the surprisingly good
performance it exhibits in many domains that contain clear attribute
dependences. No explanation for this has been proposed so far. In this paper we show that the
SBC does not in fact assume attribute independence, and can be optimal even
when this assumption is violated by a wide margin
Powered by
Proprietary. Confidential information. US Patent 7,676.446. Copyright University of Chicago and/or Decision Command.

INSIGHT: Statistically identify entries that are odd relative to facts of situation, outliers
Powered by
BIAS – Frequent Cause of Not Winning
Bias and optical illusions are similar. What you
believe is not there.
Holding hands, or not?
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO

Powered by
Bias and Hidden Risk
Biases, hidden risk, faulty assumptions tend to be at variance, out-of-kilter
with facts of situation.
Anomaly, outlier or oddity is often the clue: risk, bias, faulty
assumption
outlier

Surprise – Must the Unexpected Always be
Unexpected?
INSIGHT: Identify contradictory and odd information. The greater the level of
contradictory and odd information, the greater chance something odd is going on,
something was missed.
Powered by
MATH ADVANCEMENT– Add variable to represent surprise

Bayes System
First System to Warn of Risks/Surprises/Missed Issues
Powered by

Usual Methods Typically Have Hidden Error,
Risk and Bias, and Often We Will Not Know It.
► NO AUTOMATIC IDENTIFICATION OF RISK.
► NO AUTOMATIC MEANS TO PREVENT MATH ERRORS.
► NO AUTOMATIC CROSS-CHECKS ON ACCURACY.
► NO AUTOMATIC ANALYSIS OF RISK OR SURPRISE/MISSED
Powered by
ISSUES.
► NO AUTOMATIC ESTIMATE OF THE PROBABILITY OF SUCCESS.
► NO PROCESS TO RESOLVE DISAGREEMENTS
NEED BETTER WARNING OF THE HIDDEN PROBLEMS
“YOU CAN’T DO A GOOD JOB WITH POOR TOOLS”

TEST AT UNIVERSITY OF CHICAGO
Managers and executives made decisions
They compared quality of decision made this new way versus their
usual decision approach.
40% better and 20% faster
Powered by
Proof

Powered by
1. Stop using usual methodologies as they likely have
hidden risk, bias, errors.
3. Improve twice.
4. Follow procedure, otherwise bias and risks will slither in
secretly, stealthily and surreptitiously.

Powered by
Application to Situations in Business
Use better ways to handle math, bias, risks

INVEST IN EXCELLENT CHILEAN FIRM.
PARTNER.
In private education, growing rapidly, wanted funds for
international growth and acquisition
AAI could assist with funding growth. Gateway to future.
Several trips to Chile to speak with owner
AAI BIDDING AGAINST ARCH RIVAL ADVENT
Felt confident they would win—
got to know owner, understood situation
Powered by
Bidding to Invest in a Chilean Firm

AAI WERE HIGHLY EXPERT AND DID TRADITIONAL
ANALYSIS WELL---SURE THEY WOULD WIN :
analyzed situation, obtained information, visited Chile,
brought in experts, financial analysis, discussions.
RESISTED USE OF SYSTEMATIC METHOD,
SUPERFLUOUS.
But agreed to use it because easy, fast, and I was a friend.
Powered by
Situation: AAI Confident of Winning

Powered by
Difficulties with Usual Methods
Made decision as if they were making the decision.
No goals from perspective of Santo Tomas.
•No metrics for those goals. Prob. ST will select AAI or Advent.
No independent warning of possible biases, risks.
•No probability of missed issues/surprise.
No iterations- building of better and better solns.
(First analysis is always incomplete---Donnelly’s rule)
•No metrics to determine if decision is getting better.
No testing if decision excellent or not.
arguments, blind alleys.

5 Minute Role Play Real Decision Maker
MUST GET AT EMOTIONS. WHAT ARE THEY CONCERNED OR WORRIED ABOUT? Utilize
goals, hopes, fears, problems of real decision maker. What could cause failure? What causes
customer worries?
Santo Tomas goals.
Powered by

We Fail (Missed before because traditional had
no means to detect.)
EXPOSED HUGE RISK. ASTOUNDED. BUT THEY HAD TO AGREE BECAUSE THEY DID THE
Powered by
EVALUATION.
WHAT SHOULD WE DO?

- Test: What are the risks that cause our loss?
Why we lose
Powered by
TRADITIONAL NO WARNING OF RISKS

Powered by
Traditional Methods: Argue
►BEFORE: Argue who is right.
►SYSTEMATIC APPROACH: TEST.
Identify risks. Examine possible
improvements and check their metrics

New approach to win. BOOSTED METRICS:
success rate from 63% to 75% beating Advent’s 70%
REVISED AAI STRATEGY
Powered by
METRICS
AAI 63-75
ADVENT 70

Test—expose risks again. Still some issues, so
build higher – IMPROVE TWICE
Powered by

Build the win higher—test and improve twice
Powered by

► Bring in experts on international to our team and show Santo
Tomas, that we are really best for global expertise. Create
connection in Chile to build trust.
► Convince Santo Tomas our financial deal is really better. (less
upfront, but more later)
► Ensure CEO comfortable (CEO is owner)
Conclusion: saved them from losing deal
Traditional means fail due to hidden errors, no warning.
Powered by
Develop New Strategy - Won

New Product Development at EG Insurance.
What is going on?
Large commercial insurance firm wants to sell to
mortgage originator insurance market
Traditional product based upon annual premium—
Exciting new product they expect to launch to clearly
differentiate themselves from competitors: Chubb,
CNA, Lloyds of London:
A transaction-based plan based upon monthly
volume. Client would only pay for what was actually
used.
Powered by
WE’RE READY TO GO AHEAD.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013

Powered by
Role Play: What does customer think?
Get at worries.

Test: Complete matrix, examine metrics (cannot be done in
traditional)
New product loses. Software saved them from failure!
Forced analysis from viewpoint of actual decision maker.
Powered by
METRICS:
Prob. Success
Low, 16%, 53%.
Easy to miss
this.
WAY UNDER
GOAL OF 75%

Warning. METRIC High surprise risk – “You can
do better than this, you missed something”
Powered by
METRICS:
Success, 16%, 53%
Surprise 31%

Examine Hidden Risks
-- Countering risks leads to new concept.
Powered by
risks

NEW IDEA: Transaction based, but on closed
loans only. Should save money.
►TEST IDEA: DO NOT ASSUME: (cannot
be done with traditional because no
metrics)
Powered by

Test New idea: Transaction based, but on
closed loans only– Surprise still high
METRICS:
Prob. of success is
lower.
Surprise drops to
20% from 31%
What to do now?
Build
Powered by

Powered by
Test: Check Risks
risks

Powered by
Check More Risks and Weaknesses

Powered by
TEST another idea — Large increase in
estimated success probability metric
METRICS:
Success 71%
Surprise 16%
CLOSE ENOUGH TO 75%

TWO IMPROVEMENT CYCLES
Sizable improvement in probability of winning over
prior approach.
Value
ZERO
Powered by
Closed loans only($54)
Initial-New transaction idea($31)
Cust. choice($74)
Traditional

Powered by
Saved Them From Losing Money.
Usual Means Have Hidden Error
$4.3 MILLION SAVED.
Over 30% improvement . Three cycles.
Exposed that initial evaluation was incorrect
“ H a v i n g s o f twa r e l i k e t h i s a r o u n d
to aid in making important
d e c i s i o n s i s wo n d e r f u l . ”
“ 5 0% f a s t e r t h a n r e g u l a r me t h o d . ”

Powered by
IT Competitive Bid
WE LOSE
Prob. of
success

Test: Despite ABC Winning, Maybe ABC Can
Be Beaten
Powered by
Factors where XYZ beats ABC
where ABC beats XYZ

Powered by
Where are others strong or weak?
XYZ STRENGTHS SMS STRENGTHS
ATT STRENGTHS

Powered by
Improve XYZ to Beat ABC

XYZ WEAKNESSES
Powered by
Test: XYZ Still Has Weaknesses

Improve XYZ Second Time to Beat ABC –
IMPROVE TWICE
Powered by

Thank you! Now how about a little Q&A?
Powered by
Bill Zangwill
Email: willard.zangwill@chicagobooth.edu

Powered by
1. Stop using usual methodologies as they likely have
hidden risk, bias, errors.
3. Improve twice.
4. Follow procedure, otherwise bias and risks will secretly,
stealthily and surreptitiously slither in.

How Four Statistical Rules Forecast Who Wins a Competitive Bid

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

More from Arik Johnson

More from Arik Johnson (16)

Recently uploaded

Recently uploaded (20)

How Four Statistical Rules Forecast Who Wins a Competitive Bid