Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Four Statistical Rules Forecast Who Wins a Competitive Bid

705 views

Published on

Can Bayesian statistics really determine in advance if the bid you are offering will be the winner or just another loser? And, if the metrics forecast a loss, can the same algorithm tell you what to change in order to win instead?

Competitive bidding is where big money sales opportunities are won or lost, and there are four (4) rules that can help you turn a losing situation into a winning sale.

These four rules help you better understand what the customer wants, examine what competitors might do in response and how to beat them, while helping you to offer the best bid, optimized for yours and your prospective customer’s intended outcome. Statistical metrics evaluate your probability of success against the competition and help you more objectively determine how to win. But how can you get at the foundational issues that will determine who will win?

Learning objectives:

Learn the Four Rules that help you understand what will actually determine the customer’s decision.
Visualize your bid head-to-head against the competition and employ objective metrics to determine if you will win.
Identify weaknesses in your offer that must be improved for your bid to beat the competition.

Bill Zangwill is a Professor, Emeritus, from the University of Chicago, Booth School of Business. He has authored four published books, one of which was selected by the Library Journal as “One of the Best Business Books of the Year,” and had over 50 papers in academic journals. In addition, he has had three articles published in the Wall Street Journal. His consulting engagements include top firms such as IBM, AT&T, Motorola, many smaller firms and the US government. He has also taught at the University of Illinois and the University of California, Berkeley. He is considered one of the most innovative thinkers in his field.

Bill will present 30 minutes on how the four rules can help you turn a losing situation into a winning sale and will be joined by webinar moderator Arik Johnson, Founder & Chairman at Aurora WDC.

Published in: Business
  • Be the first to comment

How Four Statistical Rules Forecast Who Wins a Competitive Bid

  1. 1. The Intelligence Collaborative http://IntelCollab.com #IntelCollab How Four Statistical Rules Forecast Who Wins a Competitive Bid Powered by A Complimentary Webinar from Aurora WDC 12:00 Noon Eastern /// Wednesday 22 October 2014 ~ featuring ~ Dr. Bill Zangwill Arik Johnson
  2. 2. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Bill Zangwill Bill Zangwill is a Professor, Emeritus, from the University of Chicago, Booth School of Business. He has authored four published books, one of which was selected by the Library Journal as “One of the Best Business Books of the Year,” and had over 50 papers in academic journals. In addition, he has had three articles published in the Wall Street Journal. His consulting engagements include top firms such as IBM, AT&T, Motorola, many smaller firms and the US government. He has also taught at the University of Illinois and the University of California, Berkeley. He is considered one of the most innovative thinkers in his field. Email: willard.zangwill@chicagobooth.edu The Intelligence Collaborative is the online learning and networking community powered by Aurora WDC, our clients, partners and other friends and dedicated to exploring how to apply intelligence methods to solve real-world business problems. Apply for a free 30-day trial membership at http://IntelCollab.com or learn more about Aurora at http://AuroraWDC.com – see you next time!
  3. 3. Questions, Commentary & Content The Intelligence Collaborative http://IntelCollab.com #IntelCollab α Use the Questions pane on your GoToWebinar control panel and all questions will be answered in the second half of the hour. α You are welcome to tweet any comments on Twitter where we are monitoring the hashtag #IntelCollab or eavesdrop via http://tweetchat.com/room/IntelCollab α Slides will be available after the webinar for embedding and sharing via http://slideshare.net/IntelCollab α To view the recording and download the PPT file, please register for a trial membership at http://IntelCollab.com. Powered by
  4. 4. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Agenda ► Introduction ► Risk, Bias, Errors: Caused by Usual Procedures, but Usually Missed ► Application to Situations in Business ► Summary, Q&A and Discussion
  5. 5. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by The Joy When We Win.
  6. 6. How do we avoid the grief and attain the joy? The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  7. 7. Novel Approaches. Two patents, third pending. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  8. 8. Why are the patents and new approaches needed? Because usual methods typically have hidden bias, hidden risks and hidden errors. ► MAJOR CONUNDRUM: We believe in the usual approaches and keep using them because the errors are hidden so are missed and not considered. ► E.g., Behavioral research. If we believe the decision will be 70-80% The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by successful, likely it will be roughly 50%. ► First methodology that expressly helps warn of errors, bias, issues missed ► Humans miss issues. ► 2 patents and third pending ► Follow the four rules
  9. 9. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Four Rules – Boost Win Rate 1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them. 2. Do not assume. TEST. 3. Improve twice. Best ideas come after reviewing situation a couple times. 4. Follow procedure, otherwise bias and risks will slip in secretly, stealthily and surreptitiously.
  10. 10. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Risk, Bias, Errors: Caused by Usual Procedures, but Usually Missed Math. Bias. Surprises.
  11. 11. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Does Arithmetic Have a Hidden Secret? Example: SUBTRACTION 100 $100,000,000 -90 -$90,000,000 ----- --------------- 10 $10,000,000 WHAT CAN POSSIBLY BE HIDDEN HERE? WHAT MIGHT BE WRONG?
  12. 12. In Business Decisions, Most Data Are Not Known That Perfectly The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Example: SUBTRACTION ERROR 100 -90 ----- 10 But error in value 100 +/_ 10 (90 – 110) ( 10% error) -90 ------ 0 to 20 (100% error) HUGE ERROR INCREASE FROM 10% to 100% WORSE IF BOTH FACTORS HAVE ERRORS OR OVER TIME Math causes error. TYPICAL ALGORITHM: WE BELIEVE IT IS RIGHT YET INSTEAD PRODUCES ERROR
  13. 13. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by DCF and Other Financial Formula Can Increase Error Gordon growth model: $10 million payout in one year, g = growth rate, r= rate of return Value, DCF = $10/ (r – g) (in millions) Example 1: r = 12% , g = 10% Value = $10/ ( 0.12 - 0.10) = $10/(0.02) = $500 million Example 2: r = 12% , g = 11% growth rate increased by 10% Value = $10/ ( 0.12 - 0.11) = $10/(0.01) = $1 billion 10% error in input 100% error in answer. Proprietary material. Do not disseminate. Patent pending and copyrights University of Chicago and Decision Command, Inc. SLAM(TM) is a registered service mark of Decision Command.
  14. 14. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by More Accurate Using Better Math Two quantities, 40 and 10 What percentage is 40 of total? 40/( 40 + 10) = 40/50 80% Suppose 25% error in 40. Could be 30 – 50. What is percentage of the total now? 30/(30 +10) 75% 50/(50 + 10) 83.66% Much reduced error relative to 80% base --– error less than 7% ERROR REDUCED FROM 25% TO LESS THAN 7%. THE MATH IN MOST ALGORITHMS IN COMMON USE CANNOT BE TRUSTED. Must use better approaches that reduce error and attack the hidden secret.
  15. 15. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Improves Accuracy, Reduces Risk Pij = conditional probability in a cell, Pj = probability of column   (Uses Bayes and Laplace approaches) Suppose there is risk in a specific cell entry Plm Consider risk (percent error) in the output Pm over risk in the input Plm. The fractional risk in output is The fractional risk in the input is i ij j i ij j P P P   m lm m dP dP P 1 P lm Proprietary Information, FOUO
  16. 16.  The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Error Percentage Always Decreases, Proprietary Information, FOUO      i ij i l im i im i l im j m m 2 lm i ij m lm j lm lm m i im lm i ij j i ij i l im i im i l im i ij i im i im i im j j lm i im i im i ij i im j j i ij i im j i ij ( P ) P P P dP [ dP dP ( P ) P dP [ ](P ) ](P ) 1 P P P P ( P ) P P P ( P ) P P P (P ) ( P ) P ( P ) P ( P ) P ( P                                       j i im i ij j ) P 1 1 ( P )        Percentage error out less than percentage error in
  17. 17. Serious Problems Often Cause Us to Lose: Most Algorithms Increase Risk and Errors, and We Do Not Know It Bayesian Algorithm I Use Reduces Risk and Errors 1. “The model was previously shown to be surprisingly robust to obvious violations of this independence assumption, yielding accurate classification models even when there are clear conditional dependencies.”  “Many researchers have noted the good performance of SBC, including Clark and Niblett, Knononenko, Langley and Sage, and Domingos and Pazzani.” 2. Beyond Independence: ..is commonly thought to assume that attributes are independent given the class, but this is apparently contradicted by the surprisingly good performance it exhibits in many domains that contain clear attribute dependences. No explanation for this has been proposed so far. In this paper we show that the SBC does not in fact assume attribute independence, and can be optimal even when this assumption is violated by a wide margin The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Proprietary. Confidential information. US Patent 7,676.446. Copyright University of Chicago and/or Decision Command.
  18. 18. INSIGHT: Statistically identify entries that are odd relative to facts of situation, outliers The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by BIAS – Frequent Cause of Not Winning Bias and optical illusions are similar. What you believe is not there. Holding hands, or not? Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  19. 19. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Bias and Hidden Risk Biases, hidden risk, faulty assumptions tend to be at variance, out-of-kilter with facts of situation. Anomaly, outlier or oddity is often the clue: risk, bias, faulty assumption outlier
  20. 20. Surprise – Must the Unexpected Always be Unexpected? INSIGHT: Identify contradictory and odd information. The greater the level of contradictory and odd information, the greater chance something odd is going on, something was missed. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by MATH ADVANCEMENT– Add variable to represent surprise Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  21. 21. Bayes System First System to Warn of Risks/Surprises/Missed Issues Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  22. 22. Usual Methods Typically Have Hidden Error, Risk and Bias, and Often We Will Not Know It. ► NO AUTOMATIC IDENTIFICATION OF RISK. ► NO AUTOMATIC MEANS TO PREVENT MATH ERRORS. ► NO AUTOMATIC CROSS-CHECKS ON ACCURACY. ► NO AUTOMATIC ANALYSIS OF RISK OR SURPRISE/MISSED The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by ISSUES. ► NO AUTOMATIC ESTIMATE OF THE PROBABILITY OF SUCCESS. ► NO PROCESS TO RESOLVE DISAGREEMENTS NEED BETTER WARNING OF THE HIDDEN PROBLEMS “YOU CAN’T DO A GOOD JOB WITH POOR TOOLS” Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  23. 23. TEST AT UNIVERSITY OF CHICAGO Managers and executives made decisions They compared quality of decision made this new way versus their usual decision approach. 40% better and 20% faster The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Proof Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  24. 24. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Four Rules – Boost Win Rate 1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them. 2. Do not assume. TEST. 3. Improve twice. Best ideas come after reviewing situation a couple times. 4. Follow procedure, otherwise bias and risks will slither in secretly, stealthily and surreptitiously. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  25. 25. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Application to Situations in Business Use better ways to handle math, bias, risks Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  26. 26. INVEST IN EXCELLENT CHILEAN FIRM. PARTNER. In private education, growing rapidly, wanted funds for international growth and acquisition AAI could assist with funding growth. Gateway to future. Several trips to Chile to speak with owner AAI BIDDING AGAINST ARCH RIVAL ADVENT Felt confident they would win— got to know owner, understood situation The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Bidding to Invest in a Chilean Firm Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  27. 27. AAI WERE HIGHLY EXPERT AND DID TRADITIONAL ANALYSIS WELL---SURE THEY WOULD WIN : analyzed situation, obtained information, visited Chile, brought in experts, financial analysis, discussions. RESISTED USE OF SYSTEMATIC METHOD, SUPERFLUOUS. But agreed to use it because easy, fast, and I was a friend. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Situation: AAI Confident of Winning Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  28. 28. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Difficulties with Usual Methods Made decision as if they were making the decision. No goals from perspective of Santo Tomas. •No metrics for those goals. Prob. ST will select AAI or Advent. No independent warning of possible biases, risks. •No probability of missed issues/surprise. No iterations- building of better and better solns. (First analysis is always incomplete---Donnelly’s rule) •No metrics to determine if decision is getting better. No testing if decision excellent or not. arguments, blind alleys. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  29. 29. 5 Minute Role Play Real Decision Maker MUST GET AT EMOTIONS. WHAT ARE THEY CONCERNED OR WORRIED ABOUT? Utilize goals, hopes, fears, problems of real decision maker. What could cause failure? What causes customer worries? Santo Tomas goals. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  30. 30. We Fail (Missed before because traditional had no means to detect.) EXPOSED HUGE RISK. ASTOUNDED. BUT THEY HAD TO AGREE BECAUSE THEY DID THE Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by EVALUATION. WHAT SHOULD WE DO?
  31. 31. - Test: What are the risks that cause our loss? Why we lose Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by TRADITIONAL NO WARNING OF RISKS
  32. 32. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Traditional Methods: Argue ►BEFORE: Argue who is right. ►SYSTEMATIC APPROACH: TEST. Identify risks. Examine possible improvements and check their metrics Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  33. 33. New approach to win. BOOSTED METRICS: success rate from 63% to 75% beating Advent’s 70% REVISED AAI STRATEGY Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by METRICS AAI 63-75 ADVENT 70
  34. 34. Test—expose risks again. Still some issues, so build higher – IMPROVE TWICE Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  35. 35. Build the win higher—test and improve twice Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  36. 36. ► Bring in experts on international to our team and show Santo Tomas, that we are really best for global expertise. Create connection in Chile to build trust. ► Convince Santo Tomas our financial deal is really better. (less upfront, but more later) ► Ensure CEO comfortable (CEO is owner) Conclusion: saved them from losing deal Traditional means fail due to hidden errors, no warning. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Develop New Strategy - Won Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  37. 37. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Four Rules – Boost Win Rate 1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them. 2. Do not assume. TEST. 3. Improve twice. Best ideas come after reviewing situation a couple times. 4. Follow procedure, otherwise bias and risks will slither in secretly, stealthily and surreptitiously. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  38. 38. New Product Development at EG Insurance. What is going on? Large commercial insurance firm wants to sell to mortgage originator insurance market Traditional product based upon annual premium— The Intelligence Collaborative http://IntelCollab.com #IntelCollab Exciting new product they expect to launch to clearly differentiate themselves from competitors: Chubb, CNA, Lloyds of London: A transaction-based plan based upon monthly volume. Client would only pay for what was actually used. Powered by WE’RE READY TO GO AHEAD. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  39. 39. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Role Play: What does customer think? Get at worries. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  40. 40. Test: Complete matrix, examine metrics (cannot be done in traditional) New product loses. Software saved them from failure! Forced analysis from viewpoint of actual decision maker. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013 METRICS: Prob. Success Low, 16%, 53%. Easy to miss this. WAY UNDER GOAL OF 75%
  41. 41. Warning. METRIC High surprise risk – “You can do better than this, you missed something” The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013 METRICS: Success, 16%, 53% Surprise 31%
  42. 42. Examine Hidden Risks -- Countering risks leads to new concept. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by risks Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  43. 43. NEW IDEA: Transaction based, but on closed loans only. Should save money. ►TEST IDEA: DO NOT ASSUME: (cannot be done with traditional because no metrics) The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  44. 44. Test New idea: Transaction based, but on closed loans only– Surprise still high The Intelligence Collaborative http://IntelCollab.com #IntelCollab METRICS: Prob. of success is lower. Surprise drops to 20% from 31% What to do now? Build Powered by Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  45. 45. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Test: Check Risks risks Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  46. 46. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Check More Risks and Weaknesses Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  47. 47. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by TEST another idea — Large increase in estimated success probability metric Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013 METRICS: Success 71% Surprise 16% CLOSE ENOUGH TO 75%
  48. 48. TWO IMPROVEMENT CYCLES Sizable improvement in probability of winning over prior approach. Value ZERO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Closed loans only($54) Initial-New transaction idea($31) Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013 Cust. choice($74) Traditional
  49. 49. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Saved Them From Losing Money. Usual Means Have Hidden Error $4.3 MILLION SAVED. Over 30% improvement . Three cycles. Exposed that initial evaluation was incorrect “ H a v i n g s o f twa r e l i k e t h i s a r o u n d to aid in making important d e c i s i o n s i s wo n d e r f u l . ” “ 5 0% f a s t e r t h a n r e g u l a r me t h o d . ” Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
  50. 50. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Four Rules – Boost Win Rate 1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them. 2. Do not assume. TEST. 3. Improve twice. Best ideas come after reviewing situation a couple times. 4. Follow procedure, otherwise bias and risks will slither in secretly, stealthily and surreptitiously. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  51. 51. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by IT Competitive Bid WE LOSE Prob. of success
  52. 52. Test: Despite ABC Winning, Maybe ABC Can Be Beaten The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Factors where XYZ beats ABC where ABC beats XYZ
  53. 53. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Where are others strong or weak? XYZ STRENGTHS SMS STRENGTHS ATT STRENGTHS
  54. 54. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Improve XYZ to Beat ABC Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  55. 55. The Intelligence Collaborative http://IntelCollab.com #IntelCollab XYZ WEAKNESSES Powered by Test: XYZ Still Has Weaknesses Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
  56. 56. Improve XYZ Second Time to Beat ABC – IMPROVE TWICE Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by
  57. 57. Thank you! Now how about a little Q&A? The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Bill Zangwill Email: willard.zangwill@chicagobooth.edu
  58. 58. The Intelligence Collaborative http://IntelCollab.com #IntelCollab Powered by Four Rules – Boost Win Rate 1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them. 2. Do not assume. TEST. 3. Improve twice. Best ideas come after reviewing situation a couple times. 4. Follow procedure, otherwise bias and risks will secretly, stealthily and surreptitiously slither in. Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO

×