Probability Calibration


Published on

A presentation from 2008 to a private security conference in the UK.

Published in: Business
  • Be the first to comment

  • Be the first to like this

Probability Calibration

  1. 1. Phil HugginsPrivate Conference Spring 2008
  2. 2.  Risk Uncertainty Cardinals versus Ordinals Probability Estimation Accuracy Suppressing Uncertainty Foxes and Hedgehogs Uncertainty Calibration
  3. 3.  A state of uncertainty Set of possible ?negative? outcomes A set of possibilities with quantified probabilities and consequences
  4. 4.  Lack of complete certainty More than one possibility True outcome not known A set of probabilities assigned to a set of possibilities
  5. 5.  Cardinal -> Size Cardinal -> A number or % Ordinal -> Position in Set Ordinal -> High/Medium/Low or 1/2/3/4/5 Ordinals often have different meanings depending on the person
  6. 6.  An estimate of reality The usefulness of an estimate has two measures  Calibration ▪ How close to reality is the estimate  Discrimination ▪ The range of the estimate
  7. 7.  We’re the ‘experts’ Client doesn’t want doubt Client pushes the risk of the decision onto you Hitting targets for risk reduction focuses on goals not outcomes
  8. 8. ‘The fox knows many things but the hedgehog knows one big thing’ – Isiah BerlinFoxes Hedgehogs• Probability is an ad-hoc • Probability is a deductiveestimation exercise• Low confidence in their • Highly confident in theirestimates estimates• Tolerant of uncertainty • Intolerant of uncertainty
  9. 9.  Foxes are generally better calibrated and discriminated Less ‘foxy’ foxes are slightly worse at discrimination Best overall at making short term predictions in field of expertise Hedgehogs worst overall when making long term predictions in field of expertise Hedgehogs win less but win big
  10. 10.  90% Confidence Interval (CI) ‘You are 90% certain the probability of occurrence is between 60% and 50%’  95% sure below the upper bound  95% sure above the lower bound Ask yourself 20 general knowledge questions asking for numerical answers  Assign lower and upper bounds to your 90% CI of being correct Ask yourself 20 true/false questions  Set a value to how certain you are 50%, 60%, 70%, 80% 90%, 100%
  11. 11.  General Knowledge  You expected to get 9 out of 10 right (90% CI)  What did you get?  You probably got less than that right True / False  Convert % to decimal (50% = 0.5)  Add them up  Compare to number of questions you got right  If number of correct answers was 2.5 or more less then you are overconfident
  12. 12.  Take multiple tests in succession Equivalence Bets (next slide) Pros and Cons  Consider two pros why you should be confident and two cons why you could be wrong Avoid Anchoring  95% certain below upper bound  95% certain above lower bound
  13. 13.  Two choices: 1. Win £1000 if answer falls within your upper and lower bounds. 2. Spin a dial split into 90% and 10% slices. You win £1000 if it finishes on the 90% slice Most people (80%) prefer to spin the dial Ask yourself which you would prefer  Preferring 1 is over confident  Preferring 2 is under confident Just pretending to bet money makes you better
  14. 14.  After 3 tests 70% of people are perfectly calibrated After 3 tests 20% of people are significantly improved 10% of people never really get it Testing has shown that most people relied on to make estimates are in the first two groups already
  15. 15. Thanks for listening