Upcoming SlideShare
×

# Probability Calibration

144 views

Published on

A presentation from 2008 to a private security conference in the UK.

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Probability Calibration

1. 1. Phil HugginsPrivate Conference Spring 2008
2. 2.  Risk Uncertainty Cardinals versus Ordinals Probability Estimation Accuracy Suppressing Uncertainty Foxes and Hedgehogs Uncertainty Calibration
3. 3.  A state of uncertainty Set of possible ?negative? outcomes A set of possibilities with quantified probabilities and consequences
4. 4.  Lack of complete certainty More than one possibility True outcome not known A set of probabilities assigned to a set of possibilities
5. 5.  Cardinal -> Size Cardinal -> A number or % Ordinal -> Position in Set Ordinal -> High/Medium/Low or 1/2/3/4/5 Ordinals often have different meanings depending on the person
6. 6.  An estimate of reality The usefulness of an estimate has two measures  Calibration ▪ How close to reality is the estimate  Discrimination ▪ The range of the estimate
7. 7.  We’re the ‘experts’ Client doesn’t want doubt Client pushes the risk of the decision onto you Hitting targets for risk reduction focuses on goals not outcomes
8. 8. ‘The fox knows many things but the hedgehog knows one big thing’ – Isiah BerlinFoxes Hedgehogs• Probability is an ad-hoc • Probability is a deductiveestimation exercise• Low confidence in their • Highly confident in theirestimates estimates• Tolerant of uncertainty • Intolerant of uncertainty
9. 9.  Foxes are generally better calibrated and discriminated Less ‘foxy’ foxes are slightly worse at discrimination Best overall at making short term predictions in field of expertise Hedgehogs worst overall when making long term predictions in field of expertise Hedgehogs win less but win big
10. 10.  90% Confidence Interval (CI) ‘You are 90% certain the probability of occurrence is between 60% and 50%’  95% sure below the upper bound  95% sure above the lower bound Ask yourself 20 general knowledge questions asking for numerical answers  Assign lower and upper bounds to your 90% CI of being correct Ask yourself 20 true/false questions  Set a value to how certain you are 50%, 60%, 70%, 80% 90%, 100%
11. 11.  General Knowledge  You expected to get 9 out of 10 right (90% CI)  What did you get?  You probably got less than that right True / False  Convert % to decimal (50% = 0.5)  Add them up  Compare to number of questions you got right  If number of correct answers was 2.5 or more less then you are overconfident
12. 12.  Take multiple tests in succession Equivalence Bets (next slide) Pros and Cons  Consider two pros why you should be confident and two cons why you could be wrong Avoid Anchoring  95% certain below upper bound  95% certain above lower bound
13. 13.  Two choices: 1. Win £1000 if answer falls within your upper and lower bounds. 2. Spin a dial split into 90% and 10% slices. You win £1000 if it finishes on the 90% slice Most people (80%) prefer to spin the dial Ask yourself which you would prefer  Preferring 1 is over confident  Preferring 2 is under confident Just pretending to bet money makes you better
14. 14.  After 3 tests 70% of people are perfectly calibrated After 3 tests 20% of people are significantly improved 10% of people never really get it Testing has shown that most people relied on to make estimates are in the first two groups already
15. 15. Thanks for listeninghttp://blog.blackswansecurity.com