Eliciting systematic bias in probabilistic assessments through statistical comparison of assessments with outcomes. My presentation to a workshop on risk and uncertainty at the EAGE conference in Copenhagen in June.

2. Optimism and confidence in the oil industry
Graeme Keith MA PhD FIMA
Gentleman of Leisure
(On gardening leave following Total’s acquisition of Maersk Oil)

3. 0
0,02
0,04
0,06
0,08
0,1
0,12
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
Probability
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
F S
Probability
Single prospect with 20% chance of success
2 outcomes, both quite plausible
100 prospects each 20% chance of success
2100 outcomes, some of which are quite implausible

4. Aggregate sequences to reduce the
range of reasonable outcomes
• Introduce some unreasonable outcomes
• Study sequence mean and variance
Validate sequences not prospects
• Give up inference on individual prospects
• Open up for inference on systematic biases
Infer probabilities by fitting biases to data
• Making eliciting systematic biases the object of study
• Mode matching, maximum likelihood, Bayesian inference

5. The ubiquitous iniquitous accumulating sequence plot
• Tradition is to cumulate sequences
• The purpose of accumulating is to
reduce range of reasonable
outcomes, so all sequence plots are
meaningless without some sense of
the range…

6. The ubiquitous accumulating sequence plot with 80% confidence interval
• Simulate distributions (dashed)
• Or use Central Limit Theorem
(solid green)
• Vertical scale set by number of
discoveries and not by deviations…

7. Accumulating sequence plot showing success rates (sequence mean)
• Moving to mean (success rate)
shows reduction of uncertainty
• False time evolution – some of
trend due to changes in sequence
length, not comparing like with like
• Every point on accumulation plot
has different sequence length

8. Sliding window plot showing success rates
• Pick a window length and slide it
along the sequence
• Here window length is 21 in a
sequence of 125 wells
• First point is the sequence
from well 1 to well 21
• Second is from well 2 to 22
• Last is from 105 to 125
• Use range to set window length
• Long enough to see errors, short
enough to see time evolution

9. All the same considerations apply to volumes, including and not including failures

10. The mean is not the only sequence property we can scrutinize
• Sample Variance
= Variance in prospect means
+ Mean of prospect variances
• Prospect variance = p(1-p)
• Variance measures uncertainty,
high when you don’t know anything
• Underprediction shows over-confidence
• Overprediction shows vagueness
0 0,5 1

11. Similar considerations apply to volume variance, here restricted to discoveries
• Sample Variance
= Variance in prospect means
+ Mean of prospect variances
• Prospect variance
• Variance measures uncertainty,
high when you don’t know anything
• Underprediction shows over-confidence
• Overprediction shows vagueness
• But distributions are highly skewed

12. Traditionally assess variance (confidence) through other visualizations, like the
Iniquitous probability plot
• Binning probabilities is another way of
looking at confidence
• Actually just sequence aggregation
• Meaningless without distributions
0-20 20-40 40-60 60-80 80-100

13. Probability plot with 80% confidence interval
• Binning probabilities is another way of
looking at confidence
• Actually just sequence aggregation
• Meaningless without distributions
• Distribution of prospects within bins is
not usually uniform, especially for the
highest and lowest
• Calculate distributions for the actual
prospects in the bin
• 0.85=33% chance that all of the bins fall
in 80% CI!
• Ranges function both of probability (high
and low probabilities have smaller
ranges) and number of prospects in bin
0-20 20-40 40-60 60-80 80-100

14. Similar approach to looking at confidence and variance in volume results
• Distribution of percentiles should be
uniform, percentile plot is one (rather
crude) way of examining this
• Meaningless without distributions
>P20P40-P20P60-P40P80-P60P100-P80

15. Percentile plot with 80% confidence interval
• Each prospect has a 20% chance of
falling in a given bin.
• Distribution in a given bin is Poisson with
N=number of wells and p=0.2
• 0.85=33% chance that all of the bins fall
in range!
>P20P40-P20P60-P40P80-P60P100-P80

16. Empirical distribution provides a slightly more elegant way of looking at volume
variance without looking at volume variance
• Empirical distribution looks at
proportion of discoveries that fall above
given percentiles
P0
P100

17. Investigate systematic biases by postulating a form of a systematic bias (same for
every prospect in a sequence) and then analysing outcomes relative to predictions
to elicit the bias parameters. Looking first at volume
Assessed
probability
”Faithful”
probability
Systematic
bias and
confidence
Outcomes
log 𝑉𝑖 ∼ 𝒩(𝜇𝑖, 𝜎𝑖)
𝜇𝑖
𝑡
= 𝜇𝑖 + 𝛿𝜇
𝜎𝑖
𝑡
= 𝜙𝜎𝑖
log 𝑉𝑖 − 𝜇𝑖
𝑡
𝜎𝑖
𝑡 ∼ 𝒩(−𝛿𝜇,
1
𝜎𝑖
)
f
dm

18. Optimism
Volumes fall
consistently short
Vagueness
Volumes gather
around P50
Pessimism
Volumes consistently
exceed expectation
Overconfidence
Volumes gather
around extremes
Biases distort empirical distribution curve in predictable way

19. Pangloss
Oil & Gas
No oil left behind
P0
P100
Small specialist exploration company.
Humble with respect to uncertainty, but
biased after their first, lucky, large discovery

20. Hubris
Industries
World class geoscience
P0
P100
Small exploration company made from ex-
chiefs from large companies. Solid
predictions on average but no humility at
all. If it’s good, it’s very very good and if it’s
bad it’s horrid.

21. Now looking at success and failures
Assessed
probability
”Faithful”
probability
Systematic
bias and
confidence
Outcomes
𝑝𝑖
𝑞𝑖 = 𝑝𝑖 + 𝜖(𝑝𝑖 − ҧ𝑥)
𝑟𝑖 = 𝑞𝑖 + 𝛿
Choose d and e to minimize Ψ = σ𝑖 𝑥𝑖 − 𝑟𝑖
2
Two parameter transform,
d captures optimism,
e captures confidence
𝜋𝑖 = 10 log
𝑝𝑖
1 − 𝑝𝑖
𝜃𝑖 = 𝜋𝑖 + 𝜖(𝜋𝑖 − 𝜋0)
𝜌𝑖 = 𝜃𝑖 + 𝛿
𝑟𝑖 =
10 Τ𝜌 𝑖 10
1 + 10 Τ𝜌 𝑖 10
𝛿 = ҧ𝑟 − ҧ𝑥
𝜖 =
𝑟2 − ҧ𝑟2
𝑥𝑟 − ҧ𝑥 ҧ𝑟
− 1
Solve numerically

22. Optimism
Probabilities shifted
up
Vagueness
Probabilities gather
around mean
Pessimism
Probabilities shifted
down
Overconfidence
Probabilities pushed
out to extremes
Can’t visualize outcome biases directly, but can use inferred parameters to plot
relationship between ”faithful” and biased probabilities

23. Lucky to survive a run of failures and non-
commercial discoveries, EE are consistently
conservative in their assessments
Eeyore Enterprises
Every silver lining has its cloud

24. Sybil Oil
Better vaguely right than
certainly wrong
Culture of deciding by committee and can’t
agree on anything. All probabilities close to
base rate and very large P10/P90 ratios.
Have falling in uncertainty range as KPI (but
not occasionally falling outside).

25. A taxonomy of systematic bias
• Presentation shows methods to elicit and quantify systematic assessment errors
• Methods reveal systematic trends otherwise hidden in traditional lookback visualizations
• Systematic bias models can be used to simulate lookback results to
• Develop intuition on what systematic failures can be seen in various visualizations
• And what can not reasonably be seen for noise with the sequence length available
Outcome Volume
Optimism Probabilities consistently too high,
mean overpredicted1
P50 volumes too high2
Pessimism Probabilities consistently too low,
mean underpredicted
P50 volumes too low
Over-confidence Probabilities too polarized,
variance underpredicted3
Ranges2,3 too narrow, variance
underpredicted
Under-confidence Probabilities too close to baseline,
variance overpredicted
Ranges too wide
1) Probabilities biased up towards 50% will also result in an increase in variance and probabilities biased downwards away from 50% will result in a lower variance
2) It is mathematically more sound to work with normal mean and variance, i.e. the mean and variance of the (approximately) normal distribution with which the logarithm of the volumes is distributed.
For this reason, we work with P50 (which corresponds to the logarithm of the normal mean) and P10/P90 ratios (proportional with the normal standard deviation)
3) Over-confidence is often mistaken for optimism or (less often) pessimism. Falling outside of a confidence interval can be because the confidence interval is too small.
One of the big advantages of the methods advocated here is the ability to tell the difference

26. Future work: Disutility of poor probabilistic prediction
drill
drop
success
failure
V
zero
Cost of well
develop
abandon
V
$
Cost of wellSystematic bias model may be used to predict the
expected value erosion from the biases.
In the decision model, decisions are based on
biased probabilities, i.e. probabilities given by the
bias model, but expectations are performed using
the original ”faithful” probabilities
Figure shows value erosion for simple decision
model (no appraisal, but no development if volume
is less than commercial threshold, NPV is linear
with volume above threshold).
Results show substantial value erosion
Note asymmetry: Fortune favours the brave! (well,
more than the correspondingly timid at any rate)