Lookback analysis provided by Stochastic ApS for oil and gas exploration departments. The company, wells and dates are (obviously) made up.

1. Lookback analysis of exploration assessments
covering the five year period from 1973 to 1977
Prepared for Pangloss Oil & Gas Ltd.
by Graeme Keith, Stochastic ApS

2. Accumulated sequence plot of success rate
- Each point in the figure corresponds to a
sequence of wells. The first, leftmost, point
corresponds to the first well, the second to the
first two wells, and so on to the last, rightmost,
point which shows the full sequence of 75
wells.
- For each point, the actual success rate up to
that point is plotted together with the expected,
P90 and P10 success rate predicted by the
pre-drill assessments of the probabilities of
success of the wells in the sequence up to that
point.
- Actual success rate is well under the
predicted for the full sequence, indicating
possible optimism bias

3. Sliding window plot of success rate
- Each point in the figure corresponds to a
sequence of 25 wells. The first, leftmost, point
corresponds to wells 1 to 25 in the sequence,
the second to wells 2 to 26. The last, rightmost,
point corresponds to wells 51 to 75.
- For each window, the actual success rate for
the wells in that windown is plotted together
with the expected, P90 and P10 success rate
predicted by the pre-drill assessments of the
probabilities of success of the wells in the
window.
- The constant sequence length in the sliding
window plot gives a clearer, less ambiguous
picture of the evolution of the fidelity of
predictions over time (later points
uncontaminated by early statistical
aberrations)
- Actual success rate is consistently under
predicted. Average predicted success rate is
high relative to global averages.

4. Probability plot for success rate
- The full sequence is divided into five groups
based on assessed pre-drill probability of
success: 0 to 20%, 20% to 40%, 40% to 60%,
60% to 80% and 80% to 100%
- Actual success rates are shown as columns
together with the expected, P90 and P10
success rates predicted by pre-drill
assessments of the prospects in the interval.
- The plot is designed to reveal polarization - the
tendency to overestimate the significance of
evidence and assess probabilities too high or
too low - and the converse baselining - a
tendency not to move probabilities far enough
on the basis of evidence.
- Polarization leads to over-prediction in the
lower intervals and under-prediction in the
upper intervals.
- Baselining leads to under-prediction in the
lower intervals and over-prediction in the upper
intervals.
- Actuals are over-predicted in all probability
ranges. Together with the absence of any
probability assessment between 0% and 20%,
this confirms a consistent optimism

5. Inferred probability bias parameters
- Stochastic ApS has developed a methodology
that quantifies the amount by which
probabilities are systematically over- or under-
estimated (optimism, pessimism) and the
amount by which probabilities are moved away
from or towards a base-rate (polarization,
baselining).
- These biases are captured in two numbers, the
probability bias parameters, that can be used to
compare performance across regions and trap
types as well as with other companies and
industries.
- The analysis uses a Bayesian model to mimic
the effects of the different biases. The
parameters are then inferred from predictions
and outcomes using Bayesian inversion.
- The Bayesian analysis returns a probability
distribution on the parameters revealing not
only the most likely combination of biases, but
also the uncertainty that necessarily arises
from short sequences.
- Pangloss’ bias footprint reveals a
pronounced, consistent optimism, though
little in the way of polarization.

6. Visualization of probability biases
- Using the probability bias parameters
presented in the previous slide, it is possible to
reconstruct the relationship of the probabilities
as given to the probabilities as they ought to
have been.
- The optimism can clearly be seen with
probabilities being pushed up across the full
range of ”correct” values (red).

7. Accumulated sequence plot of discovery resources
- Each point in the figure corresponds to a
sequence of discoveries. The first, leftmost,
point corresponds to the first discovery, the
second to the first two discoveries, and so on
to the last, rightmost, point which shows the
full sequence of 28 discoveries.
- For each point, the actual cumulative
discovered volume up to that point is plotted
together with the expected, P90 and P10
cumulative volume predicted by the pre-drill
assessments of the volume distributions of the
wells in the sequence up to that point.
- Notwithstanding, an early run of relative
success, the rate of resource of addition is
clearly over-predicted

8. Sliding window plot of discovery resources
- Each point in the figure corresponds to a
sequence of 10 discoveries. The first, leftmost,
point corresponds to wells 1 to 10 in the
sequence, the second to wells 2 to 11. The last,
rightmost, point corresponds to wells 11 to 28.
- For each window, the actual average volume
per discovery in that windown is plotted
together with the expected, P90 and P10
volume per discovery predicted by the pre-drill
assessments of the volume distributions of the
wells in the window.
- The constant sequence length in the sliding
window plot gives a clearer, less ambiguous
picture of the evolution of the fidelity of
predictions over time (later points
uncontaminated by early statistical
aberrations)
- The pattern from the cumulative sequence
plot is confirmed with an early run of resource
additions in line with predictions, followed by
a long sequence of over-prediction.

9. Percentile plot
- Discoveries are split into ranges according to
the distribution percentile corresponding to the
discovered volume
- The proportion of the total number of
discoveries that fall into each group is plotted
in the percentile plot.
- The distribution of percentiles is uniform so we
should expect to see roughly 20% of
discoveries falling in each percentile interval
- However, there will be a considerable amount
of statistical variance around this 20% value, as
shown the uncertainty range, which is a
function of the number of discoveries.
- The plot is designed to reveal over-confidence -
a tendency to predict ranges that are too small
- and vagueness - a tendency to predict
unecessarily large uncertainty ranges.
- Over-confidence leads to a higher proportion of
discoveries landing in the lowest and highest
percentile intervals.
- Vagueness leads to a higher proportion of
discoveries landing in the middle percentile
intervals.
- Actuals show a systematic tendency to come
in low, skewing the proportions to the lower
percentile ranges. Consistent with optimism
in the assessment of resource parameters.

10. Resource uncertainty bias parameters
- Stochastic ApS has developed a methodology
that quantifies the amount by which probability
distributions are systematically over- or
under-estimated (optimism, pessimism) and
the amount by which ranges are over- or
under-estimated (over-confidence, vagueness).
- These biases are captured in two numbers, the
volume bias parameters, that can be used to
compare performance across regions and trap
types as well as with other companies and
industries.
- The analysis uses a Bayesian model to mimic
the effects of the different biases. The
parameters are then inferred from predictions
and outcomes using Bayesian inversion.
- The Bayesian analysis returns a probability
distribution on the parameters revealing not
only the most likely combination of biases, but
also the uncertainty that necessarily arises
from short sequences.
- Pangloss’ volume bias footprint also reveals a
pronounced tendency to over-optimism,
although ranges are reasonable predicted.

11. Empirical distribution plot
- An alternative way to illustrate the deviation
from uniformity of the prediction of percentiles
is by plotting the actual attained percentiles in
an ordered list from least to greatest.
- Theory predicts that - as always, within a
certain range of uncertainty - these percentiles
should fall on a straight line according to their
uniform distribution.
- The empirical distribution of percentiles can
also be predicted based on the resource bias
parameters presented in the previous slide.
- The plot reveals both optimism and pessimism,
as well as over-confidence and vagueness.
- Optimism leads to percentiles that are
consistently under the no-bias curve.
Pessimism, conversely, leads to an upward lift
on the empirical curve
- Over-confidence leads to a steepening of the
curve as prospects come in with unexpectedly
low and high percentiles.
- Conversely vagueness leads to a flattening of
the curve, as everything comes in around the
middle of the wide ranges
- Both the empirical distribution and the
predicted distribution based on the bias
parameters show a tendency to overestimate
the central parameters of the distribution.