1) The document describes a study using structured expert judgement (SEJ) to elicit predictions of future political violence frequency from multiple experts in the (re)insurance industry. 2) In the SEJ method, experts were given seed questions to rate their forecasting ability, and then asked a target question about political violence forecasts. Their answers to the seed questions determined performance-based weightings that were used to combine their answers to the target question. 3) The results showed that weighting the experts by their performance on seed questions and combining their judgements produced a forecast that was better calibrated and more tightly estimated than either an equal-weighted combination or the forecasts of individual experts.

1. FORECASTING POLITICAL
VIOLENCE FREQUENCY
A FIRST STRUCTURED EXPERT JUDGEMENT
ELICITATION IN (RE)INSURANCE
DR RAVEEM ISMAIL
SPECIALTY TREATY UNDERWRITER, ARIEL RE
raveem.ismail@arielre.com team@terrorismcyberinsurance.comraveem.ismail@oxon.org

2. Overview:
Opinions & Judgements

3. There are no hard facts, just endless opinions. Every day, the news media deliver forecasts without
reporting, or even asking, how good the forecasters really are. Every day, corporations and
governments pay for forecasts that may be prescient or worthless or something in between. And
every day, all of us - leaders of nations, corporate executives, investors, and voters - make critical
decisions on the basis of forecasts whose quality is unknown.
Superforecasting: The Art & Science Of Prediction.
Tetlock & Gardner, 2015, Crown Publishers.
We put investor capital at risk in exchange for premium.
Decision-making should ideally arise from objective criteria: exhaustive data / sound physical principles.
Yet we often have to act in data-poor environments, where we rely heavily on Expert Judgement.
3

4. • A single expert’s opinion could be an outlier …
… but consulting 10 experts will yield 10 answers.
Each answer is an unknowable function of that expert.
• No method of selecting between judgements…
… decision makers often stick to what they know best.
No indicators demanded of forecasting capability.
• How to makes sense of multiple opinions?
Simple averaging: only limited gains.
Each expert weighted equally without regard for capability.
Final answer may be worse than individual answers.
4

5. Structured Expert Judgement (SEJ)

6. SEJ differs from, and extends, previous opinion pooling methods.
1. Experts must be rated on previous performance.
• Ask set of seed questions, to which answer is already known.
• Performance on seed questions ascertains expert’s weighting.
2. Ask the experts the target question.
3. Weightings drawn from seed questions then used to combine experts’
judgement on target.
This produces one outcome which truly combines different judgements in a
performance-based way, and is potentially better than any individual answer.
• Seed question design is critical!
6

7. SEJ differs from, and extends, previous opinion pooling methods.
1. Experts must be rated on previous performance.
• Ask set of seed questions, to which answer is already known.
• Performance on seed questions ascertains expert’s weighting.
2. Ask the experts the target question.
3. Weightings drawn from seed questions then used to combine experts’
judgement on target.
This produces one outcome which truly combines different judgements in a
performance-based way, and is potentially better than any individual answer.
• Seed question design is critical!
7
Seed
Question A
Seed
Question B
Seed
Question C
Expert 1 Weighting 1

8. Expert 1 Weighting 1
Expert 2 Weighting 2
Expert 3 Weighting 3
Expert 4
Expert 5
Seed
Questions
Weighting 4
Weighting 5

9. SEJ differs from, and extends, previous opinion pooling methods.
1. Experts must be rated on previous performance.
• Ask set of seed questions, to which answer is already known.
• Performance on seed questions ascertains expert’s weighting.
2. Ask experts the target question (the actual judgement being sought).
3. Weightings drawn from seed questions then used to combine experts’
judgement on target question.
This produces one outcome which truly combines different judgements in a
performance-based way, and is potentially better than any individual answer.
• Seed question design is critical!
9
Target
question
Weighted
Judgement
Weighted
Experts 1
2 3
4
5

10. A First (Re)insurance Elicitation:
Future Political Violence Frequency

11. SEJ method: Cooke’s Classical Model.
Two metrics requested from experts, within the range of possible values:
• A 5% to 95% confidence interval.
• A central median value.
11
Minimum
value
Maximum
value
Lower
Uncertainty
Bound (5%)
Upper
Uncertainty
Bound (95%)
Central Value
(50%)
These provide a measure of:
“Calibration” How accurate the expert is.
“Information” How tightly the expert gauges his/her own uncertainty.

12. 12

13. Expert Calibration Information
Equal
weights
Performance
weights
1 0.2295 1.864 0.1111 0.428
2 <0.0001 1.783 0.1111 <0.0001
3 0.002 2.04 0.1111 0.0041
4 0.2274 1.665 0.1111 0.3785
5 0.0002 2.153 0.1111 0.0006
6 <0.0001 3.01 0.1111 0.0002
7 <0.0001 1.505 0.1111 <0.0001
8 <0.0001 2.495 0.1111 <0.0001
9 0.0001 0.734 0.1111 <0.0001
Equal-weighted
combination
0.6286 0.869
Performance-
weighted combination
0.5173 1.701
0.4242
0.7561

14. 14

15. 15

17. Conclusion

18. • Structured Expert Judgement is still judgement.
• But it is not pure guess work, it is:
• a transparent method of pooling multiple opinions,
• which are weighted according to comprehensible performance criteria,
• those performance criteria are directly aligned to the actual judgements being sought.
• Where data and models are lacking, provides objective and auditable method of producing decision-
making judgements and inputs to models.
• We have described a first SEJ in our area of interest, where the method identifies true expertise and
outperforms uncalibrated methods.
• So not a magic bullet, but where applicable, it enhances decision-making and risk appraisal:
Stochastic pricing, Reinsurance purchase, Capital Modelling, ILS, etc.
18

19. Questions?
19
Notes:
• The full study, Structured Expert Judgement (SEJ) For Political
Violence In (Re)insurance, will be a forthcoming publication in
a scientific journal, permanent URL until then:
http://1drv.ms/1VZkuGh .
• Cooke Classical Method for elicitation of Structured Expert
Judgement: Experts In Uncertainty, 1991, Oxford University
Press.
• The ISCH EU COST Action IS1304 for Structured Expert
Judgement aims to bridge the gap between scientific
uncertainty and evidence-based decision making. The political
violence elicitation referenced here took place in London in
January 2016, kindly hosted by Dickie Whitaker and the
Lighthill Risk Network, run by the COST Action’s Reinsurance
Special Interest Group, chaired by Dr Raveem Ismail, with
other principal investigators being Christoph Werner
(Strathclyde University), and Professor Willy Aspinall (Bristol
University).