APM RISK MANAGEMENT SIG CONFERENCE Alexander Budzier

2. Using an outside view of project risk for
realistic risk and capability assessments
Alexander Budzier, DPhil
Fellow in Management Practice
Saïd Business School, University of Oxford

3. Our Research Insights Helped Us to Understand the
Root Causes of Issues and Develop Long-Lasting Solutions
3
Symptoms
Root
causes
Issue Fix
Cost overruns,
schedule delays,
benefit shortfalls
Fixed price contact,
risk sharing, sticks &
carrots
Causes Unknown geology,
scope changes etc.
Reactive risk
management
• Optimism and
political bias
• Culture, mindsets
and behaviors
Predict & provide
Predict & prevent
▪ Long-lasting solutions need to
understand and address real
causes and their root causes
▪ Quick fixes only offer limited help

4. 2 Strategies to De-Risk Projects
4
Accurate
planning
Outside view to
▪ Accurately estimate risk
(inside view leads to
optimism bias)
▪ Reference Class
Forecasting as one method
to systematically take the
outside view
Outside view to
▪ Identify potential
weaknesses in projects that
actually matter
▪ Identify early warning signs
and leading risk indicators
Challenged
capabilities
Inaccurate
planning
Improved
capabilities
Predict & provide
Predict &
prevent

5. The 3 Steps of RCF
1. Identify relevant reference class of past, similar
projects
2. Establish probability distribution for the selected
reference class
3. Compare specific project with distribution, in order to
establish most likely outcome
5© Bent Flyvbjerg

6. * The p-value of statistical tests indicates the strength of the evidence, if p < 0.05 the test is significant – here
indicating that there is strong statistical evidence that these project types are different from HSR
Source: Oxford Database, August 2015 (Sample of n=361 projects)
6
Statistical analysis of risk profile
• Risk profile characterized by distribution of cost risk in reference classes
• Peak = P50 risk
• Tail = Risks > P50
Building a Reference Class

7. 7
• Bottom-up QRA estimates the
35% contingency to be a P90
• P90 = 9 out of 10 projects would
not exceed this envelope
• RCF shows that 35% contingency
is equivalent to P66
Building a Reference Class for HS2

8. 8
• HS2: 35% contingency = P66
• P90 (conservative estimate) = 120%
• P95 (very conservative estimate) = 152%
Building a Reference Class for HS2

9. Comparing the Inside View OBC with
the Outside View RCF Estimate
9
2
1
Regress most likely estimate
toward median of reference class
1
Expand estimate of tail risk to
to tail of the reference class
2

10. Increased Design Certainty Should (in Theory)
Reduce Cost Uncertainty
10
Project cycle
SBC OBC FBC Tender Construction Completion
Final cost
Increased design
certainty
Reduced cost
uncertainty

11. Different Reference Classes for Different
Points of the Estimate
11
Strategic BusinessCase
Outline BusinessCase
Full BusinessCase
Contract ControlTotals
(total value of contracts)
-10%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
20 25 30 35 40 45 50 55 60 65 70 75 80
CostRisk
Level of Certainty of the Estimate (P Value)
• 35% contingency covers cost risk
exposure of
• SBC: P63
• OBC/FBC: P66
• Contract control totals: P77
35%

12. Key Concern is the Tail Risk in the
Reference Class Forecast
-70%
-60%
-50%
-40%
-30%
-20%
-10%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
140%
150%
0 10 20 30 40 50 60 70 80 90 100
Contingencyneeded
P-value
12
35%
• Key concern is tail risk, when
risks start to grow
exponentially with increasing
uncertainty
• Tail of the RCF begins at P66
• 1 in 3 projects in reference
class had cost overruns >
35%

13. Example of HSR Project in the Tail
• NSB Nuremberg-Ingolstadt, Germany
• First proposed 1973, planning started 1991, decision to build 1994
• Construction start 2000
• Opening 2006
• Estimated cost: 2.3 bn EUR (YoE); estimated opening 2003
• Length 77.4 km incl. 15 km in tunnels, 82 viaducts
• Cost overrun of +133%
• Reasons for cost overrun
• Geological problems in tunneling sections; hydrology problems along the alignment
• Changed security concept after the Eschede accident in 1998
• Increased environmental impact mitigation cost
• Underground archeological remains
• Unanticipated inflation
• Late design submission of pre-fab track bed
• Cost savings: not upgrading several station and using instead of upgrading existing track,
change of tunneling method
• Cost for interest payments
• Issues with integrating the HSR line into operational railway meant that travel time
savings were only achieved for end-to-end journeys, no clear concepts for tact
times, critique of abandoning same platform connections
• 2010 travel speed reduced from 300 km/h to 160 km/h due to quality issues with
the rubber mats in the track bed
13

14. Characteristics of Projects Exceeding
Contingency
14
Area Description HS2 Response
Design Late design changes due to external
demands
Hybrid Bill process controls scope
Funding Problems with funding causing
changes to profile. Including
excessive interest payments.
Agreed funding profile up front and
funding mechanisms including
necessary insurance strategy.
Amplification Problems on critical path escalate
rapidly without ‘damping’
Designing schedule to create higher
confidence at critical points eg end of
pre-construction. Early contractor
involvement in design
Integration Failing to appreciate and understand
scale of integration required to deliver
Structure of contracts. Assurance
approach. Technical specification and
assurance
Quality Problems with quality of work
completed
Assurance design. Hand over points
control.
Archaeology Significant volumes of archaeology
and geology issues
An allowance made and base is case
is towards the worst case
Inflation Inflation greater than expected Specific recognition and approach to
this issue.

15. Front-End Capability Maturity Model
15
5. Master Builder
Leadership
3. Reference
Class Forecasting
1. Benchmarking
2. Due Diligence
4. Black Swan
Management
▪ Projects planned with
an inside view only
▪ Optimism bottom-up in
estimates unchecked
▪ No internal capability to
challenge cost and
schedule forecast of
contractors
▪ No capability to identify
‘low balled’ bids
▪ No understanding of the
uncertainty of estimates
▪ Wide range of
capabilities with few
pockets of
excellence
▪ Careful balance
between individual
and organisational
capabilities
▪ High-powered
experts lack
understanding of
way of working
▪ Lack of
empowerment of
experts (stifled by
process)
▪ No culture of
excellence but a
culture of heroes
▪ No systematic
approach to taking the
outside view
▪ Variability between
projects’ and sub-
projects’ ability to
reduce bias
▪ Ignorance of unknown-
unknowns
▪ Incentives to de-risk
projects and safeguard
contingencies not
aligned in supply chains
▪ Slow management
information leads to
predictable surprises
▪ Biased and narrow
management reporting
deaf to weak signals
▪ Overly complex
projects (social, political
and time complexity)
with management of
symptoms not causes
(ie sources of
uncertainty and
complexity)
▪ Tight coupling and
interactive complexity
make project fragile
▪ Is the project
conducting internal and
external benchmarks?
▪ What level of realism
does this show?
▪ Have lessons learned
been incorporated?
▪ How is the procurement
strategy and early
contractor involvement
structured?
▪ What are the incentives
for forecasters?
▪ Is probabilistic
forecasting used
consistently?
▪ Has the project taken a
systematic view to
compare itself to other
projects?
▪ How mature is the risk
management (if any)
and other PM
disciplines to manage
the front-end process?
▪ Has the project
analysed its sources
of complexity?
▪ Have complexities
been actively
managed or
mitigated?
▪ How compressed is
the schedule?
▪ How confident is the
project in the
capability of its
leaders?
▪ How mature are the
PM processes?
▪ Do the project and
supply chain share
understanding of
success factors?

16. Risk Maturity Level
16
Maturity
Level
Area Problem factors Potential response (examples)
1 Bench
marking
Inside view only with optimism bias
unchecked
Programme of learning from others.
Benchmarking against other projects,
industries and sectors
2 Due
diligence
No challenge to cost, schedule
forecasts of contractors.
Minimal understanding of
uncertainty in estimate
Internal assessments of costs, schedules.
Separate estimating from contractors
3 Reference
Class
Forecasting
No systematic outside view.
Variability between projects and
subprojects ability to align and
reduce bias.
Ignorance of ‘unknown unknowns’;
lack of exploration.
Systematic and ongoing Reference Class
Forecasting across different aspects of
programme.
Exploration of scenarios to understand
‘unknown unknowns’.
Understand characteristics of tail
4 Black Swan
(predictable
surprises)
Slow management information
communication.
Management of symptoms not
causes.
Tight coupled system.
Overly complex
Quick, clean, independent management
information reporting
Amplification of weak signals.
Recognise complex scenarios and break
down
5 Leadership High powered experts lack
understanding of how to work
effectively. No culture of excellence
rather one of heroes. Lack of
empowerment of experts
Identify necessary pockets of excellence
and encourage culture. Continuous
learning.
Focus on ways of working
Leadership engage in scenario planning
Increase reliance on ‘qualitative’
(needs more development)

17. Summary: Risk Management is about
Building Confidence not Certainty
17
Accurate
planning
Outside view taken to
▪ Establish de-biased view of
risk
▪ Engage with funder to
move away from P95
perceptions
Outside view taken to
▪ Learn systematically and
quickly from others
▪ Work on project capabilities
through maturity framework
thinking
▪ Stress importance of
improving planning and not
only construction phase
Challenged
capabilities
Inaccurate
planning
Improved
capabilities
Predict & provide
Predict &
prevent

18. Any Questions?
Alexander.Budzier@sbs.ox.ac.uk