Advisian studied the performance of risk-based cost estimating based on data obtained from 23 water infrastructure projects delivered by several water authorities in NSW over the period 2002 - 2012

1. www.advisian.com
Michael Quinnell – Senior Associate, Advisian
November 2016
Risk-based cost
estimating for water
infrastructure projects –
does it stack up?

2. Inaccurate cost estimates can lead to
suboptimal project selection, inefficient
budget management and disruption to
the delivery of other essential projects.

3. To overcome these issues and improve
the management of their investment
programs, a number of NSW water
authorities have adopted a stochastic
approach to estimating: Risk-based
cost estimating (RBCE).

4. Risk-based Cost Estimating

5. What is it?
A stochastic process utilising Monte-Carlo Simulation
modelling to reflect the degree of uncertainty (risk
and opportunity) associated with a cost estimate.
Base estimate
Inherent risk:
Uncertainty within
base estimate,
defined assumptions
and known scope
Contingent risk:
Uncertainty from
unplanned events
Risk
workshop
Monte Carlo
Simulation
Sample output of RBCE process
BASE COST
ESTIMATE
COST
BASE RISK CONTINGENCY
RANGE
CUMULATIVE
PROBABILITY
FORECAST EXPECTED
COST (MEAN OR
MEDIAN [P50])
MAKE FINANCIAL
PROVISION FOR WORSE
COST OUTCOME
RESIDUAL RISK
EXPOSURE RANGE
20%
40%
60%
80%
100%
PROBABILITY(CUMULATIVE)

6. Why use the RBCE process?
1.Identify, quantify and
incorporate risks and
opportunities into the
project budget
2.Incorporate project
knowledge and
experience from a wide
stakeholder group
through the risk
workshop process
3.Provides organisations
the information
required for decision
makers to set project
budgets that match the
risk tolerance
4.A more sophisticated
approach to allocating
appropriate project
contingency

7. Understanding the performance
of the RBCE process

8. water infrastructure
projects studied 2002 – 2012
range in total
actual cost values
23 $4.2m to
$191.7m
Projects delivered by
several water authorities
in NSW over the period

9. Study objectives:
Obtain a better understanding of the
performance of the RBCE process
Identify areas where the process can be
further improved in the future
1.
2.

10. Data collected
Final actual project cost
Actual cost incurred by the water
authority at completion of the
project; includes initial contract
price, any variations paid and
client costs
Data collected from each project included:
Project timing data
Date when the estimate was
prepared, contract award date
and the date of practical
completion
Original cost estimate
Prepared at project appraisal
stage at the time where the
decision was made to go ahead
with the project

11. Data Adjustments
Adjustments were made to ensure a ‘like-for-like’ comparison between budget and
actual project cost. These included:
Escalation adjustment to account for
the difference between base period
of estimate and actual project
delivery timeframe
Increased or decreased
scope change

12. So… does it stack up?

13. Findings 1: How accurate
are RBCE budgets?
• The water projects in the studies sample
performed well compared to other studies
• Cost underruns were found to occur more
frequently than overruns
• Weighted average of projects under
budget was 8.5%
• Out of the 23 projects:
• 17% of projects experienced cost
overruns
• 83% of projects were on
or under budget
Difference (%) =
Actual Project Cost
(adjusted to ensure consistent comparison)
Project Budget (raw)
X 100
% overrun/underrun vs. Estimate date
20%
10%
0%
-10%
-20%
-30%
-40%
-50%
A B C D E F G H I J K L M N O P Q R S T U V W
%overrun/underrunvs.Estimatedate
Projects sorted by estimate preparation date
Older Newer

14. Findings 2: RBCE success
in forecasting outcomes
The main output from the RBCE process is not
a single estimate; it is a range of potential
outcomes.
• The range of outcomes were between the
P1 to P99 value
• Only 57% of the sample fell within the
expected range
Possible causes:
• Project Managers ‘anchor’ on the base
estimate (i.e. they’re not willing to adjust
or expand the range to capture the best
and worst case scenario)
• Results in the forecasted range are too
narrow to capture the true level of
uncertainty
50%
57% of actual project
costs were within the
expected range
CostProbability(cumulative)
100%
“It was expected that 98%
of projects would fall
within the defined range”

15. Findings 3: Reliability of
confidence from RBCE
A key feature of the RBCE process is that it
provides a level of confidence for each value
in the range of potential project outcomes.
• Between P50 and P90, confidence levels
in the process were reliable
However…
120%
100%
80%
60%
40%
20%
0%
P01 P99P10 P20 P30 P40 P50 P60 P70 P80 P90
Well calibrated
results
(i.e. accurate
representation
of p-value)
P Value
%ofprojectatorbelowbudget(underrun)
Expected % Underrun
Actual % Underrun

16. Findings 3: Reliability of
confidence from RBCE
Below the P50 and above the P90 level there
was a wide difference between the expected
and the actual result.
This suggests:
• The process isn’t placing enough focus on
identifying opportunities
• The process isn’t capturing the ‘extreme
risks’ (low likelihood and/or very high
consequence)
A possible explanation could be that it
may not be worth the effort of accurately
estimating figures outside of the P50
and P90 range (which are used the
most often).
120%
100%
80%
60%
40%
20%
0%
P01 P99P10 P20 P30 P40 P50 P60 P70 P80 P90
Extreme risks not
adequately considere
P Value
%ofprojectatorbelowbudget(underrun)
Expected % Underrun
Actual % Underrun
Results indicate value
<P50 too conservative

17. Findings 4: The Portfolio
Perspective
Public agencies delivering projects face the
dual challenge of neither exceeding their
allocated funding nor significantly
underspending the allocated funding.
• Results highlighted an opportunity for
more efficient contingency allocation
• Agencies could set a less conservative P-
value as the budget and reduce the
portfolio cost underrun
• Means accepting that a greater number
of projects will overrun vs individual
project budgets, but the overall
portfolio is sufficient to deliver all
projects
• Provides ability to invest this
released contingency on other
valuable projects or agency
priorities % deviation from budget
Total portfolio actual cost
Total portfolio budget
P01 P99P10 P20 P30 P40 P50 P60 P70 P80 P90
1,400
1,200
1000
800
600
400
200
-
10.00%
5.00%
0.00%
-5.00%
-10.00%
-15.00
Millions
%deviationfrombudget
Had the program budget been set
at P50, the percentage underrun
would have reduced from 8.5%
under the budget to 4%

18. Areas for improvement

19. Issues relating to cognitive bias
or delusions - such as
anchoring and adjustment -
affected the results of the
process.
This was particularly
highlighted by the limitations
to fully capture the range of
expected outcomes and in
producing reliable confidence
levels below the P50 and
above the P90 levels.
This can be mitigated by
placing more emphasis on
identifying key opportunities
rather than only focusing on
risks, as well as calibration
techniques and training.
1. 2. 3.

21. DISCLAIMER
This presentation has been prepared by a representative of Advisian.
The presentation contains the professional and personal opinions of the presenter, which are given in good faith. As such, opinions presented
herein may not always necessarily reflect the position of Advisian as a whole, its officers or executive.
Any forward-looking statements included in this presentation will involve subjective judgment and analysis and are subject to uncertainties, risks and
contingencies—many of which are outside the control of, and may be unknown to, Advisian.
Advisian and all associated entities and representatives make no representation or warranty as to the accuracy, reliability or completeness of
information in this document and do not take responsibility for updating any information or correcting any error or omission that may become
apparent after this document has been issued.
To the extent permitted by law, Advisian and its officers, employees, related bodies and agents disclaim all liability—direct, indirect or consequential
(and whether or not arising out of the negligence, default or lack of care of Advisian and/or any of its agents)—for any loss or damage suffered by a
recipient or other persons arising out of, or in connection with, any use or reliance on this presentation or information.