Estimating what does good look like

Estimating Skills Training In Methods Approaches Techniques & Analysis EST.i.MAT-Ax
x
xx
x
© 2012-2017 Estimata Limited X
Cost Estimating and Forecasting
What does ‘good’ look like?
Promoting TRACEability in Estimating
“Prediction is very difficult,
especially if it’s about the future.”
Niels Henrik David Bohr
1885-1962
Danish Physicist and Nobel Laureate
1
ICE Lancashire Branch Joint Presentation with APM Management Group
Wednesday 15 November 2017
Speaker:
Alan R Jones
Estimata Limited

x
xx
x
Introduction and Objectives
Who am I?
Why am I here?
EST.i.MAT-A
2

x
xx
x
Who am I?
You’re probably sitting there, looking at me, thinking, “… and you are?”
• Graduate in Mathematics (four decades ago)
• Worked for BAE Systems here in the North West for over 38 years (until 2015)
• Mostly in Estimating or in related disciplines
• Architect of its End-to-End Estimating Process, cited by the MoD as being “Best Practice”
• Now, an independent consultant in Estimating Capability Development
• Certified Cost Estimator & Analyst through the US-based ICEAA
• Certified Cost Engineer through the UK-based Association of Cost Engineers (ACostE)
• Director of the ACostE, and Chair of its Accreditation Board
• Estimating Author, including:
• Series of five ‘Working Guides to Estimating & Forecasting’ to be published by Routledge
in 2018
• Contributing Author to APM / ACostE Joint Publication: ‘A Guide to Estimating Methods
and Approaches for Project Management’ to be published also in 2018
3

x
xx
x
What does ‘Good’ look like? Inside the Head of an Estimator
The domain knowledge required includes an
understanding of an individual organisation
and its supply chain … and an appreciation of
its customers / clients
However, the underlying skills, methods and
techniques employed by estimators and
forecasters are fundamentally generic across
all industry sectors
Also, just like any other profession, Estimators
need to work effectively with others and be in
possession of a liberal dose of organisational
and interpersonal skills
4
Domain
Knowledge
Technical
Skills
Organisational and
Interpersonal Skills
Domain knowledge within an Engineering sector is of paramount importance to
give the appropriate context and interpretation of requirements and information

x
xx
x
Let’s talk about the Principles and Practice of Estimating
1. Is it possible to have a robust estimating process that is also flexible? Why
does it need to be flexible?
2. Who should be involved in the estimating process? Is it just the “number
jockeys”, and if not, who else gets to share in the blame?
3. Does a robust estimating process mean a more accurate estimate, or just an
estimate that is more precisely wrong? Isn’t an estimate only as good as the
last set of assumptions?
4. What’s the difference between an Estimating Method, Approach and
Technique, and how many different ones of each are there?
5. What is meant by a 3-Point Estimate; and isn’t that just avoiding the
question of “what is it going to cost”?
6. How do we deal with Risks and Opportunities? What is good practice, and
what is questionable practice?
5

x
xx
x
Course Agenda
Good Practice in Estimating & Forecasting
(The Science and Dark Art of Estimate Creation)
• What is an Estimate?
• Planning and Managing the Estimating Process
• Establishing Estimate Readiness
• Estimate Creation
• Methods Approaches and Techniques
• Accuracy Precision and 3-Point Estimates
• Risk Opportunity and Uncertainty
• Estimate Validation and Challenge
• Summary
6
x
xx
x
A Good Practice Estimating Process Framework
Estimate Creation is just a small part of the Estimating System
Customer
Requirement
& Estimate
Initiation
Estimate Planning, Management & Change Control
Baseline
Estimate
Creation
Estimate
Validation,
Challenge
&
Clearance
Project
Realisation,
Performance
Monitoring &
Forecasting
Feedback
Risk
Opportunity
&
Uncertainty
Evaluation
Stakeholder
Agreements
(ADORE
or MDAL)
Pricing,
Negotiation
& Customer
Acceptance
Historical Performance
(Data and Context)
Estimating Capability Development and Governance
(Process, Skills Development & Deployment, Tools and Techniques)
ADORE = Assumptions Dependencies Opportunities Risks & Exclusions MDAL = Master Data and Assumptions List
Feedback
10
ADORE
TRACE

x
xx
x

 ? 

What is an Estimate?
A Definition
7

x
xx
x
What do we mean by an Estimate?
What is an Estimate? Is it …
a) A number that is “roughly right”?
b) An educated guess?
c) Something that is always slightly lower than we usually achieve?
d) Something to get us on contract but that we can conveniently forget
once we are?
e) All of these?
f) None of these?
Definition:
An Estimate for “something” is a numerical expression of the approximate value
that might reasonably be expected to occur based on a given context, which is
described and is bounded by a number of parameters and assumptions, all of
which are pertinent to and necessarily accompany the numerical value provided.
8
What do you call an Estimate without
its contextual framing?
… a Random Number

x
xx
x
What makes a good estimate?
All estimates are inherently ‘wrong’ … (in an exact sense)
Due in part to:
• Uncertainty in the scope of what we are being asked to estimate
• Factors outside of our control or influence (e.g. performance)
• Lack of sufficient appropriate data on which to base our estimate
• Insufficient contextual history for the data we do have, in which case we
cannot be sure that we are comparing on a “like or like” basis
… and for complex projects, products or services:
• Too many variables, not all of which have been identified or understood
• Insufficient time to create the estimate
9

x
xx
x
So, what makes these ‘inexact’ estimates “right” enough?
✓ There is a statement of scope that has been developed and agreed with all
stakeholders covering:
• What has been included within the baseline, or as a risk or an opportunity
• What has been excluded
• What the key parameters are that describe and bound the “environment”
in which the estimate is expected to be transformed into reality
+ If different estimators using different Approaches, Methods and Techniques,
can generate comparable estimate values for the same contextual framework
+ If the perceived accuracy can be quantified with an upper and lower limit
+ If there is a clear Basis of Estimate that explains the rationale behind the
estimate calculations and sources of information used
… so how do we get there?
10

x
xx
x
5W+H
Planning and Managing the Estimating Process
Setting, agreeing and sticking to the Contextual Framework
… a question of good project management discipline
Who else should be involved in the estimating process?
If it’s not just the “number jockeys”, who else gets to share in the
blame?
11

x
xx
x
Create
the
Estimate
Review
the
Estimate
Plan the
Estimate
Estimates don’t just happen!
If only there was some Magic Hat with
Estimating Rabbits popping out …
… but regrettably, there’s no such thing
Estimates need to be properly planned and
managed
12
… and every estimate is different, so every plan
will be different, but there are some good practice
principles we can follow
In its Simplest Form, the
Estimating Process is an
Iterative Process

x
xx
x
A Good Practice Estimating Process should pass the CLIFF test:
Closed Loop – Iterative – Flexible – Functional
• Closed Loop: Requires feedback from within the process and on completion of the
work in question to improve process performance and output
accuracy
• Iterative: Recognises that requirements get clarified or changed progressively
and assumptions evolve. Configuration Control is paramount
• Flexible: The process needs to be able to respond appropriately where
timescales are short and levels of detail information are lacking; this
may require some elements of the process to be optional, conditional
or discretionary, whereas other will be mandatory
• Functional: Fit for Purpose - Regardless of optional steps omitted, the process
must always produce a range of numerical values against an agreed
scope of work, and a supporting Basis of Estimate all of which have
been challenged or validated and cleared for release
13


x
xx
x
Estimate Creation is just a small part of the Estimating System …
Customer
Requirement
& Estimate
Initiation
Pricing,
Negotiation
& Customer
Acceptance
(Data and Context)
Estimate
Validation,
Challenge
&
Clearance
Baseline
Estimate
Creation
Risk
Opportunity
&
Uncertainty
Evaluation
Project
Realisation,
Performance
Monitoring &
Forecasting
Feedback
Feedback
14
Stakeholder
Agreements
(ADORE
or MDAL)

x
xx
x
It’s a Question of Good Project Management Discipline:
5 W’s and an H Explanation
Why … is the Estimate being produced? Pricing, budgeting etc
When … is the Estimate required? Target completion date
What … is the scope of the work The ADORE or MDAL
Who … needs to be involved or consulted? The Stakeholders
Where … will the Estimate be created? In-house and/or Supplier
How
… will we compile the estimate and bridge
any gaps?
Process, Data, Approach,
Methods, Tools and Techniques
• Start planning as soon as possible
• Use all the time available, … and use it wisely
• Plan to iterate some elements of the estimate more than once
15

x
xx
x
Responsibility Assignment Matrix (RAM)
Who else gets to share in the blame?
Individual Function
Example Deliverable to or Role within the
Estimating Process
Name 1 Engineering
The Technical Solution around which the estimate
will be based
Name 2 Project Management Configuration Control of the ADORE or MDAL
Name 3 Procurement Selection of Suppliers and Obtaining Quotations
Name 4 Accounts Pricing, Budgeting, Cost Reporting
Name 5 Commercial Contract Terms and Conditions
Name 6 Operations Estimate Validation and Commitment
Name 7 Estimating
Process, Data, Approach, Methods, Tools and
Techniques
There may be several stakeholders from a single function in the RAM
16

x
xx
x
Estimators should ADORE the Scope of their Estimates
17
Acronym Definition
A Assumption Something that is taken to be true, or something that will be
considered as being true for planning purposes until changed
D Dependency Something outside the control of the internal organisation but which
must occur for the estimate to be valid
O Opportunity Something that may or may not occur, but if it does occur it will have
a beneficial impact. It is assumed not to occur in the base case
scenario.
R Risk Something that may or may not occur, but if it does occur it will have
a detrimental impact. It is assumed not to occur in the base case
scenario.
E Exclusion Something that is specifically not included in an estimate and has
been specifically recorded as such. If it does subsequently occur its
impacts will not have been covered by the estimate
In some areas, this may be referred to as the Master Data Assumptions List (MDAL)

x
xx
x
The Estimate Structure defines how we agree to “build and box” our estimate
It should reflect how the resulting project or operational budget is to be
managed
Usually achieved through slicing and dicing of multiple Breakdown Structures:
• WBS or Work Breakdown Structure,
or alternatively,
• PBS Product Breakdown Structure
• SBS Service Breakdown Structure
• OBS Organisational Breakdown Structure
• RBS Resource Breakdown Structure,
which in turn aligns with a
CBS Cost Breakdown Structure
Estimate
Structure
The different
estimating elements
do not all have to be
at the same level of
breakdown
Structuring the Estimate
18
OBS
RBS
or CBS
WBS or
PBS / SBS

x
xx
x
Establishing Estimating Readiness
On your marks; get set … wait!
Managing Information Maturity
Don’t start until you’re ready to start
Part of Planning and Managing an Estimate
19

x
xx
x
Concept of Estimating Readiness
The fundamental objective of Planning and Managing the Estimate is to have all
the information available that we need, along with any assumptions,
dependencies, opportunities, risks and exclusions agreed
The biggest risk of creating an estimate that is less robust than we would like,
comes from starting to create the estimate value before we are ready:
• The information we need is not ready, or is incomplete
• Yet we often work to the schedule start date, and not to the plan’s critical path!
• Typically, this leads to reworking of the estimate through unplanned or
unwarranted iterations of the estimate, and long, late hours playing catch-up
Some of this can be avoided proactively through regular Estimating Readiness
Reviews
20

x
xx
x
Estimating Readiness Reviews (ERR)
What are they?
• Often a part of a wider Project Readiness Review
• The primary focus of an ERR is on whether we are “ready to estimate”, and
not whether “the Estimate is ready” i.e. completed and cleared
ERRs typically:
• Are drumbeat reviews, often held weekly, to manage and progress the
Estimating Plan
• Provide an essential control and filter for developing and evolving the ADORE
pack on which the Estimate will be based
• Determine the answer to the question: “Are the information requirements and
the scope sufficiently mature for the organisation to commence creating the
estimate and supporting Basis of Estimate?”
21

x
xx
x
Estimate Creation
What does good look like?
The Practice and Procedure of Estimate Creation
22
6.2832
Estimate
Generator
Machine
x
xx
x
Customer
Requirement
& Estimate
Initiation
Baseline
Estimate
Creation
Estimate
Validation,
Challenge
&
Clearance
Project
Realisation,
Performance
Monitoring &
Forecasting
Feedback
Risk
Opportunity
&
Uncertainty
Evaluation
Stakeholder
Agreements
(ADORE
or MDAL)
Pricing,
Negotiation
& Customer
Acceptance
(Data and Context)
Feedback
10

x
xx
x
Generic Estimating Procedure with TRACEability
Document
the Basis of
Estimate
Review the
Requirement
Select the
Approach &
Method
Obtain
Historical
Data
Analyse the
Normalised
Data
Review wrt
Sensibility &
Sensitivity
Normalise
the Data
Select an
Analytical
Technique
Create a
3-Point
Estimate
ChangeRequired
TRACEability
Submit for
Independent
Validation
Good to go
23
We will look a little closer
at some of these steps

x
xx
x
TRACEability in Estimates
24
Acronym Key Characteristic
T Transparent There should be nothing hidden in the way an estimate has been
compiled, including the rationale for any judgement calls. Links
directly to the Basis of Estimate (BoE)
R Repeatable Other estimators should be able to reproduce the estimate using
the same facts, assumptions, approaches, methods, techniques
and acknowledging the same judgement calls
A Appropriate The estimate should satisfy the purpose for which it is intended,
in terms of scope content and level of accuracy that can be
expected with the information available
C Credible The rationale and interpretation of the assumptions should be
logical, and the use of professional judgement should be within
supportable bounds
E Experientially
based
Estimates should be based on historical evidence. Where
professional judgement is applied, it should be made by a
professional with the appropriate and credible experience

x
xx
x
Basis of Estimate v ADORE: What’s the difference?
ADORE is used to set the higher level over-arching planning boundaries that
everyone is expected to work within when developing an estimate
As such, a Basis of Estimate (BoE) should include, or make reference to, the
ADORE planning elements
25
Estimate
Structure
S BoE
The BoE will contain more detail on the specific data
assumptions and mechanics of the calculations that we
would normally not include within the higher level
ADORE statements
Typically, an estimate would comprise of a single set
of common ADORE statements and multiple Bases of
Estimate covering each of the different Estimating
Elements defined in the agreed Estimate Structure.

x
xx
x
Methods Approaches and Techniques
… for Creating an Estimate
What’s the difference between an Approach, Method and Technique?
How many different ones are there?
26

x
xx
x
Methods Approaches and Techniques
Approaches
• Top-down
• Bottom-up
• Ethereal
Methods
• Analogy (or “Comparative”)
• Parametric (or Formulaic)
• Trusted Source
Techniques
• Quantitative (Numerical and Statistical)
• Qualitative
27
The general direction employed to compiling the estimate
 Minimalist structured approach
 Often favoured for internal buy-in
 Least robust approach
The way in which the various pieces fit together
 Quickest method
 More robust method
 Method of last resort
How specific estimate values are generated
Multiple Techniques

x
xx
x
WBS 1
1.1
1.1.1
1.1.1.1 1.1.1.2 1.1.1.3
1.1.2
1.1.2.1 1.1.2.2
1.2
1.2.1
1.2.1.1 1.2.1.2 1.2.1.3
1.2.2
1.2.2.1 1.2.2.2
1.2.3
1.2.3.1 1.2.3.2
WBS 1
1.1
1.1.1
1.1.1.1 1.1.1.2 1.1.1.3
1.1.2
1.1.2.1 1.1.2.2
1.2
1.2.1
1.2.1.1 1.2.1.2 1.2.1.3
1.2.2
1.2.2.1 1.2.2.2
1.2.3
1.2.3.1 1.2.3.2
Top-down Approach to Estimating
• In a Top-down Approach to Estimating, we would move down each branch of
the WBS structure until we reach a level at which we believe we could create
an Estimating Element
• We don’t have to stop at the same level on each branch
• Each Estimating Element must cover the work content of all the Estimating
Elements below it
• There must be no gaps in the structure
28

x
xx
x
WBS 1
1.1
1.1.1
1.1.1.1 1.1.1.2 1.1.1.3
1.1.2
1.1.2.1 1.1.2.2
1.2
1.2.1
1.2.1.1 1.2.1.2 1.2.1.3
1.2.2
1.2.2.1 1.2.2.2
1.2.3
1.2.3.1 1.2.3.2
WBS 1
1.1
1.1.1
1.1.1.1 1.1.1.2 1.1.1.3
1.1.2
1.1.2.1 1.1.2.2
1.2
1.2.1
1.2.1.1 1.2.1.2 1.2.1.3
1.2.2
1.2.2.1 1.2.2.2
1.2.3
1.2.3.1 1.2.3.2
Bottom-up Approach to Estimating
• In a Bottom-up Approach to Estimating, we would move up each branch of the
WBS structure until we reach a level at which we believe we could create an
Estimating Element … as a result it is the most labour intensive approach
• We don’t have to start at the same level on each branch
• Each Estimating Element must cover the work content of all the Estimating
Elements below it
• There must be no gaps in the structure
29

x
xx
x
WBS 1
1.1
1.1.1
1.1.1.1 1.1.1.2 1.1.1.3
1.1.2
1.1.2.1 1.1.2.2
1.2
1.2.1
1.2.1.1 1.2.1.2 1.2.1.3
1.2.2
1.2.2.1 1.2.2.2
1.2.3
1.2.3.1 1.2.3.2
Ethereal Approach to Estimating
• However, what if we are receiving inputs for Estimating Elements from a Third
Party, and we do not know whether the value of those Estimating Elements
have been created by a Top-down or Bottom-up Approach?
• The Estimate Elements just appear into the system without full TRACEability
but we choose to accept them anyway
• This is the Ethereal Approach
(Again, there must be no gaps)
30
WBS 1
1.1
1.1.1
1.1.1.1 1.1.1.2 1.1.1.3
1.1.2
1.1.2.1 1.1.2.2
1.2
1.2.1
1.2.1.1 1.2.1.2 1.2.1.3
1.2.2
1.2.2.1 1.2.2.2
1.2.3
1.2.3.1 1.2.3.2
Examples where an Ethereal Approach
might be used:
• Low Value Commodity Items
• Unsubstantiated Vendor Quotations
• RoM Estimates for poorly defined tasks
RoM = Rough Order of Magnitude

x
xx
x
Choosing an Appropriate Estimating Method
There are only three basic Estimating Methods that we can say are APT :
Analogy
Scope
Cost
Actual
Estimate
+20% growth
Single Cost Reference Point
Parametric
Cost
Estimate
Actuals
Multiple Cost Reference Points
Trend
Analysis
Scope
Trusted Source
Scope
Cost
??? Eeny, meeny,
miny, moe
No Cost Reference Points
This might be high,
this might be low
31
Theoretically, the Parametric Method is the most robust

x
xx
x
Choosing an Appropriate Estimating Method
There are only three basic Estimating Methods that we can say are APT :
There are references in some texts to other Methods, but these are really
variations on these three basic Methods. For example:
• Extrapolation from Actuals
• Engineering Build-up
• Simulation
32
… really a special case of the Parametric Method
… more of a reference to a Bottom-up Approach in
which all 3 APT Methodologies might be utilised
… more appropriately considered to be a technique
that can be utilised with an Analogous or
Parametric Method

x
xx
x
Accuracy, Precision and 3-Point Estimates
What is meant by a 3-Point Estimate?
Why isn’t one point not good enough?
Is it not just sitting on the Fence?
What are the 3 Points anyway?
33
Scope
Performance
x
xx
x
Customer
Requirement
& Estimate
Initiation
Baseline
Estimate
Creation
Estimate
Validation,
Challenge
&
Clearance
Project
Realisation,
Performance
Monitoring &
Forecasting
Feedback
Risk
Opportunity
&
Uncertainty
Evaluation
Stakeholder
Agreements
(ADORE
or MDAL)
Pricing,
Negotiation
& Customer
Acceptance
(Data and Context)
Feedback
10

x
xx
x
What do we mean by a 3-Point Estimate?
34
Estimating is not an exact science!
We have to estimate because there are variables that we cannot predict with
certainty
Uncertainty as a Measure of Estimate Accuracy
• Uncertainty is a reflection of the sensitivity of an Estimate in relation to one or
more internal or external parameters, which we know will occur, but the
values of which we cannot control or define exactly
• Uncertainty is also an expression of the Accuracy of an Estimate in relation to
the eventual Outcome
• A high degree of Accuracy would be depicted by a narrow Uncertainty Range
• A low degree of Accuracy would be signified by a wide Uncertainty Range
• Don’t confuse Accuracy with Precision

x
xx
x
Accuracy v Precision
35
Accuracy is not the same as Precision:
Accuracy
Accuracy is an expression of how close a measurement, statistic or estimate
is to the true value or to a defined standard
Precision
There are two uses of this term in relation to Estimating
1. Precision is an expression of how close repeated trials or measurements
are to each other
2. Precision is an expression of the level of exactness reported in a
measurement, statistic or estimate
We should avoid inappropriate exactness in estimating!

x
xx
x
Accuracy v Precision
36
Precisely Accurate
Accuracy HighLow
Precision
Low
Precisely Inaccurate
High
Imprecisely Inaccurate Imprecisely Accurate
Utopian Aim:
Narrow scatter
around the true
scope of work
Pragmatic or
Acceptable Aim:
Repeatable scatter
around the true
scope of work
Undesirable:
Repeatable
process but scope
of work and/or likely
performance is
poorly understood
Unacceptable:
Poor control of the
Estimating Process
and information
management

x
xx
x
What do we mean by a 3-Point Estimate?
37
Consider a discrete element of work:
A 3-point Estimate expresses the range of Uncertainty in the outcome of that
element of work in terms of the time, cost or some technical parameter (such as
weight) that we are estimating
We express this range of uncertainty using 3 values:
1. An Optimistic Value: one which we might achieve if everything falls neatly into
place, but that we are unlikely, realistically, to better. This is sometimes referred to
as the Minimum, but may not be in an absolute sense
2. The Most Likely Value* that we will achieve. This would often be the single point
deterministic estimate value
3. A Pessimistic Value: one at which we might outturn if we do not perform as well
as expected on the element of work. This is sometimes referred to as the
Maximum, but it may not necessarily be the true absolute maximum
* Note: There will be occasions when the middle point is expressed as an Average value or sometimes the Median Value
depending on the circumstances. These will be discussed as they arise

x
xx
x
Why Not Just Stick with the Most Likely Value?
38
• Estimates are usually based on topical or historical evidence around a similar
scope of work or similar elements of work
• The more dissimilar the scope is in relation to that of our evidence, the less
confidence we will have in any specific value, and the greater the range of
uncertainty will be around a particular value
• At an individual cost element level, the degree to which we can underspend,
or perform better than we expect, is less than the degree to which we can
overspend or perform worse than we would like:
• The Elapsed Time (duration) and Cost of a task both have a lower absolute bound
that is greater than zero (they cannot be negative and will not be free)
• Theoretically, the Time and Cost of the task are unbounded to the right (upper
level) ... they could go on indefinitely if we let them!
• Pragmatically, they will be capped at an upper limit (e.g. a poor performing project
might be cancelled before completion)

x
xx
x
The Sum of the Most Likely Values is Bad for Business
39
• Making business pricing decisions on Most Likely Values alone will have a
detrimental effect on the margin
• We are more likely to overspend the Most Likely value than we are to
underspend
• If you were the CEO of your organisation, which would you prefer to have:
• A smaller chance of achieving the required margin based on a more competitive
price?
• Or, a better chance of achieving the margin based on a more realistic price?
0

Time or Cost
Most Likely
> 50% Chance of
exceeding the
Most Likely
< 50% Chance of
being less than the
Most Likely

x
xx
x
The Cone of Uncertainty
40
Estimating is naturally an iterative process:
• An Estimate evolves as the uncertainty around its scope is resolved
• Initial provisional assumptions will be replaced by more evidence-based
knowledge and agreements of requirements, and the solution to deliver them
• Over the life of a product or project, from concept through to delivery, the
range of uncertainty will narrow, funnelling in towards the eventual outcome
This is known as the Cone of Uncertainty
Range of
Uncertainty Elapsed Time from Inception
Greater Uncertainty of
Overspend than Underspend
Progressive Iterations
Links back to evolution of
the ADORE under
Configuration Control

x
xx
x
Risk, Opportunity and Uncertainty
What’s the difference between Risk Opportunity and Uncertainty?
What is good practice?
What is questionable practice?
41
T1_
T2_
T3_
T4_
T5_
_
_
_
_R1
_R2
_R3
x
xx
x
Customer
Requirement
& Estimate
Initiation
Baseline
Estimate
Creation
Estimate
Validation,
Challenge
&
Clearance
Project
Realisation,
Performance
Monitoring &
Forecasting
Feedback
Risk
Opportunity
&
Uncertainty
Evaluation
Stakeholder
Agreements
(ADORE
or MDAL)
Pricing,
Negotiation
& Customer
Acceptance
(Data and Context)
Feedback
10

x
xx
x
Is Uncertainty not the same as Risk and Opportunity?
42
Definition of a Risk
• A Risk is an event that may or may not occur, but if it does occur it will
have a detrimental effect on the overall outcome
Definition of an Opportunity
• An Opportunity is an event that may or may not occur, but if it does occur
it will have a beneficial effect on the overall outcome
Uncertainty around Risks and Opportunities
• If a Risk or an Opportunity does occur, then in the majority of cases, there will
be an associated range of Uncertainty around its impact
• This will be expressed as a 3-Point Estimate around the Most Likely Value for
the Risk or Opportunity
• Clearly, if the Risk or Opportunity does not occur, the impact will be zero
Uncertainty is something that will occur, wherever we have
variables, i.e. values that we cannot control or define exactly

x
xx
x
Risk, Opportunity and Uncertainty (ROU)
ROU Evaluation is closely linked to Risk & Opportunity Management, requiring
information or assumptions on:
• Description of the Risk or Opportunity
• Probability of the Risk or Opportunity Occurring
• The cost of any approved Risk Mitigation plan (included as a Baseline Task)
• The cost of any approved Opportunity Promotion plan (also included as a
Baseline Task)
• 3-Point Estimate for each Risk or Opportunity (Optimistic, Most Likely,
Pessimistic Values) if the Risk or Opportunity were to occur
• Retirement date for each Risk and Opportunity (i.e. when the probability of
occurrence becomes zero)
43
It is essential that the
Estimator only uses the
approved Project Risk &
Opportunity Register

x
xx
x
Risk, Opportunity and Uncertainty (ROU)
ROU Evaluation differs from Risk & Opportunity Management in that the
Estimator must consider the Risks and Opportunities in the context of the
Baseline Tasks as a single interactive system
• One does not exist without the other!
Approaches to ROU Evaluation
• As with the Baseline Tasks, we can approach this in three ways:
• Bottom-up Approach
• Top-down Approach
• Ethereal Approach
44
<<< We’ll discuss these two in a moment
… which is really asking for a Risk Expert’s opinion

x
xx
x
Risk Opportunity & Uncertainty: Bottom-up Approach
45
Bottom-up Approach to ROU Estimating
• Discrete elements of the Baseline Task should only reflect the Uncertainty
around the baseline task, i.e. excluding Risks and Opportunities
• The Uncertainty around the impact of specific individual Risks and
Opportunities (should they arise) will also be expressed as a 3-Point Estimate
• At the overall system level, the aggregation of the discrete work elements of
known Baseline Tasks and any Risks and Opportunities, allows us to express
a 3-Point Estimate of overall variability
• Typically, we would evaluate this using Monte Carlo Simulation
What is Monte Carlo Simulation?
• Shocking though it may sound, it is Estimating by Random Numbers!

x
xx
x
Monte Carlo Simulation
46
What is Monte Carlo Simulation?
3-Point Estimates of Input Variables are
described by Statistical Distributions
Values are drawn at random from each
Input Distribution
Similarly for Probability of Risks and
Opportunities Occurring (on or off)
Random input values are aggregated
Process is repeated thousands of times
Output Distribution is based on the
frequency that particular aggregated
output values occur
Baseline Baseline Risk
S
Repeat … Repeat … Repeat
on off

x
xx
x
Monte Carlo Simulation – Caveat Augur I
Monte Carlo Simulation inherently will narrow the range of potential outputs
• Realistic Optimistic Output > Sum of the Input Minima
• Realistic Pessimistic Output < Sum of the Input Maxima
• Not all the good things in life occur together, nor do the bad things
However, it is a common mistake to assume that all input variables to a Monte
Carlo Simulation are independent of each other
• This will lead to excessive narrowing of the output model range
• A background correlation of around 20% to 30% between all variables is a more
reasonable starting premise
47
CAVEAT AUGUR I
It is naïve to think that all the cost inputs to a Monte Carlo
Simulation are independent of one another.

x
xx
x
Monte Carlo Simulation – Caveat Augur II
Monte Carlo Simulation is an elegant and relatively easy tool to use
It is very good for modelling the net uncertainty in a system of known variables
However, there is often too much reliance placed on its output for assessing
confidence levels around Risk, Opportunity and Uncertainty combined …
… even if all the input assumptions are considered to be a reasonable reflection
of reality … including those somewhat subjective ‘probabilities of occurrence’
It’s nothing to do with the theory, or the accuracy of the mathematics involved,
… it’s simply because there is usually something fundamentally missing!
48
CAVEAT AUGUR II
Monte Carlo Simulation of Risk, Opportunity and Uncertainty is
fundamentally Optimistically Biased - it understates reality!

x
xx
x
Risk, Opportunity & Uncertainty – Mind the Gap
Likelihood of Occurrence
Will Occur May or May Not Occur
MaturityofTaskDefinitionand
Performance
Unknown
orPoorlyDefined
Unknown Knowns
We know we have to do the task but
its exact scope is not clear,
… or we know the task but we do
not know how well we will perform it
Unknown Unknowns
These are those genuine Risks and
Opportunities that we have not
considered because they haven’t
occurred to us
Known
orWellDefined
Known Knowns
We know we have to do the task
and we are clear of the requirements
and understand our likely
performance.
Known Unknowns
We have identified tasks that may or
may not need to be carried out.
These are our defined Risks and
Opportunities.
49
Baseline Tasks
– as they will
occur
Risks or Opportunities
– as they may or
may not occur

x
xx
x
Performance
Unknown
orPoorlyDefined
Unknown Knowns
We know we have to do the task but
its exact scope is not clear,
… or we know the task but we do
not know how well we will perform it
Unknown Unknowns
These are those genuine Risks and
Opportunities that we have not
considered because they haven’t
occurred to us
Known
orWellDefined
Known Knowns
We know we have to do the task
and we are clear of the requirements
and understand our likely
performance.
Known Unknowns
We have identified tasks that may or
may not need to be carried out.
These are our defined Risks and
Opportunities.
50
“There are known knowns. These are things
we know that we know. There are known
unknowns. That is to say, there are things
that we now know we don’t know. But there
are also unknown unknowns. These are
things we do not know we don’t know.”
Donald Rumsfeld
United States Secretary of Defense
DoD news briefing
12 February 2002
“To know that we know what we
know, and that we do not know what
we do not know, that is true
knowledge.”
Confucius
Chinese Philosopher
551–479 BC

x
xx
x
Performance
Unknown
orPoorlyDefined
Immature
Baseline Tasks
Expected
Performance
Undefined Risks
(Unknown unknowns)
Known
orWellDefined
Defined
Baseline Tasks
Risk and
Opportunity Register
Missing from
Monte Carlo
Analysis
51
Baseline Most
Likely Estimate
Bottom-up
Uncertainty
Assessment
Bottom-up Risk
& Opportunity
Assessment

x
xx
x
Risk Opportunity & Uncertainty: Top-down Approach
52
We can test or counter this bottom-up optimism bias with a generally more
pessimistic …
Top-down Approach to ROU Estimating, including:
Assuming that Schedule Slippage Risk is indicative of Cost Risk increase
Flexing Uplift Factors used on non-schedule related risks (e.g. Escalation)
Applying a percentage uplift factor to represent the Unmitigated Risk Exposure:
• Based on the level of unmitigated risk on a previous similar project (Analogy)
• Based on the average level of unmitigated risk on a number of previous
similar projects (Parametric)
• By asking a Subject Matter Expert on the likely risk exposure (Trusted
Source)
“Time is money!”

x
xx
x
Estimate Validation and Challenge
Recommended Practice
Estimate Maturity Assessments
53
TRACE
ADORE
Validated
Estimate
x
xx
x
Customer
Requirement
& Estimate
Initiation
Baseline
Estimate
Creation
Estimate
Validation,
Challenge
&
Clearance
Project
Realisation,
Performance
Monitoring &
Forecasting
Feedback
Risk
Opportunity
&
Uncertainty
Evaluation
Stakeholder
Agreements
(ADORE
or MDAL)
Pricing,
Negotiation
& Customer
Acceptance
(Data and Context)
Feedback
10

x
xx
x
Estimate Validation and Challenge
It is considered to be good practice for all estimates to undergo an
Independent Validation and Challenge:
• Independence: Someone who has not been directly involved in the
compilation of the estimate
• Validation: Reviews that the estimate is compatible with the ADORE
planning statements, and TRACEable to the detailed
Basis of Estimate (BoE)
Reviews whether the data values used are appropriate and
defendable
• Challenge: Where the Validation fails or cannot be completed due to
omissions, this step identifies remedial actions required
For major Estimates, it is not uncommon for these reviews to be conducted by a
Red Team of senior personnel in the function
54

x
xx
x
Estimate Maturity Assessment (EMA)
An EMA provides a “Health
Warning” on the robustness
of an Estimate by reviewing
the Basis of Estimate rather
than the estimate value
It is driven by what has been
used, how and why
The lower the EMA Rating,
the greater the Uncertainty
should be in the Baseline
Estimate
This is the BAE Systems
version
Source: Smith, E (2013) “Estimate Maturity Assessments”,
Association of Cost Engineers Conference, BAE Systems,
London
55
EMA
Level
Estimate Based on …
9
Precise definition with recorded costs of the exact
same nature to the Estimate required
8
Precise definition with recorded costs for a well-defined
similar task to the Estimate required
7
Precise definition with validated metrics for a similar
task to the Estimate required
6
Good definition with metrics for a defined task similar to
the Estimate required
5
Good definition with historical information comparison
for a defined task similar to the Estimate required
4
Defined scope with good historical information
comparison to the Estimate required
3
Defined scope with poor historical data comparison to
the Estimate required
2
Poorly defined scope with poor historical data
1
Poorly defined scope with no historical data

x
xx
x
Summary
Reminder: What have we covered?
56

x
xx
x
Let’s talk about the Principles and Practice of Estimating
1. Is it possible to have a robust estimating process that is also flexible? Why
does it need to be flexible?
2. Who should be involved in the estimating process? Is it just the “number
jockeys”, and if not, who else gets to share in the blame?
3. Does a robust estimating process mean a more accurate estimate, or just an
estimate that is more precisely wrong? Isn’t an estimate only as good as the
last set of assumptions?
4. What’s the difference between an Estimating Method, Approach and
Technique, and how many different ones of each are there?
5. What is meant by a 3-Point Estimate; and isn’t that just avoiding the
question?
6. How do we deal with Risks and Opportunities? What is good practice, and
what is questionable practice?
57

x
xx
x
What does ‘good’ look like in Cost Estimating & Forecasting?
1. A good Estimating Process should pass the CLIFF Test (Closed Loop – Iterative –
Flexible – Functional) that is professionally planned and managed
2. The Estimating Process is more than crunching numbers; it is a business-wide
process requiring the involvement or participation of all relevant stakeholders
3. Estimates need to be intrinsically linked to a Contextual Framework that defines its
Scope through ADORE statements, with an appropriate maturity health check (EMA)
4. Use more than one Approach, Method and Technique to create and validate an
Estimate or Forecast:
• There are 3 APT methods for up to 3 approaches, but a plethora of techniques
• Document the Basis of Estimate so that it meets the objectives of TRACEability
5. A 3-Point Estimate quantifies the inherent uncertainty in an estimate, bounding the
Most Likely Value by an Optimistic (lower bound) and Pessimistic (upper bound) value
6. Use a Top-down and Bottom-up approach in combination to evaluate Risk
Opportunity & Uncertainty as a single system of partially correlated variables
58

x
xx
x
Cost Estimating and Forecasting
What does ‘good’ look like?
Thank you for listening and engaging
Any questions?
59
EST.i.MAT-A: Promoting TRACEability in Estimating
Straight Home
Home Straight

Estimating what does good look like

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Similar to Estimating what does good look like

Similar to Estimating what does good look like (20)

More from Association for Project Management

More from Association for Project Management (20)

Recently uploaded

Recently uploaded (12)

Estimating what does good look like