How to Forecast Accurately in Uncertain
Times
A guide to the key issues that reduce the accuracy of
forecasts and how Finance functions can address them
Date – December 2008
Rational Strategies
Scientifically better decisionsTM

MANAGEMENT SUMMARY
In today’s uncertain economic conditions the ability to
forecast accurately is vitally important
• A recent survey by CFO magazine revealed that the ability to produce accurate
forecasts was the second greatest internally focussed concern of CFOs:
– CFO magazine’s findings can be found at http://www.cfo.com/article.cfm/12492577?f=related
• While the uncertain conditions do make it more difficult to forecast accurately,
it is nevertheless possible to improve on their accuracy:
– And the key to doing so is to understand the key issues that reduce the accuracy of forecasts, the majority of
which are actually present regardless of whether conditions are steady, or uncertain.
• To help finance functions do just that, this document outlines the key issues
that impact on forecast accuracies and suggests how to address them:
– It’s not intended to be a “how to” guide to forecasting, as that depends critically on the particular type of forecast a
finance function is trying to produce, the approaches they’re currently using, the availability of data etc. Rather
this document provides a starting point for finance functions, suggesting areas to investigate.
• This document is structured into the following sections:
– An overview of the key issues reducing forecast accuracies;
– Actions that Finance functions can take to improve the accuracy of existing forecasting approaches;
– An overview of a quantitative forecasting technique suited to dealing with uncertainty.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 2

So, why is it so difficult to forecast accurately?
- The key factors that reduce forecast accuracies

THE KEY FACTORS REDUCING FORECAST ACCURACY
The accuracy of many forecasts are reduced by issues in
the following three key areas...
The Generation • Forecast models, whether financial or economic, are built upon input data and the assumptions
of Forecast made about that data.
• Uncertainties in input data, assumptions that aren’t sufficiently rigorously challenged and human
Inputs nature failings in the ways that we perceive information, all reduce forecast accuracy.
• Many Finance departments rely on spreadsheets to produce financial forecasts and other outputs
- unfortunately, spreadsheet based models (of any type) are inherently error-prone.
Forecasting • Additionally most finance professionals haven’t received appropriate spreadsheet training, and
Model Issues spreadsheet design guidelines are rarely used and applied.
• Many spreadsheet based models are overly-complex, which can reduce forecast accuracy,
particularly when there’s uncertainties in the input data.
The Way • When forecast models produce unpalatable results an occasional reaction is to “massage” the
results, or worse, to request that analysts rework their models to produce “acceptable” results.
Forecast • If the forecast models are valid then either their forecasts should be regarded as valid, or the
Outputs Are model abandoned completely, in favour of qualitative forecasting approaches.
Used • Massaging the message or reworking models simply reduces forecasting accuracy.
We’ll briefly examine each of these areas over the next few pages, to gain an insight into the
actions Finance functions can take to improve the accuracy of their forecasts.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 4

THE KEY FACTORS REDUCING FORECAST ACCURACY – THE GENERATION OF FORECAST INPUTS
Starting with forecast inputs, there are three major issue
areas that can significantly reduce forecast accuracies
The inputs to forecasts are frequently undermined by …and depending on the situation, these can
the following issues… significantly reduce forecast accuracies.
Uncertainties, • Inevitably there will be shortcomings in input business
data , particularly data gaps and uncertainties:
Ambiguities and – For example, in today’s volatile business environment there will
Gaps in Input be significant uncertainties around any data based on
projections of historic performance (eg. sales, order volumes).
Data
– There will also be outright gaps in the input data, for example
when entering new markets, or launching new products.
– As outlined later in this paper, rigorous handling of these data
shortcomings will help ensure high forecast accuracy.
• While much effort is expended obtaining input data, the
The Generation Insufficiently same attention is rarely paid to key input assumptions:
of Forecast Rigorous – All forecasts are built on key assumptions and the validity and
Inputs Assumptions accuracy of any forecast can only be assessed when presented
alongside those assumptions. For example, a forecast based on
growing product sales by 10% would have a very different
validity than another assuming growth of 2%.
– Identifying and challenging key assumptions therefore needs to
form a core part of the forecasting process.
• Behavioural economics reveals that human beings are
Human Nature rather less rational than we might think:
Failings – Particularly with respect to how we perceive data and
information in the light of what we’d like to believe.
– As explained on the following page these human failings have
important implications for any attempts at forecasting.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 5

THE KEY FACTORS REDUCING FORECAST ACCURACY – THE GENERATION OF FORECAST INPUTS
Similarly to the three wise monkeys, human beings don’t
see, seek, or use information in the way that we think we do
Behavioural economics research has recently revealed shortcomings in the
way that human beings interact with information, in particular: 1:
• That we fail to “see” key information, particularly when we’re busy:
– For example, Merck disclosed to regulators in 2000 major problems with the drug Vioxx. In spite of this, an additional 1 million
prescriptions were written by doctors between then and when the drug was withdrawn from the market in 2004, contributing to a n
estimated 25,000 heart attacks and strokes.
– Doctors, like corporate decision makers, are busy, imperfect decision makers and overlooked these critical findings.
• That we fail to “seek” information that contradicts our viewpoints:
– Frequently decision makers favour a particular outcome and so fail to seek out contradictory information.
– The most devastating recent example of this led to the Spaceship Challenger’s destruction. The night before the launch NASA
executives argued about whether that night’s low temperatures would affect the O-rings but concluded that it wouldn’t. Subsequent
investigations into the disaster revealed that had they checked the O-ring data on the prior 24 launches they would have discovered that
the probability of a disaster exceeded 99%.
• That we fail to “use” information:
– For example, information about competitors. Research at Carnegie Mellon University demonstrated that decision makers focus on how
well they can perform a task but tend to ignore how well the competition can do the same task. As a result decision makers have a
tendency to enter product domains offering easy access, and to enter more difficult product domains too infrequently.
– And as all the competitors tend to behave in the same way then competition is increased in the “easy markets” and decreased in the
more difficult markets.
The upshot of which is that even when we think we’re rationally building the most accurate
forecast model we can, we may be overlooking key data items, or moulding the data to fit our
viewpoints, or simply not making appropriate use of the information we’ve got.
1. Source: Adapted from the Harvard Business Review, Decision Making, January 2006.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 6

THE KEY FACTORS REDUCING FORECAST ACCURACY – FORECASTING MODEL ISSUES
Having considered input related issues, we now need to
consider issues relating to forecasting models themselves
Forecasting model issues fall into two major categories, those relating to the spreadsheet technologies
underpinning most forecasts and those relating to the complexities of the models used:
• Spreadsheets are an inherently error-prone technology:
– Data can be accidentally over-written or misreferenced, cell formulas can be
incorrectly written or copied and can be hard to read and audit;
– Hard coded values such as tax rates can be buried deep within sheets and
missed when updates are needed.
• They’re poorly suited to dealing with uncertainties and ambiguity:
Spreadsheet – Spreadsheets force the entry of single values into cells, making it
complicated to allow for uncertainties and ambiguities in data; the generation
Related Issues of multiple scenarios to allow for these ambiguities can be time consuming .
• These issues are compounded by the fact that most financial
practitioners lack appropriate spreadsheet training:
Forecasting – Frequently resulting in poorly structured, hard to audit models, where the
logic, assumptions and calculation approaches are hard to follow and difficult
Model Issues to subsequently change without introducing errors.
• And further compounded by the typical lack of in-house
spreadsheet design guidelines and standards:
Overly Complex – With individual financial practitioners developing their own personal
Models approaches to developing spreadsheet models, increasing the risk of errors
and reducing their auditability.
• The upshot of which is that there are pervasive levels of errors in
spreadsheets:
– For example, separate surveys carried out by KPMG and Coopers and
Lybrand in the late 90’s found that 91% of corporate finance spreadsheets
contained significant errors.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 7

THE KEY FACTORS REDUCING FORECAST ACCURACY – FORECASTING MODEL ISSUES
Having considered input related issues, we now need to consider
issues relating to forecasting models themselves (continued)
Forecasting model issues fall into two major categories, those relating to the spreadsheet technologies
underpinning most forecasts and those relating to the complexities of the models used:
• In most cases, increasing the complexity of a forecast model will
actually lead to a decrease in the accuracy of its forecasts. The
reasons for this include:
• The uncertainties in the input data:
– As noted earlier, there are always uncertainties in input data. As the number
of variables in a forecast model rises, these uncertainties will combine in
Spreadsheet unpredictable ways, potentially significantly degrading forecast accuracies.
Related Issues • Overlooked correlations between input data items:
– The correlations between input data items needs to be factored into forecast
models and as the number of data items rises, correlations are likely to
overlooked.
Forecasting – For example if a sales forecast model includes items relating to individual
Model Issues product sales and to product commissions paid to sales people, then the
correlations between those two items needs to be factored in, as changes to
commission structures will clearly have a correlated impact on product sales.
Overly Complex • The reduced clarity as to what the major drivers are underlying the
Models forecasts:
– As model complexities increase, it becomes increasingly difficult to determine
which elements of the model are driving it’s key forecast characteristics,
therefore also making it more difficult to refine the model.
– In addition, when it’s not possible to determine the key drivers of forecast
outputs, then decision makers are deprived of key levers that they can use to
drive the business’ performance.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 8

THE KEY FACTORS REDUCING FORECAST ACCURACY – THE WAY FORECAST OUTPUTS ARE USED
Finally, the value and accuracy of forecasts can be
significantly reduced by the way in which they’re used
This is a complex issue and it’s sufficient to simply note that:
• Overt or subtle pressure placed on analysts can reduce the accuracy of
forecast models:
– Overt pressure simply taking the form of senior personnel or colleagues not believing the outputs of a
model and telling analysts to rework them, to deliver more “acceptable” results.
– More subtle pressures include peer expectations. For example during the housing bubble in the US
and the UK between 2005 and 2007, few housing analysts were comfortable forecasting house price
falls, in spite of the fundamental indicators pointing in that direction (credit over-stretch, surplus
housing stocks etc).
• Forecast models have, by their nature, to deal with probable outcomes, as
the future is never 100% predictable:
– If this aspect of a forecast model is not understood by the output users, then erroneous decisions may
be made.
– For example if a model forecasts that sales will be, say, £10m in a given region for Q1 2010 and a firm
only has capacity to deal with £7.5m of sales, then it may seem reasonable to invest £500k in sales
capacity to realise that extra £2.5m in sales. Reasonable that is, unless the forecast is only made with
a 60% certainty level, in which case significantly different management decisions might be taken.
So, having identified the major issues that reduce forecast accuracies, let’s now consider the
actions we can take to improve the accuracy of our forecasting approaches.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 9

What can we do to improve the accuracy of our existing
forecasting approaches?

IMPROVING THE ACCURACY OF EXISTING FORECASTING APPROACHES
Working through each identified issue in turn, there are actions
we can take to improve the accuracy of existing approaches
Issues Reducing Actions Finance Can Take To
Additional Comments
Forecast Accuracy Address These
• Spreadsheet models are ill-suited to
• Use expert input and /or benchmark dealing with uncertainties, so either move
Inputs – data data to close gaps and remove to qualitative forecast approaches, or if
1 uncertainties,
ambiguities and gaps
ambiguities, logging these as
assumptions but importantly investigating
quantitative outputs are needed then
consider using Monte Carlo techniques
their impacts on the forecasts. to deal with the uncertainty (see page
14).
• Identify key assumptions, review and • These issues are best tackled together,
challenge them in light of current and by altering existing forecasting
future circumstances. processes, critically incorporating key
Inputs – Insufficiently
2 rigorous assumptions
• Investigate their impacts on forecasts. challenge stages as the inputs and
• Provide key assumptions alongside assumptions are being reviewed and
forecasts, so that forecast validity can be using rigorous group review processes to
assessed. try and avoid group think and human
failings.
•Remedying this issue is exceedingly • Edward De Bono’s works, particularly
difficult as it applies both to individuals his “Thinking Course” provide tools and
and groups (group think issues).
3 Inputs – Human failings
• Rigorous challenge and review
techniques that can be used to ensure
rigour in the inputs.
processes are the most pragmatic http://www.amazon.com/Bonos-Thinking-
approach to trying to address these. Course-Revised/dp/0816031789
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 11

IMPROVING THE ACCURACY OF EXISTING FORECASTING APPROACHES
Working through each identified issue in turn, there are actions we
can take to improve the accuracy of existing approaches (continued)
Issues Reducing Actions Finance Can Take To
Additional Comments
Forecast Accuracy Address These
•Introduce in-house spreadsheet design • Best practice spreadsheet design
guidelines and provide training in guidance can be found throughout the
Forecasting Model spreadsheet modelling and design. web. A useful starting point is the
4 Issues – Spreadsheet
related issues
• Have IT convert key forecasting
spreadsheets into VBA based
Institute of Chartered Accountants guide
to spreadsheet best practice, which can
spreadsheets – they’ll run faster and with be found at:
fewer errors. http://www.eusprig.org/smbp.pdf
• Rebuild simpler forecast models from
the ground up, focussing on business • Partial model rebuilds and refocusing
Forecasting Model drivers. are tricky and there’s no guarantee that
5 Issues – Overly
complex models
• Where time or resources prevent this,
focus on identifying key business drivers
the resulting models will be more
accurate than the original models.
in existing models and making those
model elements as robust as possible.
• Again this is a difficult issue to address
as it’s impacted on by the corporate
decision making culture.
The Way Forecast
6 Outputs are Used
• Presenting forecasts along with their
assumptions and their likely probabilities
are pragmatic approaches to dealing with
this problem.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 12

And if I need to produce the most accurate possible
quantitative forecasts in today’s uncertain environment?
- Which forecasting techniques are the most accurate in
these conditions?

ACCURATE QUANTITATIVE FORECASTING TECHNIQUES FOR VOLATILE CONDITIONS
From a Finance perspective, the single most difficult
forecasting issue to deal with is that of uncertainty in inputs
Conventional forecasting and budgeting ...unfortunately in volatile conditions, many
approaches require certainty in inputs... inputs are likely to be uncertain
• For example inventory volumes will depend on sales
volumes and these are likely to lie in an estimated range:
– For example, perhaps lying between £10m and £15m, with a best
guess most likely out turn of £12m.
– Conventional forecast approaches like the one represented on the
left require a single best guess estimate to be entered for this input,
say £12m, which if sales come in at the bottom of the range, will
have overstated sales by £2m, or if sales come in at the top of the
range, will have underestimated sales by £3m, both being poor
outcomes.
• While uncertainties in a single input can be dealt with by
generating multiple forecasts, this is impractical for higher
numbers of inputs:
– Forecasters sometimes allow for the uncertainties in a range of
inputs by creating multiple forecasts (eg. high, medium, low) that try
to encompass all the possible outcomes of all the uncertainties.
– Given the complexities in the way that uncertainties (or probabilities)
combine and aggregate up, then the likelihood that such aggregate
forecasts will effectively capture the total range of outcomes, and
most likely outcomes, from such combined uncertainties, is almost
vanishingly small.
– However, generating sufficient numbers of forecasts to allow for the
range of possible outcomes in each of the input items is impractical
when dealing with more than 4 or 5 uncertain inputs (as, for
example, generating 3 forecasts for each of 3 uncertain inputs would
require the generation of 27 separate scenarios to capture all the
possible outcomes).
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 14

ACCURATE QUANTITATIVE FORECASTING TECHNIQUES FOR VOLATILE CONDITIONS
The most accurate way of dealing with these uncertainties is
to use “Monte Carlo” modelling techniques
• Quantitative modelling techniques that can be incorporated
into spreadsheet models:
What They Are – Importantly allowing input information to be entered as ranges
(probabilities) and using these to determine the range of likely outcomes
and the probabilities with which they’ll occur, for example:
Monte Carlo Where and • They’re well established, widely used techniques:
Modelling How They’re – Being used in such widely disparate fields as finance, project
management, energy, manufacturing, engineering, research and
Techniques Used development, insurance, oil and gas, transportation, and the
environment.
• There are many benefits to using such approaches when
Benefits Of developing forecasts:
Using Them – The ability to effectively (in time and resources) identify the most
probable outcomes, along with the full range of outcomes and their
probabilities of occurring.
– The ability to determine the key factors that drive the forecast outcomes
and hence provide decision makers with the levers with which to
influence their business outcomes.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 15

ACCURATE QUANTITATIVE FORECASTING TECHNIQUES FOR VOLATILE CONDITIONS
While commercial packages are available that simplify
Monte Carlo modelling, care is still needed!
• Two major vendors of spreadsheet (Excel) compatible Monte Carlo
modelling packages are Palisade Corporation and Oracle:
– Palisade Corporation sells the DecisionSuite stable of products, one of which, @Risk, is a Monte Carlo
simulation package. Information on Palisade’s products can be found at http://www.palisade.com/
– Oracle sells a package of similar capability, Crystal Ball, details of which can be found at
http://www.oracle.com/crystalball/index.html
• While these and similar packages can help produce more accurate
forecasts, they do introduce another issue that needs to be approached with
care – that of model risk:
– Quite simply these packages allow relatively complex forecasting and financial models to be built with
relative ease and in inexperienced hands, seemingly realistic forecasts can be produced that are in
fact wildly inaccurate;
– For example, the uncertainties in the input data need to be specified with care and the pre-defined
distributions supplied with such packages make it easy for users to select inappropriate and unrealistic
representations of those uncertainties;
– Similarly, correlations between input parameters need to be handled with care if they’re not to
undermine the model’s forecast accuracy.
And that brings our overview of forecasting in uncertain times to an end. If you’d like further
information on any of the topics covered in this paper then our contact details are noted on the
following pages.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 16

About Rational Strategies

ABOUT RATIONAL STRATEGIES
Rational Strategies helps Institutions use modelling
approaches to make better business decisions
• Rational Strategies brings academic rigour combined with For further information on
real world pragmatism: any of the topics covered in
– Dr Giulio Galassi, Rational Strategies’ founder, is a leading expert in the use of this paper, contact:
modelling and simulation techniques to help institutions make better business
decisions.
Dr Giulio Galassi
– Having completed a doctorate in Theoretical Physics, where he gained world Rational Strategies
class experience in the use of modelling and simulation techniques, Dr Galassi
entered the Banking Industry, initially putting his modelling and simulation
Email:
expertise to work on the Royal Bank of Scotland’s dealing floor. ggalassi@RationalStrategies.co.uk
– Dr Galassi’s career subsequently progressed through increasingly senior roles in
major Blue Chip companies and leading consultancies, including stints leading
Abbey’s internal consulting team and heading up CSC’s Financial Services
consulting group.
– In late 2007 Dr Galassi set up Rational Strategies to focus on helping clients
make better business decisions using the modelling and simulation approaches
in which he’s a leading expert.
– With over 15 years experience in industry, Dr Galassi has an excellent
understanding both of the challenges facing senior industry executives and the
difficulties facing them in making major decisions, when information is lacking,
incomplete, there’s a high degree of complexity or changes in market conditions
throw up uncertainties.
– In spite of his deep academic background Dr Galassi is a strong believer in using
pragmatic approaches to help clients make their decisions and resolve their
issues. Having been an academic in the past he’s able to avoid the trap of
“analysis paralysis” that hits so many institutions, making sure that his clients get
pragmatic, actionable insight from his work with them, rather than analyses that
are overly academic or simply not actionable.
Rational Strategies • Proprietary and Confidential How To Forecast Accurately in Uncertain Times | 18