This document discusses approaches for dealing with uncertainty in demand modelling and forecasting. It identifies four main sources of forecasting error: errors in the demand model, limitations of base year data, errors in projecting future year data, and scenario uncertainty. It presents techniques for addressing forecasting uncertainty, including output deconstruction to assess sensitivity to model parameters, Monte Carlo simulation to account for variability in key inputs, and scenario planning to evaluate outcomes under different assumptions. The document emphasizes that long-term forecasting cannot predict with accuracy and advocates focusing on flexibility and evaluating expected outcomes rather than single forecasts.

1. DEALING WITH UNCERTAINTY IN DEMAND
MODELLING AND FORECASTING
Luis Willumsen

2. MODELOUTPUTS
2
A case of theory-induced blindness?

3. LITIGATION
3
USA

4. CONTENTS
1. How accurate can we be?
2. Sources of forecasting error: Model, Base and Future Year
Data, Black Swans
3. Precision and accuracy
4. Dealing with forecasting uncertainty
1. Output de-construction
2. Monte Carlo Risk Analysis
3. Scenario Planning
5. Forecasting Accuracy?
6. Recommendations
4

5. HOW ACCURATE CAN WE BE?
5

6. ASSUMPTIONS
 Three uses of modelling
 To better understand travel behaviour
 To test alternatives policies and interventions and select the
“best one”
 To produce forecasts of future traffic, revenue, costs, benefits
 For the last two forecasting accuracy is important
 Evidence suggests we are not very accurate
 Why?
 What can be done about it?
6

7. REALITY IS NOT VERY “ACCURATE”
7

8. YORK
8
Flow variability ~ 10%

9. RECURRENTTRIPS?
9
Lendall Bridge repeated observations of number plates
The other bridges displayed similar patterns
Flow Variability ~ 10-12%, hides this
27-Jun 28-Jun 06-Sep 07-Sep 08-Sep 11-Sep 13-Sep 27-Sep 18-Oct
27-Jun 39 18 18 19 19 14 20 16
28-Jun 37 19 19 16 17 16 20 18
06-Sep 21 24 39 34 32 25 29 23
07-Sep 22 24 39 37 32 25 27 25
08-Sep 23 21 35 38 33 26 27 24
11-Sep 22 21 31 31 31 30 31 24
13-Sep 19 22 28 27 28 34 28 26
27-Sep 24 24 28 26 26 31 25 33
18-Oct 18 22 23 24 23 24 23 32
Lendall Bridge numberplate observations between 8:00 and 9:00 in year 2000
Clegg, R (2005) An empirical study of day-to-day variability in driver travel behaviour
Percentage matches between days (corrected for errors)

10. VARIABILITY
 Bonsall, P., Montgomery, F. and Jones, C. (1984) in Leeds,
found a maximum of 45% matches, at any time, the
following day.
 Cherrett & McDonald (2002) in Southampton, found that
between 25% and 49% of vehicles are observed on a
subsequent day on particular streets.
 Del Mistro & Behrens (2008) in Cape Town, found that
between 31% and 46% were observed on subsequent days
(more on residential, less in arterial streets).
 In urban toll roads we find that 25% to 30% only are
frequent users.
 But in surveys most drivers (around 90%) state they travel
the same route every day.
10

11. SURVEYS
11
Universe
of trips
Representative
sample of trips
Trips from O to D
Expanded to represent an
average universe of trips
O-D
5-2
1-5
1-3
1-3
8-3
9-1
4-7
1-5
7-3 6-5
2-8

12. SAMPLING
12
Universe
of trips
Sample of trips
Recurrent trips
Non-recurrent trips
Observed trips
Expanded to represent an
average universe of trips2-8
2-8
1-8
2-8
4-4
5-6
7-3
1-9
6-71-4
2-8

13. FLAWOFAVERAGES
The belief that assessing conditions under average inputs
produces average outcomes.
13* Savage, S. (2009) The Flaw of Averages. Wiley, New York
Average
Inputs
•Future Population,
Economy, etc.
Model
•Assumptions
•Functional forms
•Parameters
Average
Outcomes
•Average results:
flows, costs, access
•CBA
This is only true if the relationships (in the model) are LINEAR
… they are not.

14. NEWSENSORS
 An abundance of sensors is now providing information on
recurrent and non-recurrent trip making:
 Navigation GPS
 Bluetooth data
 Smart cards
 Anonymised Mobile Phone data
14

15. ERRORSINFRECASTING

16. NZGROSSDOMESTICPRODUCT

17. SOURCES OF FORECASTING ERRORS
17

18. FORECASTINGERRORS
• Errors and limitations of the MODEL
• Limitations of the BASE YEAR DATA
• Errors in FUTURE YEAR DATA
• Scenario Uncertainty (Black and spotted swans)

19. UNCERTAINTY1
Uncertainty due to
model quality
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
19

20. UNCERTAINTY1
Uncertainty due to
model quality
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
 Human errors (coding, scripts, theoretical)
 Simplifications: trips, tours, activities, real capacity constraints, Static or
DTA, ignore some periods, assume constant parameters, etc.
 Over specifying the model (too much unwarranted detail)
 Cross section vs. time series data
 Mis-representations of individual behaviour (Homo Economicus assumption)
 Limited rationality
 Limited capacity to compare alternatives
 Imperfect information (knowledge of alternatives)
 Valuing loses more than equivalent gains
 Ignoring lags in responses
 Low sensitivity to small changes
 As time passes the model is likely to be more erroneous
20

21. UNCERTAINTY2
Uncertainty due to
model quality
Base Year
Data
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
21

22. UNCERTAINTY2
Uncertainty due to
model quality
Base Year
Data
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
 Cross – sectional data only
 Small sample (2-3% for Home Interviews, larger for RSI but only
some locations)
 Respondent fatigue
 Imperfect blend of single day observations, sometimes even from
different years
 Poor data for Attractions, freight, deliveries, taxis, motorcycles and
bikes
 Stated Preference surveys, reliable enough?
 Traffic count errors
 Transcription, correction and processing errors
 As time passes, the errors in the base year data are likely to be
less important 22

23. UNCERTAINTY3
Uncertainty due to
model quality
Base Year
Data
Future data
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
23

24. UNCERTAINTY3
Uncertainty due to
model quality
Base Year
Data
Future data
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
 If base year data is imperfect, future year “data” is even more so
 The greater the granularity of future year data the greater its inaccuracy
 Witness GDP forecasting, even for next year
 Future population synthesis is also fraught with uncertainties
 As time passes, the errors in the future year data are likely to grow,
significantly
24

25. UNCERTAINTY4
Uncertainty due to
model quality
Base Year
Data
Future data
Scenario Uncertainty
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
25
Sometimes referred to as “deep uncertainty”

26. UNCERTAINTY4
Uncertainty due to
model quality
Base Year
Data
Future data
Scenario Uncertainty
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
NoonalUncertaintyIndex
Years a er forecast
No onal sources of uncertainty in forecas ng
 Known unknowns:
 Energy prices
 Disruptive technologies: Self Driving Vehicles, Internet of Things
 Changes in human values and behaviour : “peak car”
 Unforeseen changes in activities: video rentals, weekly shopping,
working away from office
 Widespread electronic payment for transport (and other uses)
 Unknown unknowns
26

27. TRENDSANDTHEIRSTABILITY
27
Peak car NZ
Autonomous Vehicles

28. DEALING WITH FORECASTING UNCERTAINTY
28

29. ACCURACYANDPRECISION
29

30. ACCURACY&PRECISION
• Sometimes pursuing precision may lead to inaccuracy
• It is better to be approximately right than precisely wrong
ONE SHOT is
not enough!
then
30
What shots?

31. APPROACHES
 The model: Output de-
construction
 Future inputs: Monte
Carlo Simulation
 Scenarios: Scenario
Planning
0.85
0.9
0.95
1
1.05
1.1
1.15
2003 2005 2007 2009 2011 2013 2015
Year
RevenueFactor
2015 Future A
Future B
Future C
31

32. PUBLICTRANSPORTDEMANDDE-CONSTRUCTION
32

33. STRENGTHSANDLIMITATIONS
 Requires care in setting up the model so that outputs can be
de-constructed
 Deals with different degrees of confidence in the
behavioural responses incorporated in the model
 It is possible to allocate probabilities to different levels of
contribution to final demand
 Does not deal with errors in the DATA (present and future);
but one layer may depend on a promised but uncertain
action, for example re-structuring of the bus network
 Does not account for changes in behaviour (tastes) or
external factors (black swans)
33

34. STOCHASTICRISKANALYSIS
 Preferred by Financial Institutions
 Generally based around Base Case and a desired outcome or
KPI (Traffic, Revenue, etc.)
 Take 2-5 key variables : GDP, SVT, etc.
 ..and look into their historical variability (standard deviation
 )
 The model is used to track variability in KPI resulting from
variability in key inputs, usually via an elasticity model in
Excel
 FutureKPI = KPI_Factor * Base Case KPI
 KPI_Factor = 1 indicates Base Case or no change
 Also presented as the level of KPI (revenue) that is likely to
be exceeded 90 or 95% of the time (P90 and P95)
34

35. TYPICALOUTPUT
0.85
0.9
0.95
1
1.05
1.1
1.15
2003 2005 2007 2009 2011 2013 2015
Year
RevenueFactor
 GDP variation σ: 0.5%
 SVT variation σ : 2.5%*mean VST
35

36. FORECASTSCASE2
'Four Roads- Case 2 WRe P90 sd 1.5
2
7
12
17
22
2008 2010 2012 2014 2016 2018 2020 2022 2024 2026 2028 2030 2032 2034 2036
Mx$Billions
36

37.  Did not protect Financial Institutions nor Rating Agencies
from the mistakes leading to the Global Financial Crisis
 Deals with errors in Future DATA
 But not errors in Base Year Data or Model
 Ignores changes in behaviour and scenarios (black swans)
 Useful as a general way to process uncertainty
37

38. SCENARIOS
 And check which project performs better under different
conditions
 The technique for scenario building are simple and useful
 Approach already incorporated into DfT Road Traffic
Forecasts 2015
 But needs localising and to be made Project specific
 But the results may not be what decision makings would
prefer.
2015 Future A
Future B
Future C
38

39. SCENARIOEXAMPLE
Scenarios
1. Decide drivers for
change/assumptions
2. Bring drivers together into
a viable framework
3. Produce 7-9 initial mini-
scenarios
4. Reduce to 2-4 scenarios
5. Draft the scenarios
6. Identify the issues arising
and how they affect
decisions
NZ Futures Transport Demand
1. Fuel prices; self-driving
cars; GDP growth; energy
costs, digital realm….
39

40. SCANARIOPLANNING
40

41. INTEGRATION
 It is possible to integrate these 3 views of uncertainty
 Starting with Scenario Planning: assign probabilities to each
 Then use the contributions from each sub-model and assign
probabilities to each level
 Finally combine all in a Monte Carlo Risk Analysis
41
Base
Scenario
1
Scenario
2
Scenario
3
Sub
model A
Sub
model B
Sub
model C
Monte
Carlo

42. INPRACTICE
 Only a simplified version of the scheme would be used
 But the issue of uncertainty has some real implications:
 Only an Expected Net Present Value (ENPV) can ever
be produced
 We must develop methods to deal with uncertainty in
decision making
 Flexibility in the design of our projects and plans
acquires a greater value
 Real Options (in contrast with Financial Options) is a
good framework to think about flexibility
42

43. CONCLUSIONS
43

44. FORECASTING
•Accuracy in medium to long term forecasting is an impossible target
•If we cannot (should not) claim to be able to forecast accurately,
what should we aim for?
“Economists (modellers) have allowed themselves to walk into a trap
where we say we can forecast, but no serious economist thinks we
can. You don't expect dentists to be able to forecast how many teeth
you'll have when you're 80. You expect them to give good advice and
fix problems.”
Tim Harford
44

45. FORECASTING
•Acknowledge uncertainty and risk from the outset: identify
sources of risk, estimate their importance and focus on dealing with
them
•Aim for conditional accuracy for short term (1-5 years) forecasts
•Accuracy is not the right objective for longer term forecasting (6-
30 years)
•Good advice on decision making is a much better objective
•Prefer schemes and plans that offer flexibility and adapt them as
conditions change
•Careful use of existing techniques, even with the limitations shown,
is a reasonable approach. But, support forecasts from different
complementary perspectives and good judgment
•Where appropriate undertake risk analysis
•Test the impact of interventions under different future scenarios:
ONE SHOT IS NOT ENOUGH
45

46. THANK YOU