The document summarizes a presentation given at the NAEC Seminar on November 20, 2013 about OECD forecasts during and after the financial crisis. It discusses three aspects of the project: a quantitative evaluation of forecast errors, structural and financial factors related to errors, and interviews with other organizations. Several key points are made: forecasters did not foresee the downturn in 2008-2009 and overestimated the recovery; errors were largest for 2009 growth and in vulnerable euro area countries; factors like trade openness, foreign bank penetration, and pre-crisis bank capital were correlated with errors. The presentation also notes changes made to forecasting practices, like increased focus on financial data and risk assessments.

1. NAEC Seminar, 20 November 2013
PROJECT A2
OECD FORECASTS
DURING & AFTER THE
FINANCIAL CRISIS
A POST-MORTEM
Speakers:
• Sveinbjorn Blondal, ECO, OECD
• Nigel Pain, ECO, OECD
• Christine Lewis, ECO, OECD

2. Motivation and context
• NAEC: reviewing the crisis and identifying
lessons for future forecasting performance.
Overview
• Regular forecast post-mortems by ECO.
• Three aspects of the project:
– Quantitative evaluation of the forecasts.
– What structural, financial and policy-related factors
are related to the errors?
– Interviews on post-crisis changes with experts from
other international organisations.
2

3. Data set and definitions
• Three sets of forecasts:
The data
– May Economic Outlook for the next year;
– November Economic Outlook for the next year;
– May Economic Outlook for the same year
• Outturn data
– The published data in May after the year being
forecast
• Focus on calendar year GDP growth forecasts
– The project also considered inflation forecasts
and quarterly forecasts for the large economies
3

4. PROPERTIES OF THE
RECENT FORECAST
ERRORS
4

5. Properties of the forecast errors
The downturn was not foreseen
%
Forecasts of GDP growth in 2008
3
%
3
2
2
1
1
0
0
-1
-1
-2
Forecasts of GDP growth in 2009
-2
-3
May-07
Nov-07
May-08
-3
Nov-08 Outturn
May-09 May-08
Nov-08
May-09
Nov-09 Outturn
May-10
• Forecasts of (Q4/Q4) OECD GDP growth in 2008 and
2009 were revised down substantially.
5

6. Properties of the forecast errors
The recovery has been mixed
%
Forecasts of GDP growth in 2010
4
%
4
3
3
2
2
1
Forecasts of GDP growth in 2011
1
0
May-09
Nov-09
May-10
0
Nov-10 Outturn May-10
May-11
Nov-10
May-11
Nov-11 Outturn
May-12
• OECD GDP growth (Q4/Q4) rebounded more quickly
than initially expected in 2010.
• But disappointments resumed in 2011 and 2012
6

7. Properties of the forecast errors
The biggest error in the 18-month ahead
calendar year growth forecasts was 2009.
7

8. Properties of the forecast errors
Errors are typically smaller at shorter
forecast horizons.
Forecast error = outturn less forecast
8

9. Properties of the forecast errors
On average, the errors for the BRIICS are
similar to those for OECD countries.
Forecast error = outturn less forecast
9

10. Properties of the forecast errors
Forecast errors were largest in the
vulnerable euro area countries
Average growth forecast error 2007-12
%pt
0.5
0.0
-0.5
-1.0
-1.5
May current year
Nov forecast for following year
-2.0
May forecast for following year
-2.5
OECD
Euro area
core
Euro area Other OECD
vulnerable
Europe
Rest of
OECD
BRIICS
10

11. Properties of the forecast errors
The pattern of forecast accuracy is
similar, though differences are smaller
%pt
5
4
RMSE of GDP growth forecasts 2007-12
May current year
Nov forecast for following year
May forecast for following year
3
2
1
0
OECD
Euro area
core
Euro area Other OECD
vulnerable
Europe
Rest of
OECD
The root mean squared error (RMSE) could be used to put
confidence intervals around a point forecast.
BRIICS
11

12. EXPLAINING FORECAST
ERRORS
12

13. Global interconnectedness rose substantially
prior to the crisis, with rising global & national
imbalances.
Global trade and financial linkages
Explaining the errors
80
160
60
120
40
80
20
40
0
0
1995
2007
1995
2007
1995
2007
Trade openness
Foreign banks
Financial openness
(LHS, % of GDP)
(LHS, % of total)
(RHS, % of GDP)
Note: Trade openness is the average of all countries; the share of foreign
banks and financial openness are the median OECD country.
13

14. What factors could be correlated with
recent growth forecast errors?
Explaining forecast errors
• International trade and financial openness.
• Banking sector information.
• Economy-wide regulations.
• Pre-crisis imbalances.
• Survey information.
• Fiscal consolidation.
• The euro area crisis.
14

15. Factors correlated with forecast errors
(+/- indicate growth stronger/weaker than expected)
In the downturn (May 2008 for 2008-09)
Relationship
Explaining the errors
Trade openness
-
Foreign bank penetration
-
In the recovery
Relationship
(May 2010 for 2010-11 and May 2011 for 2011-12)
Better capitalised banks pre-crisis
+
Increases in non-performing loans
-
Equity price growth
+
Improved consumer sentiment
+
15

16. The downturn was stronger than projected in
more open economies (negative spillovers)
Cumulative growth forecast errors for 2008-09, made in May 2008
Forecast error (%pt)
Forecast error (%pt)
Explaining the errors
0
-4
-8
-12
0
-4
-8
-12
-16
-16
0
100
200
Trade openness
0
50
100
Foreign banks' assets (% total)
• Forecast error = Outturn - Forecast
• This relationship is similar for average errors over the full period
16

17. There were downside surprises in 2010-11 in
countries with lower pre-crisis bank capital
Growth forecast errors for 2010-11, from May 2010
4
Forecast error (%pt)
Explaining the errors
8
0
-4
-8
8
12
16
20
Bank regulatory capital in 2007 (%)
Bank capital is the capital adequacy of deposit-takers, measured as
a ratio of total regulatory capital to risk-weighted assets.
17

18. And also where the financial system
was weakening
Growth forecast errors for 2011-12, from May 2011
0
Forecast error (%pt)
Explaining the errors
4
-4
-8
-12
-5
0
5
10
15
Change in non-performing loans, 2011-12 (%pt)
18

19. Growth forecast errors over 2007-12 were
larger in more regulated economies
RMSEs of November projections for next year
Explaining the errors
3
Group mean by product market
Group mean by labour market
2
1
0
Least regulated
Middle
Most regulated
Degree of regulation for market indicated (2008)
Indicators are the OECD product market regulation index and the OECD
measure of the strictness of employment protection (for regular workers)
19

20. Explaining forecast errors
Forecast errors, fiscal consolidation and
fiscal multipliers
• IMF: fiscal multipliers under-estimated in
recovery:
– growth weaker than expected in countries with
stronger projected fiscal consolidation.
• Alternatively, actual consolidation could have
been stronger than projected consolidation.
• Growth disappointments also coincided with
the euro area crisis.
• What does the OECD evidence say?
20

21. Explaining the errors
Growth weaker than projected in countries with
more fiscal consolidation, but only in Europe.
21

22. Explaining the errors
Incorrect assumptions about euro crisis and
govt. bond spreads also a source of error at the
same time.
22

23. Explaining forecast errors
Forecast errors, fiscal consolidation and
fiscal multipliers: OECD evidence
• Yes, growth disappointments in countries with
stronger projected consolidation.
• Yes, growth disappointments in countries with
stronger consolidation than projected.
• But only in Europe, and only if Greece is
included.
• The bond spread errors are a more important
source of growth forecast errors
– confirmed by econometric evidence.
23

24. PUTTING THE FORECAST
ERRORS IN CONTEXT
24

25. Putting the errors in context
An historical perspective: OECD growth forecast
errors in 2009 and 2010 are similar to the early 1970s
25

26. Putting the errors in context
Recent errors were larger than in the “Great
Moderation” but smaller than the 1970s…
%pt
4
RMSEs of growth forecasts - G7 countries
May projection for current year
November projection for next year
3
May projection for next year
2
1
0
1971-1981
1982-1990
1991-2006
Unweighted average of errors in G7 economies
2007-2012
26

27. …and are not that different if growth volatility
is accounted for
Putting the errors in context
RMSEs adjusted for outcomes - G7 countries
2.5
2.0
May projection for current year
November projection for next year
May projection for next year
1.5
1.0
0.5
0.0
1971-1981
1982-1990
1991-2006
2007-2012
Ratio of GDP growth RMSE to standard deviation of GDP growth
27

28. Putting the errors in context
The forecasts were considerably worse
than usual for some countries
Growth forecast errors in 2007-12 compared to the past
From May projections for the next year
%pt
0.0
-0.5
-1.0
-1.5
-2.0
Additional effect in 2007-12
Average error 1982-2006
-2.5
CAN
FRA
DEU
ITA
JPN
GBR
USA
Dark blue and orange indicate statistical significance (at the 10% level)
28

29. Directional accuracy is good for growth
accelerations, but less so for growth slowdowns
Putting the errors in context
Directional accuracy of May growth projections
OECD countries: 2007-12
Accelerations Decelerations
Number in period
56
118
% correct: projections for same year
86
88
% correct: projections for next year
91
50
G7 countries: 1982-2006
Accelerations Decelerations
Number in period
82
78
% correct: projections for same year
79
83
% correct: projections for next year
74
45
Tools to help identify growth slowdowns are
needed.
29

30. Putting the errors in context
However, forecast errors were strikingly
similar across forecasters (also pre-crisis)
%pt
4
Average errors in forecasts from May for the year
ahead - G7 countries
%pt
From one year ahead
From the same year
4
2
2
0
0
-2
-2
-4
-6
OECD
IMF
Consensus
EC
2007 2008 2009 2010 2011 2012
-4
-6
2007 2008 2009 2010 2011 2012
Unweighted average for the G7 economies
30

31. Putting the errors in context
The OECD forecasts are projections rather
than pure forecasts
• They are conditional on a set of assumptions:
– financial market variables, fiscal policy changes,
commodity prices.
• They can point to the need for policy changes
to tackle unsustainable developments:
– If implemented, projection errors may occur.
• Politically-sensitive issues can cause errors:
– A two year projection had to assume the euro area crisis
would diminish.
31

32. CHANGES IN
FORECASTING PRACTICES
AND PROCEDURES
32

33. Post-crisis changes in forecast practices and
procedures at the OECD and elsewhere
Ongoing changes
• Greater centralisation of the forecast process.
• Enhanced monitoring of near-term
developments.
• Increased attention on financial sector.
• Enhanced focus on risk assessments and
global spillovers.
33

34. Ongoing changes
Increased centralisation of the forecast
process
• Reflects errors common to all forecasts and
the importance of global financial and trade
interconnections.
– Early view on global developments and risks
and their implications.
– Early guidance via “top down” projections
bringing together information from different
sources and models.
34

35. Enhanced monitoring of near-term
developments.
Ongoing changes
• Surveys and high frequency data can provide
early signals of big changes.
– Nowcasting (OECD indicator models).
– Composite leading indicators.
– Evidence from business contacts (forecast
discussions with BIAC and TUAC).
– Use of internet-based indicators (“big data”)
35

36. Increased attention on financial sector
Ongoing changes
– OECD aggregate financial conditions indices
(used forecasts and scenarios).
– Incorporation of broader range of financial
variables in projections.
– Enhanced discussions with internal/external
financial market specialists.
– Incorporating banking sector and global
interconnectedness in macro models.
36

37. Enhanced focus on risk assessments and
global spillovers
Ongoing changes
• Reflects uncertainty about basic assumptions
behind projections.
– More information about the risks around the main
projections.
– Greater use of quantitative scenario analysis to
illustrate possible outcomes.
– Horizon scanning to plan ahead for unlikely but
potentially costly events.
37

38. Summary
• Forecasting in recent years proved very challenging and
growth has been repeatedly over-estimated.
• Global interconnectedness, structural policy settings and
the health of the banking sector are all related to forecast
errors.
• Errors in assumptions about the speed at which the euro
crisis would ease have been an important source of growth
forecast errors.
• Important changes are now taking place to forecasting
practices and procedures.
38

39. SPARE SLIDES

40. Properties of the forecast errors
Inflation was under-estimated on
average
Inflation = percentage change in the private consumption deflator
40

41. Putting the errors in context
The accuracy of G7 forecasts over the
long-run
41

42. Putting the errors in context
The indicator models were helpful
Quarterly indicator model estimates and GDP growth
%
G7, weighted average
4
2
0
-2
-4
Estimate 1 quarter ahead
-6
Current quarter estimate
-8
Outturn
-10
2007:Q1
2008:Q1
2009:Q1
2010:Q1
2011:Q1
2012:Q1
42