This document discusses the inability to accurately forecast future energy technology costs and performance over long time horizons. It provides several examples where past energy forecasts made by models and experts were wildly inaccurate when compared to actual outcomes. This is because economic, social and technological systems lack the structural stability and predictability of physical systems. Forecasts are strongly influenced by unexpected events and changes in market structure, technology innovation, policies and other hard-to-predict factors. Therefore, no amount of additional resources or effort can overcome the fundamental limitations of long-term energy forecasting. Experimental approaches and focusing on physical constraints provide better guidance than precise cost predictions decades into the future.

1. Past performance is no
guide to future returns:
Why we can’t accurately
forecast the future
Jonathan Koomey, Ph.D.
Research Fellow, Stanford University
http://www.koomey.com
Presented on a webinar for US EPA and US DOE
May 18, 2016
1
Copyright
Jonathan
G.
Koomey
2016

2. My background
• Founded LBNL’s End-Use Forecasting
group and led that group for more than 11
years.
• Peer reviewed articles and books on
– Forecasting methodology
– Economics of greenhouse gas mitigation
– Critical thinking skills
– Information technology and resource use
2
Copyright
Jonathan
G.
Koomey
2016

3. 3
Cost-benefit analysis: the
standard approach
Copyright
Jonathan
G.
Koomey
2016

4. True or False?:
If only we had enough…
• Time
• Money
• Graduate Students
• Coffee
we could accurately predict the
cost of energy technologies in
2050
4
Copyright
Jonathan
G.
Koomey
2016

5. Widespread modeling practice
implies that the answer is “True”
5
Copyright
Jonathan
G.
Koomey
2016

6. Based on my experience and
reviews of historical
retrospectives on forecasting, I
say “No way”
6
Copyright
Jonathan
G.
Koomey
2016

7. Aside: Many of the best modelers
acknowledge the difficulties in the
pursuit of accurate forecasts, but
in their heart of hearts they still
believe they can predict accurately
with greater effort
7
Copyright
Jonathan
G.
Koomey
2016

8. Uncertainty affects even physical
systems
Es=mates
of
Planck’s
constant
"h"
over
=me.
In
this
physical
system
researchers
repeatedly
underes=mated
the
error
in
their
determina=ons.
At
each
stage
uncertain=es
existed
of
which
the
researchers
were
unaware.
The
problem
of
error
es=ma=on
is
far
greater
in
long-­‐range
energy
forecas=ng.
Taken
from
Koomey
et
al.
2003.
8
Copyright
Jonathan
G.
Koomey
2016

9. Forecasting Accuracy: The
Models Have Done Badly
• Energy forecasting models have little or no ability to
accurately predict future energy prices and demand
(Craig et al. 2002)
• Even the sign of the impacts of proposed policies is a
function of key assumptions (Repetto and Austin
1997)
• The dismal accuracy and inherent limitations of these
models should make modelers modest in the
conclusions they draw (Decanio 2003)
Craig, P., A. Gadgil, and J. Koomey (2002). “What Can History Teach Us? A Retrospective Analysis
of Long-term Energy Forecasts for the U.S.” Annual Review of Energy and the Environment 2002.
R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA, Annual Reviews, Inc. (also LBNL-50498). 27: 83-118.
Repetto, R. and D. Austin (1997). The Costs of Climate Protection: A Guide for the Perplexed. Washington, DC,
World Resources Institute.
DeCanio, S. J. (2003). Economic Models of Climate Change: A Critique. Basingstoke, UK, Palgrave-Macmillan.
9
Copyright
Jonathan
G.
Koomey
2016

10. One example: 1970s projections
of year 2000 U.S. primary energy
Source: Craig, Paul, Ashok Gadgil, and Jonathan Koomey. 2002. "What
Can History Teach Us?: A Retrospective Analysis of Long-term Energy
Forecasts for the U.S." In Annual Review of Energy and the Environment
2002. Edited by R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA:
Annual Reviews, Inc. (also LBNL-50498). pp. 83-118.
10
Copyright
Jonathan
G.
Koomey
2016

11. What drove errors in US primary
energy forecasts?
Copyright
Jonathan
G.
Koomey
2016
11
Graph
from
Hirsh
and
Koomey
2015

12. Another
example:
Oil
price
projec3ons
by
U.S.
DOE,
AEO
1982
through
AEO
2000
12
Copyright
Jonathan
G.
Koomey
2016

13. Not
any
beEer
aFer
2000:
Oil
price
projec3ons
by
U.S.
DOE,
AEO
2000
through
AEO
2007
13
Copyright
Jonathan
G.
Koomey
2016

14. Yet another example: NERC fan
Copyright
Jonathan
G.
Koomey
2016
14
US
electricity
genera=on
BkWh/year

15. Why Are Long-term Energy
Forecasts Almost Always Wrong?
• Core data and assumptions, which drive
results, are based on historical
experience, which can be misleading if
structural conditions change
• The exact timing and character of pivotal
events and technology changes cannot be
predicted
Laitner, J.A., S.J. DeCanio, J.G. Koomey, A.H. Sanstad. (2003) “Room for Improvement: Increasing the Value of
Energy Modeling for Policy Analysis.” Utilities Policy, 11, pp. 87-94.
15
Copyright
Jonathan
G.
Koomey
2016

16. Conditions for Model Accuracy
• Hodges and Dewar: models can be
accurate when they describe systems
that
– are observable and permit collection of
ample and accurate data
– exhibit constancy of structure over time
– exhibit constancy across variations in
conditions not specified in the model
Source: Hodges, James S., and James A. Dewar. 1992. Is it you or your model talking? A framework for model
validation. Santa Monica, CA: RAND. ISBN 0-8330-1223-1.
16
Copyright
Jonathan
G.
Koomey
2016

17. ∑: Accurate forecasts require
structural constancy and no
surprises
17
Copyright
Jonathan
G.
Koomey
2016

18. Market structure can change fast
Source:
Scher
and
Koomey
2010.
18
Copyright
Jonathan
G.
Koomey
2016

19. Fast changing markets #2: US
electricity consumption
Copyright
Jonathan
G.
Koomey
2016
19
Graph
from
Hirsh
and
Koomey
2015

20. Surprises can be big:
U.S. nuclear busbar costs
Source: Koomey and Hultman 2007. Assumes 7% real discount rate.
Projected cost range from Tybout 1957
20
Copyright
Jonathan
G.
Koomey
2016

21. Implications for long-term
energy forecasting
• Forecasting models describing well-defined physical
systems using correct parameters can be accurate
because physical laws are geographically and
temporally invariant (as long as there are no surprises)
• Economic, social, and technological systems do not
exhibit the required structural constancy, so models
forecasting the future of these systems are doomed to
be inaccurate. Four big sources of inconstancy
– Pivotal events (like Sept. 11th or the 1970s oil shocks)
– Technological innovation
– Institutional change
– Policy choices
21
Copyright
Jonathan
G.
Koomey
2016

22. ∑: Economics ≠ Physics
22
Copyright
Jonathan
G.
Koomey
2016

23. So no matter how many $, coffee
cups, months, or graduate
students you have, accurate long-
run forecasting of technology
costs is impossible
23
Copyright
Jonathan
G.
Koomey
2016

24. Two senses of the word
“impossible”:
Practically
and
Theoretically
Either way, the net result is the
same: inaccurate forecasts
24
Copyright
Jonathan
G.
Koomey
2016

25. So what does this result imply
for predictions of the costs of
energy technologies?
25
Copyright
Jonathan
G.
Koomey
2016

26. Some lessons
• The world is evolutionary and path dependent
– Increasing returns, transaction costs, information
asymmetries, bounded rationality, prospect theory
– Our actions now affect our options later (so do
surprises!)
• Experimentation is the order of the day
• Use real data to prove results
– For nuclear power, we’re in the “show me” stage.
Cost projections are no longer enough
• Prefer technologies that
– are mass produced vs. site-built
– have short lead times vs. longer lead times
26
Copyright
Jonathan
G.
Koomey
2016

27. Nuke costs: here we go again?
Source: Koomey and Hultman 2007.
27
Copyright
Jonathan
G.
Koomey
2016

28. “No battle plan survives contact with the
enemy.” –Helmuth von Moltke the elder
Copyright
Jonathan
G.
Koomey
2016
28

29. More lessons
• Use physical and technological constraints to
define bounding cases. Examples:
– 2 degrees Celsius warming limit implies a carbon
budget, which implies a certain rate of
implementation of non-fossil energy sources to
avoid worst effects of climate change.
– Certain technologies use materials that are in
limited supply. Working backwards from a goal
can help identify resource constraints.
– Lifetime of power generation technologies and
buildings limits penetration of new technologies
unless we scrap existing capital
29
Copyright
Jonathan
G.
Koomey
2016

30. Reconsidering benefit-cost
analysis for climate
• "A corollary is that it is fruitless to attempt to determine the "optimal"
carbon tax. If neither the costs nor the benefits can be known with
any precision, just about the only thing that can be said with
certainty about the welfare maximizing price of carbon emissions is
that it is greater than zero. Economists have a great deal to say
about how to implement such a tax efficiently and effectively, about
the similarities and differences between a tax and a system of
tradable carbon emissions permits, about about the best way to
recycle the revenue from such a tax or permit system. And, as we
have seen above, the distributional consequences of such a tax or
permit auction plan will affect other economic variables through
system-wide feedbacks. However, any attempt to specify the exact
level of the "optimal" tax is less an exercise in scientific calculation
than a manifestation of the analyst’s willingness to step beyond the
limits of established economic knowledge."
• –DeCanio, Stephen J. 2003. Economic Models of Climate Change:
A Critique. Basingstoke, UK: Palgrave-Macmillan. p.157.
Copyright
Jonathan
G.
Koomey
2016
30

31. Conclusions
• It is impossible to accurately forecast energy
technology characteristics because of
– structural inconstancy and
– pivotal events
• Forecasting community has yet to absorb the
implications of this insight
• To cope we need new ways to think about the future
– Experimental approach to implementation (try many things,
fail fast, learn quickly, try again)
– Rely on physical and technological constraints to create
bounding cases
– Embrace path dependence (there is no optimal solution,
just lots of possible pathways of roughly similar costs)
31
Copyright
Jonathan
G.
Koomey
2016

32. “The best way to predict the future is to
invent it.” –Alan Kay
Copyright
Jonathan
G.
Koomey
2016
32

33. Some Key References
• Craig, Paul, Ashok Gadgil, and Jonathan Koomey. 2002. "What Can History Teach Us?: A
Retrospective Analysis of Long-term Energy Forecasts for the U.S." In Annual Review of Energy and
the Environment 2002. Edited by R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA: Annual
Reviews, Inc. pp. 83-118.
• Ghanadan, Rebecca, and Jonathan Koomey. 2005. "Using Energy Scenarios to Explore Alternative
Energy Pathways in California." Energy Policy. vol. 33, no. 9. June. pp. 1117-1142.
• Hirsh, Richard F., and Jonathan G. Koomey. 2015. "Electricity Consumption and Economic Growth:
A New Relationship with Significant Consequences?" The Electricity Journal. vol. 28, no. 9.
November. pp. 72-84. [http://www.sciencedirect.com/science/article/pii/S1040619015002067]
• Koomey, Jonathan. 2008. Turning Numbers into Knowledge: Mastering the Art of Problem Solving.
Oakland, CA: Analytics Press. 2nd edition. <http://www.analyticspress.com>
• Koomey, Jonathan. 2002. "From My Perspective: Avoiding "The Big Mistake" in Forecasting
Technology Adoption." Technological Forecasting and Social Change. vol. 69, no. 5. June. pp.
511-518.
• Koomey, Jonathan G., Paul Craig, Ashok Gadgil, and David Lorenzetti. 2003. "Improving long-range
energy modeling: A plea for historical retrospectives." The Energy Journal (also LBNL-52448). vol.
24, no. 4. October. pp. 75-92.
• Chapter 4: “Why we can’t accurately forecast the future”, in Koomey, Jonathan G. 2012. Cold Cash,
Cool Climate: Science-Based Advice for Ecological Entrepreneurs. Burlingame, CA: Analytics
Press. [http://www.analyticspress.com/cccc.html]
• Koomey, Jonathan. 2013. "Moving Beyond Benefit-Cost Analysis of Climate Change."
Environmental Research Letters. vol. 8, no. 041005. December 2. [http://iopscience.iop.org/
1748-9326/8/4/041005/]
• Laitner, J.A., S.J. DeCanio, J.G. Koomey, A.H. Sanstad. (2003) “Room for Improvement: Increasing
the Value of Energy Modeling for Policy Analysis.” Utilities Policy, vol. 11, no. 2. June. pp. 87-94.
• Scher, Irene, and Jonathan G. Koomey. 2011. "Is Accurate Forecasting of Economic Systems
Possible?" Climatic Change. Vol 104, No. 3-4, pp.473-479.
33
Copyright
Jonathan
G.
Koomey
2016

34. More Key References
• Armstrong, J. Scott, ed. 2001. Principles of Forecasting: A Handbook for Researchers and
Practitioners. Norwell, MA: Kluwer Academic Publishers.
• Ascher, William. 1978. Forecasting: An Appraisal for Policy Makers and Planners. Baltimore,
MD: Johns Hopkins University Press.
• Cohn, Steve. 1991. "Paradigm Debates in Nuclear Cost Forecasting." Technological
Forecasting and Social Change. vol. 40, no. 2. September. pp. 103-130.
• Grubler, Arnulf, Nebojsa Nakicenovic, and David G. Victor. 1999. "Dynamics of energy
technologies and global change." Energy Policy. vol. 27, no. 5. May. pp. 247-280.
• Hodges, James S., and James A. Dewar. 1992. Is it you or your model talking? A framework for
model validation. Santa Monica, CA: RAND. ISBN 0-8330-1223-1.
• Huntington, Hillard G. 1994. "Oil Price Forecasting in the 1980s: What Went Wrong?" The
Energy Journal. vol. 15, no. 2. pp. 1-22.
• Huss, William R. 1985. "Can Electric Utilities Improve Their Forecast Accuracy? The Historical
Perspective." In Public Utilities Fortnightly. December 26, 1985. pp. 3-8.
• Landsberg, Hans H. 1985. "Energy in Transition: A View from 1960." The Energy Journal. vol. 6,
pp. 1-18.
• O'Neill, Brian C., and Mausami Desai. 2005. "Accuracy of past projections of U.S. energy
consumption." Energy Policy. vol. 33, no. 8. May. pp. 979-993.
• Tetlock, Philip E. 2005. Expert Political Judgment: How Good Is It? How Can We Know?
Princeton, NJ: Princeton University Press.
• Tybout, Richard A. 1957. "The Economics of Nuclear Power." American Economic Review. vol.
47, no. 2. May. pp. 351-360.
34
Copyright
Jonathan
G.
Koomey
2016