1. Past performance is no guide
to future returns: What can we
really say about the future of
economic, social, and
technological systems?
Jonathan Koomey, Ph.D.
Consulting Professor, Stanford University
http://www.koomey.com
Presented at the UC Berkeley Forum on Nuclear Futures
December 10, 2010
Copyright Jonathan G. Koomey 2010 1
2. My background
• Founded LBNL’s End-Use Forecasting
group and led the group for more than 11
years.
• Peer reviewed articles and books on
– Forecasting methodology
– Economics of greenhouse gas mitigation
– Critical thinking skills
– Information technology and resource use
Copyright Jonathan G. Koomey 2010 2
3. True or False?:
If only we had enough…
• Time
• Money
• Graduate Students
• Coffee
we could accurately predict the
cost of energy technologies in
2030.
Copyright Jonathan G. Koomey 2010 3
5. Based on my experience and
reviews of historical
retrospectives on forecasting, I
say “No way”
Copyright Jonathan G. Koomey 2010 5
6. Aside: Many of the best modelers
acknowledge the difficulties in the
pursuit of accurate forecasts, but
in their heart of hearts they still
believe they can predict accurately
with greater effort.
Copyright Jonathan G. Koomey 2010 6
7. Uncertainty affects even physical
systems
Es<mates of Planck’s constant "h" over <me. In this physical system
researchers repeatedly underes<mated the error in their determina<ons. At
each stage uncertain<es existed of which the researchers were unaware. The
problem of error es<ma<on is far greater in long‐range energy forecas<ng.
Taken from Koomey et al. 2003.
Copyright Jonathan G. Koomey 2010 7
8. Forecasting Accuracy: The
Models Have Done Badly
• Energy forecasting models have little or no ability to
accurately predict future energy prices and demand
(Craig et al. 2002)
• Even the sign of the impacts of proposed policies is a
function of key assumptions (Repetto and Austin
1997)
• The dismal accuracy and inherent limitations of these
models should make modelers modest in the
conclusions they draw (Decanio 2003)
Craig, P., A. Gadgil, and J. Koomey (2002). “What Can History Teach Us? A Retrospective Analysis
of Long-term Energy Forecasts for the U.S.” Annual Review of Energy and the Environment 2002.
R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA, Annual Reviews, Inc. (also LBNL-50498). 27: 83-118.
Repetto, R. and D. Austin (1997). The Costs of Climate Protection: A Guide for the Perplexed. Washington, DC,
World Resources Institute.
DeCanio, S. J. (2003). Economic Models of Climate Change: A Critique. Basingstoke, UK, Palgrave-Macmillan.
Copyright Jonathan G. Koomey 2010 8
9. One example: 1970s projections
of year 2000 U.S. primary energy
Source: Craig, Paul, Ashok Gadgil, and Jonathan Koomey. 2002. "What
Can History Teach Us?: A Retrospective Analysis of Long-term Energy
Forecasts for the U.S." In Annual Review of Energy and the Environment
2002. Edited by R. H. Socolow, D. Anderson and J. Harte. Palo Alto, CA:
Annual Reviews, Inc. (also LBNL-50498). pp. 83-118.
Copyright Jonathan G. Koomey 2010 9
11. Not any
beEer
aFer 2000:
Oil price
projec3ons
by U.S.
DOE, AEO
2000
through
AEO 2007
Copyright Jonathan G. Koomey 2010 11
12. Why Are Long-term Energy
Forecasts Almost Always Wrong?
• Core data and assumptions, which drive
results, are based on historical
experience, which can be misleading if
structural conditions change
• The exact timing and character of pivotal
events and technology changes cannot be
predicted
Laitner, J.A., S.J. DeCanio, J.G. Koomey, A.H. Sanstad. (2003) “Room for Improvement: Increasing the Value of
Energy Modeling for Policy Analysis.” Utilities Policy, 11, pp. 87-94.
Copyright Jonathan G. Koomey 2010 12
13. Conditions for Model Accuracy
• Hodges and Dewar: models can be
accurate when they describe systems
that
– are observable and permit collection of
ample and accurate data
– exhibit constancy of structure over time
– exhibit constancy across variations in
conditions not specified in the model
Source: Hodges, James S., and James A. Dewar. 1992. Is it you or your model talking? A framework for model
validation. Santa Monica, CA: RAND. ISBN 0-8330-1223-1.
Copyright Jonathan G. Koomey 2010 13
14. ∑: Accurate forecasts require
structural constancy and no
surprises
Copyright Jonathan G. Koomey 2010 14
15. Market structure can change fast
Source: Scher and Koomey 2010.
Copyright Jonathan G. Koomey 2010 15
16. Surprises can be big:
U.S. nuclear busbar costs
Projected cost range from Tybout 1957
Source: Koomey and Hultman 2007. Assumes 7% real discount rate.
Copyright Jonathan G. Koomey 2010 16
17. Implications for long-term
energy forecasting
• Forecasting models describing well-defined physical
systems using correct parameters can be accurate
because physical laws are geographically and
temporally invariant (as long as there are no surprises)
• Economic, social, and technological systems do not
exhibit the required structural constancy, so models
forecasting the future of these systems are doomed to
be inaccurate. Four big sources of inconstancy
– Pivotal events (like Sept. 11th or the 1970s oil shocks)
– Technological innovation
– Institutional change
– Policy choices
Copyright Jonathan G. Koomey 2010 17
18. ∑: Economics ≠ Physics
Copyright Jonathan G. Koomey 2010 18
19. So no matter how many $, coffee
cups, months, or graduate
students you have, accurate long-
run forecasting of technology
costs is impossible.
Copyright Jonathan G. Koomey 2010 19
20. Two senses of the word
“impossible”:
Practically
and
Theoretically
Either way, the net result is the
same: inaccurate forecasts.
Copyright Jonathan G. Koomey 2010 20
21. So what does this result imply
for predictions of the costs of
energy technologies?
Copyright Jonathan G. Koomey 2010 21
22. Some lessons
• The world is evolutionary and path dependent
– Increasing returns, transaction costs, information
asymmetries, bounded rationality, prospect theory
– Our actions now affect our options later (so do
surprises!)
• Experimentation is the order of the day
• Use real data to prove results
– For nuclear power, we’re in the “show me” stage.
Cost projections are no longer enough
• Prefer technologies that
– are mass produced vs. site-built
– have short lead times vs. longer lead times
Copyright Jonathan G. Koomey 2010 22
23. Nuke costs: here we go again…
Source: Koomey and Hultman 2007.
Copyright Jonathan G. Koomey 2010 23
24. More lessons
• Use physical and technological constraints to
define bounding cases. Examples:
– 2 degrees Celsius warming limit implies a carbon
budget, which implies a certain rate of
implementation of non-fossil energy sources to
avoid worst effects of climate change.
– Certain technologies use materials in limited
supply (e.g. rare earths). Working backwards
from a goal can help identify resource constraints.
– Lifetime of power generation technologies and
buildings limits penetration of new technologies
unless we scrap existing capital
Copyright Jonathan G. Koomey 2010 24
25. Conclusions
• It is impossible to accurately forecast energy
technology characteristics because of
– structural inconstancy and
– pivotal events
• Forecasting community has yet to absorb the
implications of this insight
• To cope we need new ways to think about the future
– Experimental (Bayesian) approach to implementation (try
many things, fail fast, learn quickly, try again)
– Rely on physical and technological constraints to create
bounding cases
– Embrace path dependence (there is no optimal solution,
just lots of possible pathways of roughly similar costs)
Copyright Jonathan G. Koomey 2010 25
26. Some Key References
• Craig, Paul, Ashok Gadgil, and Jonathan Koomey. 2002. "What Can History
Teach Us?: A Retrospective Analysis of Long-term Energy Forecasts for the
U.S." In Annual Review of Energy and the Environment 2002. Edited by R. H.
Socolow, D. Anderson and J. Harte. Palo Alto, CA: Annual Reviews, Inc. pp.
83-118.
• Ghanadan, Rebecca, and Jonathan Koomey. 2005. "Using Energy Scenarios to
Explore Alternative Energy Pathways in California." Energy Policy. vol. 33, no. 9.
June. pp. 1117-1142.
• Koomey, Jonathan. 2001. Turning Numbers into Knowledge: Mastering the Art of
Problem Solving. Oakland, CA: Analytics Press. (2d Printing, 2004). <http://
www.analyticspress.com>
• Koomey, Jonathan. 2002. "From My Perspective: Avoiding "The Big Mistake" in
Forecasting Technology Adoption." Technological Forecasting and Social
Change. vol. 69, no. 5. June. pp. 511-518.
• Koomey, Jonathan G., Paul Craig, Ashok Gadgil, and David Lorenzetti. 2003.
"Improving long-range energy modeling: A plea for historical retrospectives." The
Energy Journal (also LBNL-52448). vol. 24, no. 4. October. pp. 75-92.
• Laitner, J.A., S.J. DeCanio, J.G. Koomey, A.H. Sanstad. (2003) “Room for
Improvement: Increasing the Value of Energy Modeling for Policy Analysis.”
Utilities Policy, vol. 11, no. 2. June. pp. 87-94.
• Scher, Irene, and Jonathan G. Koomey. 2010. "Is Accurate Forecasting of
Economic Systems Possible?" Climatic Change (forthcoming).
Copyright Jonathan G. Koomey 2010 26
27. More Key References
• Armstrong, J. Scott, ed. 2001. Principles of Forecasting: A Handbook for Researchers
and Practitioners. Norwell, MA: Kluwer Academic Publishers.
• Ascher, William. 1978. Forecasting: An Appraisal for Policy Makers and Planners.
Baltimore, MD: Johns Hopkins University Press.
• Cohn, Steve. 1991. "Paradigm Debates in Nuclear Cost Forecasting." Technological
Forecasting and Social Change. vol. 40, no. 2. September. pp. 103-130.
• Grubler, Arnulf, Nebojsa Nakicenovic, and David G. Victor. 1999. "Dynamics of energy
technologies and global change." Energy Policy. vol. 27, no. 5. May. pp. 247-280.
• Hodges, James S., and James A. Dewar. 1992. Is it you or your model talking? A
framework for model validation. Santa Monica, CA: RAND. ISBN 0-8330-1223-1.
• Huntington, Hillard G. 1994. "Oil Price Forecasting in the 1980s: What Went Wrong?"
The Energy Journal. vol. 15, no. 2. pp. 1-22.
• Huss, William R. 1985. "Can Electric Utilities Improve Their Forecast Accuracy? The
Historical Perspective." In Public Utilities Fortnightly. December 26, 1985. pp. 3-8.
• Landsberg, Hans H. 1985. "Energy in Transition: A View from 1960." The Energy Journal.
vol. 6, pp. 1-18.
• O'Neill, Brian C., and Mausami Desai. 2005. "Accuracy of past projections of U.S. energy
consumption." Energy Policy. vol. 33, no. 8. May. pp. 979-993.
• Tetlock, Philip E. 2005. Expert Political Judgment: How Good Is It? How Can We Know?
Princeton, NJ: Princeton University Press.
• Tybout, Richard A. 1957. "The Economics of Nuclear Power." American Economic
Review. vol. 47, no. 2. May. pp. 351-360.
Copyright Jonathan G. Koomey 2010 27
