In this presentation, I summarise the July 2017 NAS Report of the Assessment of the Advanced Research Projects Agency – Energy, in 5 slides.

1. US Advanced Research Projects
Agency - Energy Evaluation Report
June 2017…
The ARPA-E is supposed to identify and promote ideas
that might be revolutionary, and to support ideas that
industry is not likely to support alone.
The National Academy of Sciences reviewed how well the
agency is going after just 6 years.
AUGUST 2017

2. The assessment found that ARPA-E shouldn’t
really change what they are doing.
• They are backing both ‘truly novel’ ideas, and
ideas in ‘white spaces’.
• They have really good program directors.
• But they could do more to guide evaluation
of how well they are doing with long-term
goals.
WHAT DOES IT SAY.
AUGUST 2017

3. Much of the review was an audit of culture.
• The assessment committee gave the culture
around program directors a big tick– good
autonomy and good key staff transitions.
The review also provided insight into how to audit
the management of risk and uncertainty.
• The record over the six years showed 10% of
projects had been terminated early, showing an
appropriate tolerance of failure.
WHAT ELSE DOES IT SAY.
AUGUST 2017

4. While ARPA-E borrowed a lot from DARPA,
the defence agency, it had also evolved systems
to match a reduced scale.
Project selection processes were good.
• They don’t require projects to be ‘racked and
stacked’, according to a mean reviewer score.
• This allows discretion for directors to fund
‘truly novel’ projects.
WHAT ELSE DOES IT SAY.
AUGUST 2017

5. • 6 years is too early to assess effectiveness
of a research project/agency.
• The culture of research agencies is unique,
and the evaluation uses interesting audit
techniques.
• Promoting ‘exploration’ (new ideas) and
‘exploitation’ (incremental change) can be
done by the same agency.
The comments on these slides are the views of Tim Kirby, Sydney.
You should always read the report itself before putting you reputation of any money on the line.
WHAT DOES IT MEAN.
Tim Kirby, Sydney CA, CIA, LA-EMS
au.linkedin.com/in/timkirbysydney
AUGUST 2017