Blue Sky 19/09/16 Plenary 2

1. 1
The European Commission’s
science and knowledge service
Joint Research Centre
Auditing Composite
Indicators of
Innovation
Dániel Vértesy, PhD
OECD Blue Sky III | Ghent, 19 Sep 2016

2. 2
Name a single indicator that
measures innovation!
Innovation is a complex process,
involves multiple actors, with different
functions, values & interests
Whether you assess the effort, the process, the outcomes, or the
impacts of Innovation, you will likely need different information
= a multi-faceted phenomenon
?
Expenditure on
innovation Interactions
Introducing New-to-
the-world products
Competitiveness

3. 3
Composite Indicators of Innovation
Composite indicators – aggregate measures of multiple, individual
indicators – are popular tools to measure innovation:
• The Global Innovation Index (GII) of Cornell/INSEAD/WIPO
(since 2007; with WIPO since 2011);
• The European Commission’s European Innovation Scoreboard
with its Summary Innovation Index (SII) (since 2000); and
the Innovation Output Indicator (since 2013)
Widespread; Newspapers report it;
Like university rankings: unstoppable?

4. 4
Composite Indicators
Advantages:
• Support decision makers by
summarizing complex or multi-
dimensional issues
• Provide the “big picture”,
highlight common trends
• Measure a latent phenomenon
that is not directly measureable
• Attract public interest by
benchmarking
Pitfalls:
• Offer misleading, non-robust
policy messages if they are poorly
constructed or misinterpreted
• May invite politicians to draw
simplistic policy conclusions
• Easier to “manipulate” than
individual indicators; the selection
of sub-indicators and weights
could be the target of political
challenge
Assessing their quality and
validity is particularly relevant
The development process helps
• Better understand how a
system functions
• Identify latent dimensions,
overlaps, redundancies or
trade-offs between components

5. 5
The Quality of Composite Indicators
“Composite indicators sit between analysis and advocacy, but
quality discriminates the plausible from the rhetorical” (Saltelli, 2007)
Codified and continuously refined methodology
• The OECD-JRC Handbook; Audits – robustness and
sensitivity analyses; etc.
Quantification (modelling) involves making normative choices
• Indicators are intrinsically linked to policy priorities!
Normative choices affect:
• the concept;
• the operationalization;
• accounting for information loss
 Composite and individual
indicators are alike
What
actors to
include?
What indicators
to include
How to
address
data quality
issues?
Which
measure is
more
relevant?

6. 6
Conclusions – and questions
1. The process of searching for ways to summarize multiple indicators
helps identify choices for policy
2. The most commonly used composite indicators of innovation
include a few components that have [persistently] little or no
impact on aggregate scores.
• Innovation indicators advocate an incomplete picture of
innovation – typically [incremental] innovation in R&D- and
patent-intensive industries.
• Do not aggregate all indicators, or use scoreboards, in case of
significant loss of information
Is this sufficient to solve
today’s societal challenges?
Is this aligned with
policy needs?
Risks overlooking unintended consequences of innovations
I.e., The more a country’s firms spend on non-R&D
innovation, the less they spend on business R&D

7. THANK YOU!
 daniel.vertesy@jrc.ec.europa.eu
 http://ec.europa.eu/jrc/en/coin
Competence Centre on Composite Indicators
and Scoreboards (CC-COIN);
Joint Research Centre (JRC)
European Commission,