The document provides a critical assessment of the European Innovation Scoreboard (EIS) which is used to evaluate and compare the innovation performance of EU member states. It notes methodological issues with the EIS such as the selection and weighting of indicators, data availability and quality. It also critiques how countries' perceive and discuss their innovation performance based solely on EIS rankings, which can lead to indicator-driven rather than evidence-based policymaking. The document concludes the EIS has limitations and a different approach should be taken in its use for policy discussions.

1. The use of indicators in innovation policy debate - a critical assessment of the European Innovation Scoreboard (EIS) Wolfgang Polt Joanneum Research [email_address] Conference on Patent Statistics for Decision Makers Vienna, 3-4 September 2008 Based on: Schibany, Streicher, Gassler: The European Innvoation Scoreboard: the advantages and disadvantages of indicator-driven country rankings (in German). Joanneum Research Working Papers 65-2007, Vienna, October 2007

2. The EIS in policy debates European Innovation Scoreboad The EIS is the instrument developed by the EC to evaluate and compare the innovation performance of the Member States. Part of the Lisbon Strategy – Open Method of Coordination (OMC) A multi-dimensional scoreboard which covers a single policy field Currently covers 26 indicators Is aggregated into a synthetic ‚Summary Innovation Index (SII)‘ Recieves high policy attention, in some countries even making it into headline news. Exceplified using the case of Austria

3. Austrian innovation performance and its perception Austria‘s innovation performance: rank 10 among EU countries, but in the top-5 with respect to dynamic. „… rank 10 but with high catching up potential “ (Science ORF) In a comparison of the 25 EU countries, Austria has improved its innovation performance from 10 to 5, overtaking Norway, Ireland, the Netherlands, France and Belgium „ Austria among the top-five innovation performer in the EU“ ( Federal Chancellor) Austria has moved down from 5 to 9 in the SII „ Austria is losing ground“ (APA); „Rank 3 should be the aim of R&D-policy“ (State Secretary for Research and Innovation) 2004 2005 2006

4. Austrian innovation performance and its perception “ Considering ‘innovation inputs’, the shares of SMEs innovating in-house and introducing ‘soft’ organisational innovations are high, and Austria does extremely well compared to EU25 averages in terms of ‘innovation outputs’ such as intellectual property rights (IPR), but these high performance levels are not reflected in output indicators measuring other downstream aspects of innovation performance and added value. Exports of high technology products, sales of new-to-market products and sales of new-to-firm products, for example, are markedly lower than the EU25 averages. Overall, therefore, the main characteristics of the Austrian R&D and innovation system are high R&D expenditure levels, high public subsidy dependence, low downstream innovation performance levels and potential human resource problems. “ From the recent CREST peer review on Austria (August 2008) – a report almost exclusively using EIS data to characterize the Austrian innovation system 2008

5. A critical assessment of the EIS as a tool for policy discussion Methodological critique Critique of political discourse Suggestions for a different approach

6. Methodological critique Selection of indicators Data availability Data quality Weighting of indicators Quantitative results transformed into ranking (Summary Innovation Index – SII)

7. Selection of indicators

8. Selection of indicators Some indicators are very „structural“ by nature (long-term) Several indicators are affected by business cycle development (short-term) and show high volatility 18 of 25 indicators are defined as shares „ more-is-better“ assumption: implies a pre-defined optimal value (100% of enterprises receiving public subsidies as an optimal value?) Only 2 indicators can directly be influenced by short-run policy 7 indicators taken from on CIS Indicators chosen by majority vote...

9. Data availability Offical SII ranking at the date of publication SII ranking based on EIS 2007

10. Data quality – Example I Indicator 1.2: Population with tertiary education per 100 population aged 25-64 with tertiary education However: this is a 40-year moving average! It simply cannot change by such amounts in the course of just a few years

11. Data quality – Example II Indicator 3.4: Early stage venture capital as % of GDP Apparently, UK‘s VC has quadrupled from 2005 to 2006, thus raising the EU average considerably...

12. Weighting of indicators „ For reasons of simplicity … and to keep the weighting as simple as possible“ all indicators receive the same weight All indicators are equally important – heroic assumption, given the different dimensions of the indicators, e.g. (3.6) SMEs introduced organisational innovations (1.1) S&E graduates

13. Equal weight  strong weight Quite a few indicators exhibit strong correlation; most visible in the indicators on intellectual property (which accounts for a fifth of the SII score!)

14. Scores vs. ranking SII2007: Numerical ranking

15. Critique of the political discourse Indicator-driven perception („we are moving down the ranking“) danger of indicator-driven policy: as starting point for policy formulation („we have to improve VC in Austria because EIS demonstrates this to be a major weakness…“) as policy targets („we have to improve in the ranking“, „we want to be top 3/5/10 in the ranking…“) .. Or even of indicator-manipulating („this indicator has to be included / excluded because we perfom well / badly“)

16. Conclusions No ideal ‚catch-all‘ indicator for science or innovation has been developed so far [ – nor could be developed ! ] There is still a lack of clear theoretical models to guide selection and weighting of indicators. Room exists for manipulation by selection, weighting and aggregating indicators. As NIS differ form each other, good policy making in one country may be poor policy making in another one. By relying on composite indicators the structure and the ‚revealed‘ comparative advantage of the countries remain hidden.

17. Conclusions Limited contribution of innovation to short term changes in economic performance  publication of the EIS on an annually basis is too shortsighted Using smoothed data (3-year or longer term averages) Further development of the EIS in order to generate innovation related data (regulation, competion, new firms etc.) … or more radical: skip the synthetic ‚Summary Innovation Index‘ !

18. Conclusions There is very little statistical correlation between a country‘s performance in EIS indicators and its economic performance Any link of innovation performance indicators with innovation policy measures has to take into account the specific institutional, sectoral and economic environment of a country  combine the EIS results with detailed background information on the features of the respective (national) innovation system (e.g. Austria‘s export performance, or shape of capital markets) The OECD has resisted coming up with simple aggregate rank tables of countries‘ innovation performance – and yet still arrives at clear policy recommendations Thus – and most importantly - : take a different policy stance towards the use of EIS ! („Keep cool!“)