Koch taftie-measuring the effects of research


Published on

Presentation for TAFTIE on the use of R&D and innovation statistics in policy development.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Koch taftie-measuring the effects of research

  1. 1. How to consider the effects of research and innovation investments Per M. Koch Head of the Science Project
  2. 2. Why do policy makers look for numbers? <ul><li>To understand the world and how research functions in it </li></ul><ul><li>To develop a story that can be used to convince others to invest in R&D </li></ul><ul><li>We are number fetishists </li></ul><ul><ul><li>The prestige of the natural sciences </li></ul></ul><ul><ul><li>The power of macro economics and the ministries of finance </li></ul></ul>
  3. 3. I am going to talk about: <ul><li>The potential and limitations of evaluating the effects of </li></ul><ul><ul><li>Innovation and research policy measures on the aggregate national level </li></ul></ul><ul><ul><li>Individual policy instruments </li></ul></ul><ul><li>The fact that the complexity of social systems makes these very challenging tasks </li></ul>
  4. 4. Data sources
  5. 5. The Frascati Manual <ul><li>June 1963: OECD NESTI group (National Experts on Science and Technology Indicators) met at Villa Falconieri in Frascati. Italy </li></ul><ul><li>The document sets forth the fundamental definitions (type of activity: basic research, applied research, experimental development; research personnel: researchers, technicians, auxiliary personnel). It primarily deals with measuring the resources devoted to R&D – expenditure and personnel – in the performing sectors: higher education, government, business enterprise, private non-profit organizations. </li></ul>
  6. 6. The Oslo Manual <ul><li>OECD Oslo 1992 : Guidelines for collecting and interpreting technological innovation data. </li></ul><ul><li>Basis for the Eurostat Community Innovation Survey (CIS) </li></ul><ul><li>Revision 2005 3 rd edition: </li></ul><ul><ul><li>Marketing </li></ul></ul><ul><ul><li>Organizational innovation </li></ul></ul><ul><ul><li>Innovation linkages </li></ul></ul>
  7. 7. Canberra Manual <ul><li>NESTI Canberra, 1994: The Manual on the Measurement of Human Resources devoted to S&T (the &quot;Canberra Manual&quot;) </li></ul><ul><li>The OECD and Eurostat, the European Commission </li></ul><ul><li>Definitions of human resources devoted to science and technology in terms of qualification (levels and fields of study) and occupation </li></ul><ul><li>The International Standard Classification of Education (ISCED) </li></ul><ul><li>The International Standard Classification of Occupation (ISCO) </li></ul>
  8. 8. Other data <ul><li>Tax authorities </li></ul><ul><li>Social security agencies </li></ul><ul><li>Higher education institutions </li></ul><ul><li>Accounting data </li></ul><ul><li>Register-based statistical systems using standardized identifiers. </li></ul><ul><ul><li>Problems due to privacy considerations </li></ul></ul>
  9. 9. Beyond evidence based policy development
  10. 10. Evidence Based Medicine <ul><li>Evidence-based medicine (EBM) aims to apply the best available evidence gained from the scientific method to medical decision making. It seeks to assess the quality of evidence of the risks and benefits of treatments (including lack of treatment). (Wikipedia) </li></ul><ul><li>Using techniques from: </li></ul><ul><ul><li>Science </li></ul></ul><ul><ul><li>Engineering </li></ul></ul><ul><ul><li>Statistics </li></ul></ul><ul><ul><li>meta-analysis of medical literature </li></ul></ul><ul><ul><li>risk-benefit analysis </li></ul></ul><ul><ul><li>randomized controlled trials </li></ul></ul>
  11. 11. Evidence-based policy <ul><li>“… helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation” (P Davies, 1999) </li></ul><ul><li>Vs. “opinion-based policy, which relies heavily on either the selective use of evidence … or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices, or speculative conjecture.” (P Davies 2004) </li></ul>
  12. 12. The myth of evidence based policy-development <ul><li>“… policy making must be soundly based on evidence of what works” Professional Policy Making for the Twenty-First Century (UK Cabinet Office, 1999) </li></ul><ul><li>But how to you prove what works? </li></ul><ul><ul><li>Any social and economic phenomena are a result of interaction between a limitless number of factors </li></ul></ul><ul><ul><li>Society is constantly changing, a moving target </li></ul></ul><ul><li>Any interpretation of the reality underlying policy-development will be only that: an interpretation </li></ul><ul><li>Researchers are as fallible as other experts, anchored in their own belief systems and prejudices </li></ul>
  13. 13. There are many beliefs and narratives <ul><li>Different understanding of: </li></ul><ul><ul><li>What society is and how it works </li></ul></ul><ul><ul><li>How research interacts with society </li></ul></ul><ul><ul><li>Common terms (e.g. “innovation”, “research”) </li></ul></ul><ul><ul><li>What the role of science is </li></ul></ul><ul><ul><li>What the best theoretical and methodological foundation for policy development is </li></ul></ul><ul><li>Different cultures have different narratives they use to make sense of their surroundings. </li></ul>
  14. 14. Linear vs. systemic views of innovation
  15. 15. The linear model works perfectly in a policy setting Research and technological development in universities, RTOs and companies gives birth to an idea and relevant new knowledge Companies make use of these ideas in the development of new products and processes The company brings the new product to the market
  16. 16. Why? <ul><li>It is a good story: </li></ul><ul><ul><li>The unselfish scientist striving for the common good </li></ul></ul><ul><ul><li>It is simple: Give us the money and we will give you the results </li></ul></ul><ul><ul><li>By black boxing the whole process it communicates easily. </li></ul></ul><ul><ul><ul><li>You don’t need to know how the computer works to make use of it </li></ul></ul></ul><ul><ul><ul><li>It works well with many of the traditional macroeconomists </li></ul></ul></ul>
  17. 17. But this model leads to one specific interpretation of data <ul><li>A pipe-line model: </li></ul><ul><ul><li>Research > findings > products > profits </li></ul></ul><ul><ul><li>Science > findings > publications > scientific progress </li></ul></ul><ul><li>Main focus on input indicators </li></ul><ul><ul><li>(money, doctorates) </li></ul></ul><ul><li>To the extent there is output data: </li></ul><ul><ul><li>Publications and citations </li></ul></ul><ul><ul><li>Reported number of innovations in companies </li></ul></ul><ul><ul><li>Patents and licenses </li></ul></ul>
  18. 18. Policy consequences of the linear model <ul><li>Input indicators as policy objectives: </li></ul><ul><ul><li>3% R&D of GDP (Barcelona) </li></ul></ul><ul><ul><li>Number of PhDs </li></ul></ul><ul><li>Focus on technology push rather than market pull </li></ul><ul><li>Oversimplified idea of best practice as regards innovation (“More research!”) </li></ul>
  19. 19. The correlation problem R&D-investments Economic growth It is possible to study the correlation between R&D and growth…
  20. 20. The correlation problem R&D-investments Research Economic growth … but are we really studying the effects of research alone?
  21. 21. The correlation problem R&D-investments Research Science Development Learning Innovation Mobility Machinery Technology Education Literature Patents Licenses Design Marketing Branding Culture Regulations Globalization Experience Creativity Economic growth There is nothing wrong in having increased investments in R&D as a policy objective, but it cannot be used as a proxy for innovation “ Total factor productivity?”
  22. 22. We are missing a large part of the picture! <ul><li>Major problem: </li></ul><ul><ul><li>there is no direct correlation between national R&D investments, productivity and economic growth. </li></ul></ul><ul><li>The Norwegian paradox: seemingly low R&D investments, but record high productivity and economic growth </li></ul><ul><li>The Swedish paradox: very high R&D investments, lower productivity and economic growth </li></ul>
  23. 23. Rogers Klein Havelock/Przybylinski The linear model masks the complexity of the role of research in social systems
  24. 24. In-house learning based innovation Following the OECD systemic approach: Research is given a new and broader role as a learning tool market pull Knowledge about customers and competitors The company brings new or altered products, processes or services on to the market Market competences Tacit knowledge Acquired technology Literature Conferences and fairs Recruits Acquired R&D In-house R&D
  25. 25. Company or public institution Learning Networks Innovation Customers and users Suppliers Policy- institutions Financial Institutions R&D Institutions Consultants Public policy Cultural framework International framework Industrial structure Understanding competence flows in the innovation system
  26. 26. Challenges beyond the pipe-line
  27. 27. Indirect use of research <ul><li>Research is embedded in technology and machinery. We need to measure the effect of research in the use of advanced technology in low tech industries. </li></ul><ul><li>Research is embedded in students and employees. We need to measure the effect research based education has on the learning capabilities of men and women. </li></ul>
  28. 28. The role of competence flows <ul><li>Education </li></ul><ul><li>User-driven innovation </li></ul><ul><li>Customer/supplier relationships </li></ul><ul><li>“ Open innovation” and industry collaboration </li></ul><ul><li>KIBS </li></ul><ul><li>Global companies as gateways to the world </li></ul>
  29. 29. We need to get a better understanding of the role of services <ul><li>The largest part of the economy </li></ul><ul><li>Interacts with all of industry </li></ul><ul><li>A very heterogeneous sector </li></ul><ul><li>Often left out of statistics </li></ul>Until now: Industrial policies have been focused on manufacturing, even if 70 percent are employed in services
  30. 30. Understanding the effects of public sector innovation <ul><li>Innovation in the private sector is understood as an investment, in the public sector as an expense </li></ul><ul><li>Innovation in the public sector and the effect on industry </li></ul><ul><li>The effect of social welfare on risk taking and company behavior </li></ul><ul><li>We have no output indicators! </li></ul>
  31. 31. The Copenhagen Manual <ul><li>New Nordic project aimed at developing a statistics manual for public sector innovation </li></ul><ul><ul><li>Danish Agency for Science, Technology and Innovation, Denmark </li></ul></ul><ul><ul><li>Danish Centre for Studies in Research and Research Policy, CFA, </li></ul></ul><ul><ul><li>NIFU STEP, Norway </li></ul></ul><ul><ul><li>RANNIS, Iceland </li></ul></ul><ul><ul><li>Innovation Norway, Norway </li></ul></ul><ul><ul><li>The Research Council of Norway </li></ul></ul><ul><ul><li>DAMVAD, Denmark </li></ul></ul><ul><ul><li>Statistics Finland </li></ul></ul><ul><ul><li>Statistics Norway </li></ul></ul><ul><ul><li>Statistics Denmark </li></ul></ul><ul><ul><li>Statistics Sweden </li></ul></ul><ul><ul><li>In cooperation with: </li></ul></ul><ul><ul><ul><li>OECD NESTI </li></ul></ul></ul><ul><ul><ul><li>NESTA UK </li></ul></ul></ul>
  32. 32. Understanding socio-cultural framework conditions <ul><li>Stable macro-economic framework conditions </li></ul><ul><ul><li>Disciplined fiscal policy </li></ul></ul><ul><ul><li>Competition policy encouraging innovation </li></ul></ul><ul><ul><li>Low taxes </li></ul></ul><ul><ul><li>An open economy </li></ul></ul><ul><li>Socio-cultural framework conditions </li></ul><ul><ul><li>Egalitarian culture with high social mobility </li></ul></ul><ul><ul><li>High wages for blue collar work gives impetus towards innovation (robots, internet banking) </li></ul></ul><ul><ul><li>High educational levels brings flexibility and labor mobility </li></ul></ul><ul><ul><li>An efficient public sector helps industry </li></ul></ul><ul><ul><li>A trustworthy welfare system reduces risk </li></ul></ul><ul><ul><li>Political and social stability gives trust </li></ul></ul>
  33. 33. Understanding socio-cultural framework conditions <ul><li>European Working Conditions Survey (Translearn) The correlation between GDP and employer autonomy </li></ul>Åge Mariussen
  34. 34. We must understand the heterogeneity of innovation systems <ul><li>Industrial structure </li></ul><ul><ul><li>Large vs. small companies </li></ul></ul><ul><ul><li>Low tech vs. high tech </li></ul></ul><ul><ul><li>Traditional industries vs. services </li></ul></ul><ul><li>The knowledge system </li></ul><ul><ul><li>What kind of universities? </li></ul></ul><ul><ul><ul><li>Research? Polytechnics? Humboldtian? </li></ul></ul></ul><ul><ul><li>Other types of research institutions </li></ul></ul><ul><ul><ul><li>Institutes, government labs, academies </li></ul></ul></ul><ul><li>Globalization and integration in the world economy </li></ul><ul><li>Political stability and social welfare </li></ul>
  35. 35. European innovation scoreboard <ul><li>A praiseworthy effort to enlarge the scope of innovation indicators </li></ul><ul><li>But the selection of indicators is slanted towards R&D intensive activities </li></ul><ul><li>The same for the weighing of the indicators underpinning the composite indicator </li></ul>Norway
  36. 36. EIS 2008 This figure reflects a specific view of innovation as much as it does the facts on the ground
  37. 37. Numbers must be accompanied by stories <ul><li>Data is of no value unless they are given in a meaningful context. </li></ul><ul><li>Still: The EU keeps the Innovation Scoreboard but plans scrapping the Trend Chart on Innovation, the tool that interprets the statistics in a national context. </li></ul>
  38. 38. Measuring policy measures
  39. 39. From needs to ex-post impact assessments <ul><li>Needs: Problems to be solved </li></ul><ul><li>Objective: Outcomes intended to be achieved </li></ul><ul><li>Inputs: Resources mobilized </li></ul><ul><li>Processes: Procedures and activities employed </li></ul><ul><li>Outputs: Products and accomplishments </li></ul><ul><li>Outcomes: Changes caused by intervention </li></ul><ul><ul><li>Results: Immediate changes for addressees </li></ul></ul><ul><ul><li>Impacts: Longer term socio-economic consequences </li></ul></ul>OECD DSTI/STP/TIP(2009)18 Henri Delanghe et al Old focus New focus
  40. 40. The use of evaluations <ul><li>Traditional qualitative evaluations are important </li></ul><ul><ul><li>The generate insight into </li></ul></ul><ul><ul><ul><li>Research quality </li></ul></ul></ul><ul><ul><ul><li>The organization of research </li></ul></ul></ul><ul><ul><ul><li>Collaboration with other research institutions </li></ul></ul></ul><ul><ul><li>Say little about the effects on society </li></ul></ul><ul><ul><li>Say little about how the research agenda is shaped through interaction between research and society </li></ul></ul>
  41. 41. Using quantitative measurements for industry oriented policy measures <ul><li>The direct effects on the companies participating </li></ul><ul><ul><li>Economic additionality (Company profits over time) </li></ul></ul><ul><ul><li>Behavioral additionality (learning, absorptive capacity, ability to change) </li></ul></ul><ul><li>The direct effects on other companies and organizations </li></ul><ul><ul><li>Spill-over effects on suppliers (innovation and profits) </li></ul></ul><ul><ul><li>Spill-over effects on customers (innovation, productivity and profits) </li></ul></ul><ul><li>Indirect societal effects </li></ul><ul><ul><li>Profitable companies will expand and buy more goods and services from others </li></ul></ul><ul><ul><li>Their products and services increase productivity in other parts of society </li></ul></ul><ul><ul><li>Their competences are used by others (through for instance mobility of personnel) </li></ul></ul>Increased complexity and difficulty
  42. 42. Intended and unintended consequences <ul><li>We develop policy instruments for a lot of reasons: </li></ul><ul><ul><li>To solve a problem </li></ul></ul><ul><ul><li>To gain political visibility and influence </li></ul></ul><ul><ul><li>Because the Finns are doing it </li></ul></ul><ul><ul><li>To make the minister look good </li></ul></ul><ul><li>Policy instruments have a dynamic of their own </li></ul><ul><ul><li>Based on the competences, interests and initiatives of the people involved (They may use the instrument for a new purpose) </li></ul></ul><ul><ul><li>They may have unexpected positive and negative effects elsewhere in the system </li></ul></ul><ul><li>Policy measures outside the realm of research and innovation may have a positive impact on innovation </li></ul><ul><ul><li>Social security </li></ul></ul>Formal objective Informal objective Unintended effect If you want to ascertain the success of a measure, you have to take all of these into consideration Policy instrument
  43. 43. The role of risk <ul><li>Market failure: “When private rates of return, due to high risk, uncertainty or ‘externalities’, may be sufficiently low to deter private investment in areas of crucial long-term national interest.” </li></ul><ul><li>Public measures must allow for higher risk. </li></ul><ul><li>Still: Policy makers ask for high rate of returns! </li></ul>
  44. 44. RCN’s user-oriented R&D programs <ul><li>To measure the effects of the program participations in the </li></ul><ul><li>Research done by Møreforskning in collaboration with the Research Council of Norway since the middle of the 1990s. </li></ul><ul><li>Led by professor Arild Hervik. </li></ul><ul><li>Based on data from the databases of the RCN and interviews with the companies </li></ul>
  45. 45. To make sure that the right projects are supported <ul><li>Good projects that would have been implemented anyway should not get support </li></ul><ul><li>Low quality projects with no chance of success should be kept out </li></ul><ul><li>You should support good projects where RCN support is vital for its implementation (Input additionality) </li></ul>
  46. 46. Project selection aimed at high social effects (The Jaffe Model)
  47. 47. Understanding the social dynamics of policy instrument development <ul><li>An evaluation of the success of a policy instruments must also include an analysis of its social and political context </li></ul><ul><li>I.e.: we can not only mechanistically measure input and outcomes. We must look into the box of policy development. </li></ul><ul><li>Interim evaluations </li></ul>
  48. 48. Traditional view of evidence based policy learning Independent researchers or experts Civil servants Politicians The wall of disinterested objectivity
  49. 49. An interactive learning arena Independent researchers (experts) Civil servants (experts) Politicians (experts) Stake-holders (experts) They are all experts in their field, some in technology, others in policy
  50. 50. The ultimate challenge <ul><li>Analysis of the national innovation system </li></ul><ul><li>+ policy instrument evaluations </li></ul><ul><li> policy mix reviews </li></ul><ul><ul><li>Green papers </li></ul></ul><ul><ul><li>White papers </li></ul></ul><ul><ul><li>OECD and EU innovation policy country reports </li></ul></ul><ul><li>VINNOVA’s Portfolio Approach </li></ul><ul><li>In order to evaluate public initiatives aimed at promoting innovation, it is necessary to focus on the combined impacts of public efforts. </li></ul>
  51. 51. Conclusions <ul><li>There is no simple, mathematical model that can be used to measure all effects of research and innovation. </li></ul><ul><li>We need to combine methodologies, both quantitative and qualitative. </li></ul><ul><li>We are all experts here, and this can not be left to the economists or statisticians </li></ul><ul><li>We have to build common learning arenas, for measuring and for developing an understanding the context of innovation </li></ul>