Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CONACYT
Seminar on evaluation of STI policies
Mexico City
June 19, 2014
S&T policy in Italy:
The role of evaluation
Giorgi...
2
Outline of the presentation
Science policy in Italy and bejond
The most relevant evaluation exercises:
lessons learned
T...
3
Italy
60 million inhabitants
4
Italy, a founding member of he European Union
Italy’s policies, including R&D policy, are heavily
affected by the the Eu...
5
Some data on Italy
R&D expenditure = 19,834 million euro
Researchers = 106,151 (full-time equivalent)
R&D/GDP = 1.25%
90...
6
Sector
R&D
expenditure
Percentage
Government 2.723 13.8
Privare non profit 621 3.1
Business enperprises 10.813 54.5
Univ...
7
Sector Number Percentage
Government 18,780 17.7
Privare non profit 3,735 3.5
Business enperprises 39,808 37.5
University...
8
Science and technology policy in Italy:
main actors
Junior minister for Research (1963)
Ministry of Universities and Res...
9
Ministry of universities and research
Committee for Research Evaluation (CIVR)
National Committee for the Evaluation of ...
Start: 2011
Members of the Editorial board: 14
Collaborators: 217
Contacts: 8 million (November 2011 – June 2014)
Average ...
11
The neo-conservative wave of the 1980s
Ronald Reagan Margaret Thatcher
12
The new catchwords
New public management
Value for money
Accountability
Relevance
13
The UK
“All areas of public expenditure should
demonstrate ‘value for money’”
Thatcher’s three Es:
economy
efficiency
e...
14
The UK
Research Assessment Exercise (RAE)
Research Excellence Framework (REF) (impact)
“The REF will over time doubtles...
15
“Why shoud we pay
researchers if we make the
best shoes in the world?”
Silvio Berlusconi
“Culture does not provide food...
16
The neo-conservative wave in Italy in the 1990s
Letizia Moratti
Italian minister of education and research
“You first s...
17
Science policy in Italy
The neo-liberal approach after Reagan and Thatcher
The legitimation of R&D and higher education
18
The Entetreneurial State
The “visible hand” of the state
Innovations
micro hard disk
silicon
multi touch
inteernet
gps
...
19
Science policy in Italy
The neo-liberal approach after Reagan and Thatcher
The legitimation of R&D and higher education...
20
Is the Italian scientific system efficient and effective?
The answer is YES
21
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab.
6.1)
(dati ISI Web of Knowledge, Thomson-Reuters)
#pap...
22
Average annual growth of publications (2004-2010)
(%)
Fonte: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 3...
Number of citations to the Web of Knowledge articles (2004-2010)
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 201...
24
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)
(dati ISI Web of Knowledge, Thomson-Reuters)
•htt...
25
The reduction of public resources for R&D
Data on government budget appropriations for R&D in
Italy show that from 2009...
26
The difficult situation of the National Research
Council
In the period 2005 – 2014 the overall financing from the
minis...
27
Established in 2011
A government agency, not an authority
The relationship with MIUR (Ministry of education,
universiti...
28
Model: Research Assessment Exercise (RAE)
Objective: Evaluation of Areas, Research structures and
Departments (not of r...
29
Researchers’ products to be evaluated
- journal articles
- books and book chapters
- patents
- designs, exhibitions, so...
30
Outcome:
- mobilisation of the scientific community
- heavily criticised (methodologies, too fast)
- a measure of effec...
31
Bibliometrics vs peer review
NSE, SSH
Journals: A, B, C
Median
Examiners
National Scientific Qualification
32
Was it worth?
69,000 applications
46,000 habilitated
1,100 legal actions
cost per habilitation (5,000 euro per candidat...
33
Under way
Strong opposition from universities
Accreditation of universities
34
Too powerful?
Too bureaucratic?
Playing a political game?
The role of ANVUR
35
The relationship between ANVUR and ROARS
A “frank”, and sometimes tough, dialogue
ROARS’s methodological suggestions of...
Some estimates
VQR 300 million euro (180,000 “products”)
NSQ: 1126 million euro (46,000 candidates)
RAE: 126 million euro ...
37
Knowledge: the bundle
University
teaching
research
“third mission”
(60 indicators)
The learned society
Research agencies
research
problem solvin...
The Frankenstein monster (Martin)
League tables
Competition vs cooperation of scientists
Opportunistic behaviour
The split...
Performance based research funding systems are
national systems of ex-post university research
output evaluation used to i...
41
“The rationale of performance funding is that funds should
flow to institutions where performance is manifest:
‘perform...
42
Share of university funding dependent on
Performance-Based Research Funding Systems
Country Share (%) of what
Australia...
43
Share of university funding dependent on
Performance-Based Research Funding Systems
Hicks D., Reseach Policy (2012)
“Th...
44
Few evaluations carried out in the last years
Some analyses of innovation policies using innovation data
(CIS)
Some ana...
45
Main methodological results of the Italian
experience:
- impact difficult to measure
- very few exercises carried out
-...
46
Concluding remarks
The Italian S&T system is under pressure
Legitimation of S&T and higher education
Evaluation as a ke...
Thank you for attention
Upcoming SlideShare
Loading in …5
×

Science &Technology policy in Italy: The role of evaluation

2,895 views

Published on

Slides by Giorgio Sirilli (CERIS-CNR, ROARS)
CONACYT
Seminar on evaluation of STI policies
Mexico City
June 19, 2014

Published in: Science
  • Be the first to comment

  • Be the first to like this

Science &Technology policy in Italy: The role of evaluation

  1. 1. CONACYT Seminar on evaluation of STI policies Mexico City June 19, 2014 S&T policy in Italy: The role of evaluation Giorgio Sirilli CERIS – CNR ROARS
  2. 2. 2 Outline of the presentation Science policy in Italy and bejond The most relevant evaluation exercises: lessons learned The use of evaluation in S&T policy making Concluding remarks
  3. 3. 3 Italy 60 million inhabitants
  4. 4. 4 Italy, a founding member of he European Union Italy’s policies, including R&D policy, are heavily affected by the the European Union
  5. 5. 5 Some data on Italy R&D expenditure = 19,834 million euro Researchers = 106,151 (full-time equivalent) R&D/GDP = 1.25% 90 universities 40 Public Research Agencies (PRAs)
  6. 6. 6 Sector R&D expenditure Percentage Government 2.723 13.8 Privare non profit 621 3.1 Business enperprises 10.813 54.5 University 5.677 28.6 Total 19.834 100,0 Source: ISTAT R&D expenditure by performing sector - Italy 2012 (million euro) The Lisbon target 66.6%
  7. 7. 7 Sector Number Percentage Government 18,780 17.7 Privare non profit 3,735 3.5 Business enperprises 39,808 37.5 University 43,828 41.3 Total 106,151 100.0 Source: ISTAT Researchers by performing sector - Italy - 2011 (number in equivalent full time)
  8. 8. 8 Science and technology policy in Italy: main actors Junior minister for Research (1963) Ministry of Universities and Research (1989) Ministry of industry (Economic development) Junior minister for Innovation (2000) Ministry of Economy
  9. 9. 9 Ministry of universities and research Committee for Research Evaluation (CIVR) National Committee for the Evaluation of the University System (CNUSV) National Agency for the Evaluation of Universities and Research (ANVUR) Universities National Research Council Other research agencies Ministry of economic development The evaluation system in Italy: main actors
  10. 10. Start: 2011 Members of the Editorial board: 14 Collaborators: 217 Contacts: 8 million (November 2011 – June 2014) Average daily contacts: 8,000 November 2011; 13,000 in 2014) Articles published: 1,627 Comments by readers: 24,880 ROARS is ranked 8° among the top cultural national blogs ROARS, a genuine expression of democracy and participation, has become a very important player in the policy debate and in policy making A new player in S&T policy
  11. 11. 11 The neo-conservative wave of the 1980s Ronald Reagan Margaret Thatcher
  12. 12. 12 The new catchwords New public management Value for money Accountability Relevance
  13. 13. 13 The UK “All areas of public expenditure should demonstrate ‘value for money’” Thatcher’s three Es: economy efficiency effectiveness
  14. 14. 14 The UK Research Assessment Exercise (RAE) Research Excellence Framework (REF) (impact) “The REF will over time doubtless become more sophisticated and burdensome. In short we are creating a Frankenstein monster” (Ben Martin)
  15. 15. 15 “Why shoud we pay researchers if we make the best shoes in the world?” Silvio Berlusconi “Culture does not provide food” (Con la cultura non si mangia) Giulio Tremonti The neo-conservative wave in Italy in the 1990s
  16. 16. 16 The neo-conservative wave in Italy in the 1990s Letizia Moratti Italian minister of education and research “You first show that you use efficiently and effectively the public money, then we will open the strings of the purse” - Never happened
  17. 17. 17 Science policy in Italy The neo-liberal approach after Reagan and Thatcher The legitimation of R&D and higher education
  18. 18. 18 The Entetreneurial State The “visible hand” of the state Innovations micro hard disk silicon multi touch inteernet gps siri lcd screen litium batteries
  19. 19. 19 Science policy in Italy The neo-liberal approach after Reagan and Thatcher The legitimation of R&D and higher education The autonomy of University and PRAs - the (mis)use of autonomy - the impact on the scientific community The reduction of resources The role of evaluation as a S&T policy instrument
  20. 20. 20 Is the Italian scientific system efficient and effective? The answer is YES
  21. 21. 21 Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 6.1) (dati ISI Web of Knowledge, Thomson-Reuters) #papers/millionUSD(PPP) Number of publications (2004)/unit of R&D expenditure
  22. 22. 22 Average annual growth of publications (2004-2010) (%) Fonte: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 3.2) •(dati ISI Web of Knowledge, Thomson-Reuters) http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
  23. 23. Number of citations to the Web of Knowledge articles (2004-2010) Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3) (dati ISI Web of Knowledge, Thomson-Reuters) •http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
  24. 24. 24 Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3) (dati ISI Web of Knowledge, Thomson-Reuters) •http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf #cites/millionUSD(PPP) Number of citations (articles 2004)/unit of R&D expenditure
  25. 25. 25 The reduction of public resources for R&D Data on government budget appropriations for R&D in Italy show that from 2009 to 2012 there has been a 9.8% decrease in absolute values (from €9,778 million to €8,822 million) and a 12.7% decrease in real terms In particular, research funds granted to universities by the ministry of education and research have declined from €366 to €96 million; the Prin shrunk from €170 million in 2010 and 2011 to €38 million in 2012, and may drop to zero in 2014, while the Ordinary fund for higher education, which provides block funding for universities, shrunk from €7.5 billion in 2009 to €6.6 billion in 2013
  26. 26. 26 The difficult situation of the National Research Council In the period 2005 – 2014 the overall financing from the ministry of education decreased in real terms of 5.3% – the overall budget of the agency is about 1 billion euro. The financing has witnessed a dramatic structural change: the block grant which is managed autonomously by CNR has decreased by 20.4% while the financing linked to specific projects has rocketed by 1,675%. In 2014 the block grant of €500 million will not even be enough to cover the obligatory expenses (salaries and fixed costs) of €620 million. This means that CNR has lost its autonomy and that, in order to survive, the agency will have to activate contracts linked to research and technical services both from public and private sources
  27. 27. 27 Established in 2011 A government agency, not an authority The relationship with MIUR (Ministry of education, universities and research) ANVUR activities: 1. Evaluation of the Quality of Research (EQR) 2. National Scientific Qualification (NSQ) 3. Accreditation of universities (AVA) Evaluation
  28. 28. 28 Model: Research Assessment Exercise (RAE) Objective: Evaluation of Areas, Research structures and Departments (not of researchers) Reference period: 2004-2010 Start: 2011 Actors: - ANVUR (National Agency for the Evaluation of Universities and Research Institutes) - GEV (Evaluation Groups) (#14) (450 experts involved plus referees) - Research structures (universities, research agencies) - Departments - Subjects evaluated: researchers (university teachers and PRA researchers) ANVUR Evaluation of the Quality of Research
  29. 29. 29 Researchers’ products to be evaluated - journal articles - books and book chapters - patents - designs, exhibitions, software, manufactured items, prototypes, etc. University teachers: 3 “products” over the period 2004- 2010 Public Research Agencies researchers: 6 “products” over the period 2004-2010 Scores: from 1 (excellent) to -1 (missing) ANVUR Evaluation of the Quality of Research
  30. 30. 30 Outcome: - mobilisation of the scientific community - heavily criticised (methodologies, too fast) - a measure of effectiveness - considered a first exercise to be improved - used to assign resources by ministry ANVUR Evaluation of the Quality of Research
  31. 31. 31 Bibliometrics vs peer review NSE, SSH Journals: A, B, C Median Examiners National Scientific Qualification
  32. 32. 32 Was it worth? 69,000 applications 46,000 habilitated 1,100 legal actions cost per habilitation (5,000 euro per candidate habilitated; 25, 000 euro per candidate hired) the link between VQR and ASN National Scientific Qualification
  33. 33. 33 Under way Strong opposition from universities Accreditation of universities
  34. 34. 34 Too powerful? Too bureaucratic? Playing a political game? The role of ANVUR
  35. 35. 35 The relationship between ANVUR and ROARS A “frank”, and sometimes tough, dialogue ROARS’s methodological suggestions often accepted
  36. 36. Some estimates VQR 300 million euro (180,000 “products”) NSQ: 1126 million euro (46,000 candidates) RAE: 126 million euro (180.000 “products”) ANVUR budget: 9 million euro Rule of thumb: less than 1% of R&D budget devoted to its evaluation – this is not often the case Evaluation is expensive
  37. 37. 37 Knowledge: the bundle
  38. 38. University teaching research “third mission” (60 indicators) The learned society Research agencies research problem solving management
  39. 39. The Frankenstein monster (Martin) League tables Competition vs cooperation of scientists Opportunistic behaviour The split of the academic community (the good and the bad guys) Peer review vs bibliometrics NSE vs SSH The threat of the equilibrium amongst the components of the bundle (research at the expense of other activities) The other side of the coin of evaluation
  40. 40. Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding Performance based research funding systems
  41. 41. 41 “The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input.” Herbst, 2007 Performance based research funding systems The rationale
  42. 42. 42 Share of university funding dependent on Performance-Based Research Funding Systems Country Share (%) of what Australia 6 Total revenue Italy 2 Block grant New Zeland 10 Block grant Norway 2 Total funding Slovak Republic 15 Total funding UK 25 Research support Hicks D., Reseach Policy (2012)
  43. 43. 43 Share of university funding dependent on Performance-Based Research Funding Systems Hicks D., Reseach Policy (2012) “The distribution of university research funding is something of an illusion” “It is the competition for prestige that creates powerful incentives within university systems” “Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity” “Control by professional elites”
  44. 44. 44 Few evaluations carried out in the last years Some analyses of innovation policies using innovation data (CIS) Some analyses of the impact of a specific measure for the financing of R&S in firms through a counterfactual method Additionality difficult to evaluate Impact difficult to evaluate No major pressure (or interest) from policy makers to evaluate support to industry Evaluation of public policies and programs in Italy
  45. 45. 45 Main methodological results of the Italian experience: - impact difficult to measure - very few exercises carried out - indicators need improvement Evaluation of the “Plans for development of scientific and technological networks”
  46. 46. 46 Concluding remarks The Italian S&T system is under pressure Legitimation of S&T and higher education Evaluation as a key policy instrument Evaluation was introduced lately but it is expected to stay Evaluation was used in an ideological way Evaluation exercises heavily criticised from a methodological point of view Impact on the scientific community and on researchers’ behavior Too much evaluation is harmful Misuse of the results of evaluation Evaluation is expensive Concentration in “excellent” institutions or spreading? Paradox: the S&T Italian system is good
  47. 47. Thank you for attention

×