Slides by Giorgio Sirilli (CERIS-CNR, ROARS) CONACYT Seminar on evaluation of STI policies Mexico City June 19, 2014

1. CONACYT
Seminar on evaluation of STI policies
Mexico City
June 19, 2014
S&T policy in Italy:
The role of evaluation
Giorgio Sirilli
CERIS – CNR
ROARS

2. 2
Outline of the presentation
Science policy in Italy and bejond
The most relevant evaluation exercises:
lessons learned
The use of evaluation in S&T policy making
Concluding remarks

3. 3
Italy
60 million inhabitants

4. 4
Italy, a founding member of he European Union
Italy’s policies, including R&D policy, are heavily
affected by the the European Union

5. 5
Some data on Italy
R&D expenditure = 19,834 million euro
Researchers = 106,151 (full-time equivalent)
R&D/GDP = 1.25%
90 universities
40 Public Research Agencies (PRAs)

6. 6
Sector
R&D
expenditure
Percentage
Government 2.723 13.8
Privare non profit 621 3.1
Business enperprises 10.813 54.5
University 5.677 28.6
Total 19.834 100,0
Source: ISTAT
R&D expenditure by performing sector - Italy
2012 (million euro)
The
Lisbon
target
66.6%

7. 7
Sector Number Percentage
Government 18,780 17.7
Privare non profit 3,735 3.5
Business enperprises 39,808 37.5
University 43,828 41.3
Total 106,151 100.0
Source: ISTAT
Researchers by performing sector - Italy - 2011
(number in equivalent full time)

8. 8
Science and technology policy in Italy:
main actors
Junior minister for Research (1963)
Ministry of Universities and Research (1989)
Ministry of industry (Economic development)
Junior minister for Innovation (2000)
Ministry of Economy

9. 9
Ministry of universities and research
Committee for Research Evaluation (CIVR)
National Committee for the Evaluation of the
University System (CNUSV)
National Agency for the Evaluation of
Universities and Research (ANVUR)
Universities
National Research Council
Other research agencies
Ministry of economic development
The evaluation system in Italy: main actors

10. Start: 2011
Members of the Editorial board: 14
Collaborators: 217
Contacts: 8 million (November 2011 – June 2014)
Average daily contacts: 8,000 November 2011; 13,000 in 2014)
Articles published: 1,627
Comments by readers: 24,880
ROARS is ranked 8° among the top cultural national blogs
ROARS, a genuine expression of democracy and
participation, has become a very important player in the
policy debate and in policy making
A new player in S&T policy

11. 11
The neo-conservative wave of the 1980s
Ronald Reagan Margaret Thatcher

12. 12
The new catchwords
New public management
Value for money
Accountability
Relevance

13. 13
The UK
“All areas of public expenditure should
demonstrate ‘value for money’”
Thatcher’s three Es:
economy
efficiency
effectiveness

14. 14
The UK
Research Assessment Exercise (RAE)
Research Excellence Framework (REF) (impact)
“The REF will over time doubtless become more
sophisticated and burdensome. In short we are
creating a Frankenstein monster” (Ben Martin)

15. 15
“Why shoud we pay
researchers if we make the
best shoes in the world?”
Silvio Berlusconi
“Culture does not provide food”
(Con la cultura non si mangia)
Giulio Tremonti
The neo-conservative wave in Italy in the 1990s

16. 16
The neo-conservative wave in Italy in the 1990s
Letizia Moratti
Italian minister of education and research
“You first show that you use efficiently and
effectively the public money, then we will open
the strings of the purse” - Never happened

17. 17
Science policy in Italy
The neo-liberal approach after Reagan and Thatcher
The legitimation of R&D and higher education

18. 18
The Entetreneurial State
The “visible hand” of the state
Innovations
micro hard disk
silicon
multi touch
inteernet
gps
siri
lcd screen
litium batteries

19. 19
Science policy in Italy
The neo-liberal approach after Reagan and Thatcher
The legitimation of R&D and higher education
The autonomy of University and PRAs
- the (mis)use of autonomy
- the impact on the scientific community
The reduction of resources
The role of evaluation as a S&T policy instrument

20. 20
Is the Italian scientific system efficient and effective?
The answer is YES

21. 21
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab.
6.1)
(dati ISI Web of Knowledge, Thomson-Reuters)
#papers/millionUSD(PPP) Number of publications (2004)/unit of R&D expenditure

22. 22
Average annual growth of publications (2004-2010)
(%)
Fonte: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 3.2)
•(dati ISI Web of Knowledge, Thomson-Reuters)
http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

23. Number of citations to the Web of Knowledge articles (2004-2010)
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)
(dati ISI Web of Knowledge, Thomson-Reuters)
•http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

24. 24
Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)
(dati ISI Web of Knowledge, Thomson-Reuters)
•http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf
#cites/millionUSD(PPP)
Number of citations (articles 2004)/unit of R&D expenditure

25. 25
The reduction of public resources for R&D
Data on government budget appropriations for R&D in
Italy show that from 2009 to 2012 there has been a
9.8% decrease in absolute values (from €9,778 million
to €8,822 million) and a 12.7% decrease in real terms
In particular, research funds granted to universities by
the ministry of education and research have declined
from €366 to €96 million; the Prin shrunk from €170
million in 2010 and 2011 to €38 million in 2012, and
may drop to zero in 2014, while the Ordinary fund for
higher education, which provides block funding for
universities, shrunk from €7.5 billion in 2009 to €6.6
billion in 2013

26. 26
The difficult situation of the National Research
Council
In the period 2005 – 2014 the overall financing from the
ministry of education decreased in real terms of 5.3% –
the overall budget of the agency is about 1 billion euro.
The financing has witnessed a dramatic structural change: the
block grant which is managed autonomously by CNR has
decreased by 20.4% while the financing linked to specific
projects has rocketed by 1,675%.
In 2014 the block grant of €500 million will not even be
enough to cover the obligatory expenses (salaries and fixed
costs) of €620 million. This means that CNR has lost its
autonomy and that, in order to survive, the agency will
have to activate contracts linked to research and technical
services both from public and private sources

27. 27
Established in 2011
A government agency, not an authority
The relationship with MIUR (Ministry of education,
universities and research)
ANVUR activities:
1. Evaluation of the Quality of Research (EQR)
2. National Scientific Qualification (NSQ)
3. Accreditation of universities (AVA)
Evaluation

28. 28
Model: Research Assessment Exercise (RAE)
Objective: Evaluation of Areas, Research structures and
Departments (not of researchers)
Reference period: 2004-2010
Start: 2011
Actors:
- ANVUR (National Agency for the Evaluation of Universities
and Research Institutes)
- GEV (Evaluation Groups) (#14) (450 experts involved plus
referees)
- Research structures (universities, research agencies)
- Departments
- Subjects evaluated: researchers (university teachers and
PRA researchers)
ANVUR Evaluation of the Quality of Research

29. 29
Researchers’ products to be evaluated
- journal articles
- books and book chapters
- patents
- designs, exhibitions, software, manufactured items,
prototypes, etc.
University teachers: 3 “products” over the period 2004-
2010
Public Research Agencies researchers: 6 “products” over
the period 2004-2010
Scores: from 1 (excellent) to -1 (missing)
ANVUR Evaluation of the Quality of Research

30. 30
Outcome:
- mobilisation of the scientific community
- heavily criticised (methodologies, too fast)
- a measure of effectiveness
- considered a first exercise to be improved
- used to assign resources by ministry
ANVUR Evaluation of the Quality of Research

31. 31
Bibliometrics vs peer review
NSE, SSH
Journals: A, B, C
Median
Examiners
National Scientific Qualification

32. 32
Was it worth?
69,000 applications
46,000 habilitated
1,100 legal actions
cost per habilitation (5,000 euro per candidate
habilitated; 25, 000 euro per candidate hired)
the link between VQR and ASN
National Scientific Qualification

33. 33
Under way
Strong opposition from universities
Accreditation of universities

34. 34
Too powerful?
Too bureaucratic?
Playing a political game?
The role of ANVUR

35. 35
The relationship between ANVUR and ROARS
A “frank”, and sometimes tough, dialogue
ROARS’s methodological suggestions often accepted

36. Some estimates
VQR 300 million euro (180,000 “products”)
NSQ: 1126 million euro (46,000 candidates)
RAE: 126 million euro (180.000 “products”)
ANVUR budget: 9 million euro
Rule of thumb: less than 1% of R&D budget devoted
to its evaluation – this is not often the case
Evaluation is expensive

37. 37
Knowledge: the bundle

38. University
teaching
research
“third mission”
(60 indicators)
The learned society
Research agencies
research
problem solving
management

39. The Frankenstein monster (Martin)
League tables
Competition vs cooperation of scientists
Opportunistic behaviour
The split of the academic community (the good and
the bad guys)
Peer review vs bibliometrics
NSE vs SSH
The threat of the equilibrium amongst the components
of the bundle (research at the expense of other
activities)
The other side of the coin of evaluation

40. Performance based research funding systems are
national systems of ex-post university research
output evaluation used to inform distribution of
funding
Performance based research funding systems

41. 41
“The rationale of performance funding is that funds should
flow to institutions where performance is manifest:
‘performing’ institutions should receive more income than
lesser performing institutions, which would provide
performers with a competitive edge and would stimulate
less performing institutions to perform. Output should be
rewarded, not input.”
Herbst, 2007
Performance based research funding systems
The rationale

42. 42
Share of university funding dependent on
Performance-Based Research Funding Systems
Country Share (%) of what
Australia 6 Total revenue
Italy 2 Block grant
New Zeland 10 Block grant
Norway 2 Total funding
Slovak Republic 15 Total funding
UK 25 Research support
Hicks D., Reseach Policy (2012)

43. 43
Share of university funding dependent on
Performance-Based Research Funding Systems
Hicks D., Reseach Policy (2012)
“The distribution of university research funding is
something of an illusion”
“It is the competition for prestige that creates powerful
incentives within university systems”
“Performance-based research funding systems aim at
excellence: they may compromise other important
values such as equity and diversity”
“Control by professional elites”

44. 44
Few evaluations carried out in the last years
Some analyses of innovation policies using innovation data
(CIS)
Some analyses of the impact of a specific measure for the
financing of R&S in firms through a counterfactual
method
Additionality difficult to evaluate
Impact difficult to evaluate
No major pressure (or interest) from policy makers to
evaluate support to industry
Evaluation of public policies and programs in Italy

45. 45
Main methodological results of the Italian
experience:
- impact difficult to measure
- very few exercises carried out
- indicators need improvement
Evaluation of the “Plans for development of
scientific and technological networks”

46. 46
Concluding remarks
The Italian S&T system is under pressure
Legitimation of S&T and higher education
Evaluation as a key policy instrument
Evaluation was introduced lately but it is expected to stay
Evaluation was used in an ideological way
Evaluation exercises heavily criticised from a
methodological point of view
Impact on the scientific community and on researchers’
behavior
Too much evaluation is harmful
Misuse of the results of evaluation
Evaluation is expensive
Concentration in “excellent” institutions or spreading?
Paradox: the S&T Italian system is good

47. Thank you for attention