Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ripensare la valutazione


Published on

SottoSopra 16.3.2019

Published in: Education
  • Be the first to comment

  • Be the first to like this

Ripensare la valutazione

  1. 1. Ripensare la valutazione Francesco Sylos Labini Enrico Fermi Center Institute for Complex Systems, CNR 
  2. 2. On 5 November (2008), the Queen must have embarrassed the economists at LSE by asking her Question, which presupposed that all of them had failed to foresee the biggest Global Financial Crisis since 1929. The Queen’s Question clearly implied that something had gone seriously wrong with research in economics. Yet in the results of the Research Assessment Exercise, published the next month, research in economics was declared to be, by some margin, the best research in any subject in the UK. Moreover research in economics at LSE was given a grade of 3.55 out of 4. Who was right in their judgement? The Queen or the RAE committee?
  3. 3. •Research Assessment Systems began with the Research Assessment Exercise (or RAE) which was introduced into the UK by Thatcher in 1986 and continued by Blair. • At intervals of a few years, RAEs are carried out in all the universities of the UK. • The first step is to appoint a committee of assessors in each subject. These assessors are usually academics working in the field in question in the UK. Next most members of each department in a subject have to select a set of pieces of their research. • The department then submits all these pieces of research produced by its members to the assessment committee. The members of the committee study this research output, and, on its basis, grade the department on a scale running from very good downwards. RAE/REF
  4. 4. •It should be pointed out that the RAE is a costly operation. •First of all it involves a lot of work which has to be carried out by professional administrators. Very often universities have had to appoint extra administrative staff to deal with the volume of work generated by the RAE. •Secondly, the extra work involved in the RAE takes up a great deal of academics’ time which, in the absence of the RAE, could be spent on the more productive activities of research and teaching. As the time of academics also costs money, this too adds to the cost of the RAE. RAE/REF
  5. 5. •The question then naturally arises as to whether this expensive procedure has actually improved the research produced in the UK. Strange to say this question is rarely asked. RAE/REF
  6. 6. •Academics devote themselves with great energy to evaluating each other’s work, but seem to be little concerned with evaluating an important government policy. Perhaps the reason for this is that it seems at first sight rather obvious that the RAE should improve the UK’s research output. •The procedure conforms to common sense. If we want to improve research, we should first find out who is doing good research and then give funding to the good researchers while withdrawing funding from the bad researchers. •The RAE appears at first sight to be doing just this, and so the conclusion seems unavoidable that introducing such a system will improve research output. In social life, however, things are rarely simple, and judgements based on common sense can often mislead. RAE/REF
  7. 7. •The very costly RAE is designed to improve the research output of the UK, but could it be having the opposite effect? Could it be making the research output of the UK worse instead of better? RAE/REF
  8. 8. Kuhn in his (1962) The Structure of Scientific Revolutions presents a view of the natural sciences which has become very well-known and quite widely accepted.
  9. 9. Tempo Conoscenza Progresso
  10. 10. •According to Kuhn, mature natural sciences develop for the most part in a manner which he describes as ‘normal science’. During a period of normal science, all the scientists working in the field accept the same framework of assumptions which Kuhn calls a ‘paradigm’.
  11. 11. •However, these periods of normal science are, from time to time, interrupted by scientific revolutions in which the dominant paradigm of the field is overthrown and replaced by a new paradigm.
  12. 12. •The difference between the natural sciences and the social sciences can be put as follows. In the natural sciences, outside revolutionary periods, all the scientists accept the same paradigm. •In the social sciences, however, social scientists are divided into competing schools. Each school has its own paradigm, but these paradigms are often very different from each other.
  13. 13. •The contrast is between a single paradigm and a multi-paradigm situation.
  14. 14. •As far as the natural sciences are concerned, we can illustrate this with the example of theoretical physics. In this field, all scientists accept a paradigm whose core consists of relativity theory and quantum mechanics.
  15. 15. •However, they would argue, relativity and quantum mechanics work very well, and so it is sensible to accept them for the time being. If we turn now to economics we find a very different situation
  16. 16. The economics community is divided into different schools. The members of each of these schools may indeed share a paradigm, but the paradigm of one school can be very different from that of another. Moreover the members of one school are often extremely critical of the views of members of another school. The school of economics which has the most adherents at present is neoclassical economics.
  17. 17. These different schools are associated with different political ideologies: Neo-Classical Economics, the various versions of Keynesianism, and Marxist Economics. These schools are arranged on a political spectrum running from the right to the left.
  18. 18. •Our examination of the community of researchers in economics has led us to the following picture. This community is divided into a number of different schools of thought A, B, C, …, each with its own paradigm. •The members of each school have a very low opinion of the research work produced by the other schools. •Now if a Research Assessment System (or RAS) is applied to such a community, what result will it give?
  19. 19. •The research work of the members of whichever school has the largest number of members will receive the highest valuation. So if school A is in the majority, the members of school A will receive the highest valuation. If school B is in the majority, then the members of school B will receive the highest valuation, and so on . •The valuation received by the members of a particular school X will be roughly proportional to the number of members of that school
  20. 20. • I have argued that the effect of a research assessment system on economics is to drive the economics community away from pluralism and towards monism. • I have also argued that a pluralistic economics community is much more likely to produce good research than a monistic one. • Putting these two claims together, the conclusion is that a research assessment will make research in economics worse rather than better. Hence there is a strong case for abolishing research assessment systems in economics where they exist, and not introducing any new ones.
  21. 21. Competition Quality
  22. 22. •Forma di controllo sulla scienza da parte di organismi di governo: opportunità politica del lavoro dello scienziato •Distribuzione delle risorse: Finanziamento delle ricerce più “utili” alla società e di maggiore qualità Valutazione, politica e ricerca
  23. 23. Se una ipotetica commissione che avesse dovuto decidere del finanziamento di un “progetto” al tempo T0 che poi ha portato al “Nobel” al tempo T1>T0, ed avesse applicato un dato criterio (ad esempio bilbiometrico) avrebbe preso la decisione giusta? 29 Valutare la valutazione: come?
  24. 24. 30
  25. 25. 31 Analisi dei criteri di valutazione da un punto di vista storico
  26. 26. 32 Strategia 1: divisione della torta sulle eccellenze
  27. 27. Excellence and reality Rewarding what is today recognised as excellence is trivial !
  28. 28. “Reinforce excellence, dynamism and creativity of European research"
  29. 29. “ERC funds excellent researchers” ERC Nobel Prize
  30. 30. • Competition • Evaluation • Meritocracy The excellence dogma
  31. 31. ‘Harvard Here’ Model • “Worldclass university as the panacea for ensuring success in the global economy. • Continuous evaluation of global capacity and potential • To conform to indicators set by global rankings: • governments and institutions make profound changes to their higher education systems, • pursue more elite agendas, • alter their education programmes, • privilege some disciplines and fields.
  32. 32. ‘Harvard Here’ Model
  34. 34. • All criteria used are only loosely connected with what they intended to capture. • Several arbitrary parameters and many micro-decisions that are not documented. • Flawed and nonsensical aggregation method • «any of our MCDM (Multiple Criteria Decision Making) student that would have proposed such a methodology in her Master’s Thesis would have surely failed according to our own standards»
  35. 35. • Harvard operating expenses = 44% founds of all Italian universities • Harvard has 21,000 students  130,000 euro/student • Typically EU: 10,000 euro/student 66 Italian universities BillionsofEuros
  36. 36. ‘Harvard Here’ Model
  37. 37. • Competition • Evaluation • Meritocracy The excellence dogma
  38. 38. • Increasing number of scientific articles but more rapid increasing number of retractions • Increasing role of tecno-evaluation • Large number of PhD and Postdoc with low salaries and little possibility of obtaining a permanent position • Small number of élite researchers The international crisis: some evidences
  39. 39. • Competition • Evaluation • Meritocracy The excellence dogma
  40. 40. EC research funding • About 10% of National funding • Imitation at the National level of the EC “best practices” • Horizon 2020 (60 billions euros) • Top-down research lines • Curiosity driven programs: ERC / Marie Curie fellowships Are these really “best practices” for science?
  41. 41. 7° EU Research Framework Program IN OUT MillionsofEuro
  42. 42. 62 Strategia 1: divisione della torta sulle eccellenze Strategia 2: spazio alle fluttuazioni
  43. 43. • Ricercatori (quasi) outsiders • Laboratorio/università marginale • Sorpresa rispetto stato dell’arte Qualche scoperta
  44. 44. Superconduttività ad alta temperatura Premio Nobel in Fisica 1987
  45. 45. Superconduttività ad alta temperatura Premio Nobel in Fisica 1987
  46. 46. Superconduttività ad alta temperatura Premio Nobel in Fisica 1987
  47. 47. Microscopio a effetto Tunnel Premio Nobel per la Fisica 1987
  48. 48. Microscopio a effetto Tunnel Premio Nobel per la Fisica 1987
  49. 49. Magneto Resistenza Gigante Premio Nobel in Fisica 2007
  50. 50. Magneto Resistenza Gigante
  51. 51. Magneto Resistenza Gigante
  52. 52. Scanning Tunneling Microscope
  53. 53. Magneto Resistenza Gigante
  54. 54. Grafene Premio Nobel in fisica 2010
  55. 55. 2010 Grafene
  56. 56. 2010 Grafene
  57. 57. 2010 Graphene
  58. 58. Rischio e Ricerca
  59. 59. Rischio e Ricerca
  60. 60. Research is risky ! The real problem is to understand whom to reward today, among the large magma of good quality researchers, and how to pick those who would become excellent tomorrow !
  61. 61. “La storia della scienza è stata e deve essere una storia di programmi di ricerca in competizione … prima comincia la competizione e meglio è per il progresso”. (1970, Imre Lakatos) Competizione delle idee vs competizione ricercatori
  62. 62. We conclude that scientific impact (as reflected by publications) is only weakly limited by funding. We suggest that funding strategies that target diversity, rather than “excellence”, are likely to prove to be more productive. Research is risk
  63. 63. Research is risk
  64. 64. Science can be effective in the national welfare only as a member of a team, whether the conditions be peace or war. But without scientific progress no amount of achievement in other direction can insure our health, prosperity and security as a nation in the modern world Science The Endless Frontier A Report to the President by Vannevar Bush, Director of the Office of Scientific Research and Development, July 1945
  65. 65. WilliamH.Press (Science15November2013)
  66. 66. • Its aim was to make the EU “the most competitive and dynamic knowledge-based economy in the world capable of sustainable economic growth with more and better jobs and greater social cohesion”, by 2010 • Raise overall R&D investment to 3% of GDP by 2010. The Lisbon Strategy (March 2000)
  67. 67. 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Berd Gerd 2001 Gross Domestic Expenditure on R&D Business Enterprise Expenditure on R&D SpesainRicerca&Sviluppo EasternCountries Europa a quattro velocità
  68. 68. Impatto della crisi economica sulla spesa per istruzione: Italia, Ungheria, Irlanda e Estonia hanno diminuito la spesa
  69. 69. 0 0.5 1 1.5 2 2.5 3 3.5 4 Berd Gerd 2012 SpesainRicerca&Sviluppo EasternCountries Europa a quattro velocità
  70. 70. The invisible hand is visible and working hard !WilliamH.Press (Science15November2013)
  71. 71. Innovation requires risk The key role of the visible hand is to build the infrastructures that are necessary but not sufficient for the economic development
  72. 72. Risk in research and innovation requires • Diversification • Adaptability • Cooperation • Long times How can I reach the long term if I do not survive in the short one?
  73. 73. The sacrifice of new generations on the altar of austerity is the loss of a common heritage Long term problem ! The modern Ifigenia
  74. 74. “If you think education is expensive, try ignorance”