EVOLVING RESEARCH EVALUATION<br />RAE2008 OUTCOMES  AND REF PROSPECTS<br />JONATHAN ADAMS<br />10 NOVEMBER 2009<br />
Evaluation<br /><ul><li>Impact
Indicators vs profiles
Decision support, not substitution
What variables are relevant in  ...
Academic
Economic
Policy
Social ... Impact
Objective impact evaluation is constrained by a lack of reference data outside ‘academic’</li></li></ul><li>Thomson Reuter...
Sample coverage<br /><ul><li>Thomson Reuters coverage has broadened
Alberta journal of educational research
American journal of education
Asia Pacific journal of education
Australian journal of early childhood
British educational research journal
Chinese education and society
DidacticaSlovenica - pedagoskaobzorja
Durham and Newcastle research review
Egitimvebilim - education and science
European journal of Dental education </li></ul>All the way through to<br /><ul><li>Revista Brasileira de Ensino de Fisica
Zeitschrift fur Soziologie der Erziehung und Sozialisation </li></li></ul><li>UK coverage<br />National Citation Report<br...
Upcoming SlideShare
Loading in …5
×

Keynote 4 - Analysis of RAE 2008 and Prospects For REF - Jonathan Adams, Evidence, Thomson Reuters

844 views

Published on

Keynote 4 - Analysis of RAE 2008 and Prospects for REF - Jonathan Adams, Evidence, Thomson Reuters

Published in: Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
844
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Keynote 4 - Analysis of RAE 2008 and Prospects For REF - Jonathan Adams, Evidence, Thomson Reuters

  1. 1. EVOLVING RESEARCH EVALUATION<br />RAE2008 OUTCOMES AND REF PROSPECTS<br />JONATHAN ADAMS<br />10 NOVEMBER 2009<br />
  2. 2. Evaluation<br /><ul><li>Impact
  3. 3. Indicators vs profiles
  4. 4. Decision support, not substitution
  5. 5. What variables are relevant in ...
  6. 6. Academic
  7. 7. Economic
  8. 8. Policy
  9. 9. Social ... Impact
  10. 10. Objective impact evaluation is constrained by a lack of reference data outside ‘academic’</li></li></ul><li>Thomson Reuters and Education<br />Web of Science<br />250 + journal categories<br />3 relate to ‘education’<br />Education & educational research – 210 journals<br />Education, disciplines – 37 journals<br />Education, special – 36 journals<br />Coverage is back-filled<br />Regional spread is diversifying<br />
  11. 11. Sample coverage<br /><ul><li>Thomson Reuters coverage has broadened
  12. 12. Alberta journal of educational research
  13. 13. American journal of education
  14. 14. Asia Pacific journal of education
  15. 15. Australian journal of early childhood
  16. 16. British educational research journal
  17. 17. Chinese education and society
  18. 18. DidacticaSlovenica - pedagoskaobzorja
  19. 19. Durham and Newcastle research review
  20. 20. Egitimvebilim - education and science
  21. 21. European journal of Dental education </li></ul>All the way through to<br /><ul><li>Revista Brasileira de Ensino de Fisica
  22. 22. Zeitschrift fur Soziologie der Erziehung und Sozialisation </li></li></ul><li>UK coverage<br />National Citation Report<br />Evidence tracks all author addresses and reconciles to ‘real’ organisations<br />Profiles for each UK university<br />
  23. 23. Steady but volatile improvement<br />
  24. 24. What is used to represent excellent research outcomes?<br />
  25. 25. RAE2008 grade spread<br />
  26. 26. Previously among studies by Evidence<br />HEFCs/UUK – Maintaining Research Excellence<br />Recognition of a peak supported by a research platform<br />Evolution of 3 to 4 to 5 grade units<br />HEFCE – Role of QR funding<br />Strategic significance of the block grant<br />Universities UK – Excellence and Diversity<br />Regional network of research competence<br />Importance of an evidence base for research policy<br />OST/DIUS/BIS<br />Diversity as a critical element in research policy<br />
  27. 27. RAE2008 – a novel outcome with dispersed rewards<br />Gareth Roberts’ proposals on research profiling<br />Problem of the ‘cliff edge’<br />Recognising dynamic excellence in the research base<br />Unexpected differences between panels<br />The problem of interpreting methodology and standards<br />The absence of a commonly understood standard?<br />The dispersed map of reward-able research<br />Decisions about funding allocations<br />2* = 1 (x 3 to) 3* = 3 (x 2.33 to) 4* = 7<br />Greater relative rewards for modest gains than for very expensive world-class excellence (cf 2001)<br />
  28. 28. How can we explore the spread of excellence?<br />We have <br />Panel outcomes from RAE2008 – but opinions differ, so ...<br />Bibliometric data on a consistent basis from 1981<br />We split the HE sector into crude groups<br />All HEIs<br />Russell Group<br />1994 Group<br />Universities with a shorter history of research investment<br />And we created an elite ‘golden triangle’ band<br />Oxford (C12th), Cambridge (1209), UCL (1836), Imperial (C19th) , LSE (1895)<br />
  29. 29. Excellence is linked to selectivity<br />
  30. 30. And excellence is concentrated<br />
  31. 31. The problem with simplistic indicators<br />They don’t really express the complexity of research performance<br />Average impact (e.g. ‘crown indicator’) can be very misleading<br />Research Council studies reveal error of interpretation<br />Skewed data, median much smaller than average<br />Lots of papers are not cited<br />The interesting bit is about how much is really, really cited lots<br />So we we prefer Impact Profiles®<br />
  32. 32. UK background and ‘golden triangle’<br />This is the small but critical excess of really highly cited research output<br />
  33. 33. Even smaller differences separate the UK and USA research profiles<br />
  34. 34. Excellence extends to other institutions<br />
  35. 35. But the differences become very fine<br />
  36. 36. And if we distil further, other complexity is revealed<br />
  37. 37. Older institutions have no monopoly on quality, but the peak is distinctive<br />For the 1994 Group, there are relatively fewer low-cited papers and relatively more high-cited papers than for many in the Russell Group<br />
  38. 38. What does this imply?<br />There is a very concentrated peak of exceptional excellence<br />UK international competitiveness is associated with selectivity that increased concentration<br />There is a regional network of exceptional quality, much in dynamic institutions created in the 1960s<br />The balance that supports diversity & dynamism and the concentrated support of international excellence is very fine<br />Model first, meddle second<br />
  39. 39. EVOLVING RESEARCH EVALUATION<br />RAE2008 OUTCOMES AND REF PROSPECTS<br />JONATHAN ADAMS<br />10 NOVEMBER 2009<br />

×