Your SlideShare is downloading. ×
Keynote 4 - Analysis of RAE 2008 and Prospects For REF - Jonathan Adams, Evidence, Thomson Reuters
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Keynote 4 - Analysis of RAE 2008 and Prospects For REF - Jonathan Adams, Evidence, Thomson Reuters

482
views

Published on

Keynote 4 - Analysis of RAE 2008 and Prospects for REF - Jonathan Adams, Evidence, Thomson Reuters

Keynote 4 - Analysis of RAE 2008 and Prospects for REF - Jonathan Adams, Evidence, Thomson Reuters

Published in: Education

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
482
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. EVOLVING RESEARCH EVALUATION
    RAE2008 OUTCOMES AND REF PROSPECTS
    JONATHAN ADAMS
    10 NOVEMBER 2009
  • 2. Evaluation
    • Impact
    • 3. Indicators vs profiles
    • 4. Decision support, not substitution
    • 5. What variables are relevant in ...
    • 6. Academic
    • 7. Economic
    • 8. Policy
    • 9. Social ... Impact
    • 10. Objective impact evaluation is constrained by a lack of reference data outside ‘academic’
  • Thomson Reuters and Education
    Web of Science
    250 + journal categories
    3 relate to ‘education’
    Education & educational research – 210 journals
    Education, disciplines – 37 journals
    Education, special – 36 journals
    Coverage is back-filled
    Regional spread is diversifying
  • 11. Sample coverage
    • Thomson Reuters coverage has broadened
    • 12. Alberta journal of educational research
    • 13. American journal of education
    • 14. Asia Pacific journal of education
    • 15. Australian journal of early childhood
    • 16. British educational research journal
    • 17. Chinese education and society
    • 18. DidacticaSlovenica - pedagoskaobzorja
    • 19. Durham and Newcastle research review
    • 20. Egitimvebilim - education and science
    • 21. European journal of Dental education
    All the way through to
    • Revista Brasileira de Ensino de Fisica
    • 22. Zeitschrift fur Soziologie der Erziehung und Sozialisation
  • UK coverage
    National Citation Report
    Evidence tracks all author addresses and reconciles to ‘real’ organisations
    Profiles for each UK university
  • 23. Steady but volatile improvement
  • 24. What is used to represent excellent research outcomes?
  • 25. RAE2008 grade spread
  • 26. Previously among studies by Evidence
    HEFCs/UUK – Maintaining Research Excellence
    Recognition of a peak supported by a research platform
    Evolution of 3 to 4 to 5 grade units
    HEFCE – Role of QR funding
    Strategic significance of the block grant
    Universities UK – Excellence and Diversity
    Regional network of research competence
    Importance of an evidence base for research policy
    OST/DIUS/BIS
    Diversity as a critical element in research policy
  • 27. RAE2008 – a novel outcome with dispersed rewards
    Gareth Roberts’ proposals on research profiling
    Problem of the ‘cliff edge’
    Recognising dynamic excellence in the research base
    Unexpected differences between panels
    The problem of interpreting methodology and standards
    The absence of a commonly understood standard?
    The dispersed map of reward-able research
    Decisions about funding allocations
    2* = 1 (x 3 to) 3* = 3 (x 2.33 to) 4* = 7
    Greater relative rewards for modest gains than for very expensive world-class excellence (cf 2001)
  • 28. How can we explore the spread of excellence?
    We have
    Panel outcomes from RAE2008 – but opinions differ, so ...
    Bibliometric data on a consistent basis from 1981
    We split the HE sector into crude groups
    All HEIs
    Russell Group
    1994 Group
    Universities with a shorter history of research investment
    And we created an elite ‘golden triangle’ band
    Oxford (C12th), Cambridge (1209), UCL (1836), Imperial (C19th) , LSE (1895)
  • 29. Excellence is linked to selectivity
  • 30. And excellence is concentrated
  • 31. The problem with simplistic indicators
    They don’t really express the complexity of research performance
    Average impact (e.g. ‘crown indicator’) can be very misleading
    Research Council studies reveal error of interpretation
    Skewed data, median much smaller than average
    Lots of papers are not cited
    The interesting bit is about how much is really, really cited lots
    So we we prefer Impact Profiles®
  • 32. UK background and ‘golden triangle’
    This is the small but critical excess of really highly cited research output
  • 33. Even smaller differences separate the UK and USA research profiles
  • 34. Excellence extends to other institutions
  • 35. But the differences become very fine
  • 36. And if we distil further, other complexity is revealed
  • 37. Older institutions have no monopoly on quality, but the peak is distinctive
    For the 1994 Group, there are relatively fewer low-cited papers and relatively more high-cited papers than for many in the Russell Group
  • 38. What does this imply?
    There is a very concentrated peak of exceptional excellence
    UK international competitiveness is associated with selectivity that increased concentration
    There is a regional network of exceptional quality, much in dynamic institutions created in the 1960s
    The balance that supports diversity & dynamism and the concentrated support of international excellence is very fine
    Model first, meddle second
  • 39. EVOLVING RESEARCH EVALUATION
    RAE2008 OUTCOMES AND REF PROSPECTS
    JONATHAN ADAMS
    10 NOVEMBER 2009