This document summarizes a presentation on research assessment in the UK. It outlines the Research Assessment Exercise (RAE) process and its impact on researcher behavior. It then discusses the transition to the new Research Excellence Framework (REF), which will place greater emphasis on citations, impact, and environment. The presentation notes that the RAE and REF influence what researchers study and how they disseminate their work, and that behaviors will continue adjusting in response to assessment changes.

1. Research AssessmentDavid ClayPlaying the game of Bibliometrics, DIT20th November 2009

2. OutlineResearch Funding in EnglandResearch Assessment Exercise (RAE)Research Evaluation FrameworkImpact of the RAE/REF on research outputsOpen accessConclusions

3. RIN (2008) Making sense of research funding in UK higher education. London: RIN. http://www.rin.ac.uk/resources/factsheets/making-sense-research-funding-uk

4. Research Assessment Exercise 2008 The purpose of the RAE is:“to provide authoritative and comprehensive quality ratings for research in all disciplines carried out in universities and colleges across the UK, to inform UK higher education funding bodies’ allocation of grant for research.”

5. RAE 2008HEIs made submissions to the UOAs in which they specialisedAll submissions were reviewed by expert panels, who made judgements against a pre-defined quality assessment.Each panel issued a statement detailing its criteria and working methods. Each HEI was provided with a quality profile for each of the UOAs to which they submitted

6. Elements of assessmentResearch outputs: 4 per person (70%)Researchenvironment (20%) Esteem & impact indicators (10%)Quality profilee.g.Research incomePhD studentsStaff developmentWeighted and aggregated across each submissionSource: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt

7. Selection of staffHEFCE (2009) Selection of staff for inclusion in RAE2008. London: HEFCE. http://www.hefce.ac.uk/pubs/hefce/2009/09_34/09_34.pdf

8. RAE 2008 quality criteria for outputs* To knowledge, theory, policy or practiceSource: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt

9. RAE 2008 Quality Profilehttp://www.rae.ac.uk/Submissions/submission.aspx?id=23&type=uoa&subid=1587

10. From RAE to Research Evaluation Framework (REF)In 2006 DfES proposed moving to a metrics-based system of allocating research fundingHEFCE developed proposals for a system based on:a bibliometric indicator of quality, research income and research student data for STEM subjectslight-touch peer review informed by metrics for the other subjects 2007-08: consultation on bibliometrics2009: Announced a more unified framework, combining metrics and expert review across all subjects2009/10: consultation on Impact

11. Outputs (60%)REF Framework____________________________________________________________________________________________________________Impact (25%)Environment (15%)Quality of all types of researchEconomic, social, cultural and quality of life benefitsQuality and sustainability of the research environmentExpert review of selected outputs (informed by citation information in appropriate UoAs)Narrative statement and case studies, supported by indicatorsNarrative supported by indicatorsLanglands, A (2009) Research Funding and Assessment: The Future. HEPI Conference. 14th October. http://www.hepi.ac.uk/files/Alan%20Langlands%20Research%20Funding%20and%20Assessment.ppt

12. Impact Rosenberg, G (2009) The REF – taking account of the impact of research. AURIL Conference, 10 October. http://www.auril.org.uk/modules/download_gallery/dlc.php?file=66%20-

13. Outputs HEIs will select staff and outputs (3) to submit for assessmentThe criteria for assessing outputs will be ‘originality, rigour and significance’Sub-panels will assess outputs through a process of expert reviewSome sub-panels will also make use of citation information to inform their assessment of outputs

14. Use of citation informationUOAs for which ‘robust’ data is available will make use of citation informationPanels will be provided with citation information about submitted outputs, and with appropriate benchmarksCitation information will be used to inform and supplement the review of outputsDecisions will not be made solely on the basis of citation informationAll outputs will be treated equally, whether or not there is citation information available for them.

15. REF OutcomesFor each UoA submitted the outcomes will be presented as:a sub-profile for each of outputs, impact and environment to show the proportion of submitted work meeting each level in a five-point scalean overall excellence profile, which combines the three sub-profiles.

16. The influence of research assessment“... for the clear majority of researchers, the RAE is the dominant concern when it comes to assessing their research.”RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads

17. RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads

18. RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.38 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloadsCitation behaviour Major influence on citation are the perceived authority of the publication and authorAccessibility is a major influence on what is citedCitation practice related to length of experienceCitation practices are clearly infuenced by disciplinary norms and the policies of individual journalsEmphasis on bibliometrics is likely to change citation practice.

19. Beating the REFIncrease author self citationPublish with US authors because they overcite their own papersPublish controversial papersMake citation arrangementsCite collaborators work more often

20. RAE Rules and Institutional PoliciesUnderstanding of institutional policies has an important part in determining behaviourInstitutions are perceived to have pressured researchers to publish in high impact journalsInstitutions adopt strategies to maximise their performance in research assessments; this may constrain researchers autonomy

21. Open AccessThe use of bibliometrics in research assessment will lead to more researchers making there work available on open accessThe major research funders in the UK have open access mandatesThere are now 11 institutional mandates in the UK and at least 5 more are being discussedInstitutions are having to building better publication databases in preparation for REFSome using this as an opportunity to populate their repositoryOthers are embedding the repository into the RMS

22. Conclusions“Researchers are not fools. Whenever targets have changed in the past, academic’s behaviour has adjusted to the target. So I expect publishing behaviour to be changed to align with the requirements of the REF.”

23. This work is licensed under a Creative Commons Licence.Thank you for listeningAny Questions?