Research Assessment David Clay Playing the game of Bibliometrics, DIT 20th November 2009
Outline Research Funding in England Research Assessment Exercise (RAE) Research Evaluation Framework Impact of the RAE/REF on research outputs Open access Conclusions
RIN (2008) Making sense of research funding in UK higher education. London: RIN. http://www.rin.ac.uk/resources/factsheets/making-sense-research-funding-uk
Research Assessment Exercise 2008 The purpose of the RAE is: “to provide authoritative and comprehensive quality ratings for research in all disciplines carried out in universities and colleges across the UK, to inform UK higher education funding bodies’ allocation of grant for research.”
RAE 2008 HEIs made submissions to the UOAs in which they specialised All submissions were reviewed by expert panels, who made judgements against a pre-defined quality assessment. Each panel issued a statement detailing its criteria and working methods. Each HEI was provided with a quality profile for each of the UOAs to which they submitted
Elements of assessment Research outputs: 4 per person (70%) Research environment (20%) Esteem & impact indicators (10%) Quality profile e.g. Research income PhD students Staff development Weighted and aggregated across each submission Source: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt
Selection of staff HEFCE (2009) Selection of staff for inclusion in RAE2008. London: HEFCE. http://www.hefce.ac.uk/pubs/hefce/2009/09_34/09_34.pdf
RAE 2008 quality criteria for outputs * To knowledge, theory, policy or practice Source: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt
RAE 2008 Quality Profile http://www.rae.ac.uk/Submissions/submission.aspx?id=23&type=uoa&subid=1587
From RAE to Research Evaluation Framework (REF) In 2006 DfES proposed moving to a metrics-based system of allocating research funding HEFCE developed proposals for a system based on: a bibliometric indicator of quality, research income and research student data for STEM subjects light-touch peer review informed by metrics for the other subjects 2007-08: consultation on bibliometrics 2009: Announced a more unified framework, combining metrics and expert review across all subjects 2009/10: consultation on Impact
Outputs (60%) REF Framework____________________________________________________________________________________________________________ Impact (25%) Environment (15%) Quality of all types of research Economic, social, cultural and quality of life benefits Quality and sustainability of the research environment Expert review of selected outputs (informed by citation information in appropriate UoAs) Narrative statement and case studies, supported by indicators Narrative supported by indicators Langlands, A (2009) Research Funding and Assessment: The Future. HEPI Conference. 14th October. http://www.hepi.ac.uk/files/Alan%20Langlands%20Research%20Funding%20and%20Assessment.ppt
Impact Rosenberg, G (2009) The REF – taking account of the impact of research. AURIL Conference, 10 October. http://www.auril.org.uk/modules/download_gallery/dlc.php?file=66%20-
Outputs HEIs will select staff and outputs (3) to submit for assessment The criteria for assessing outputs will be ‘originality, rigour and significance’ Sub-panels will assess outputs through a process of expert review Some sub-panels will also make use of citation information to inform their assessment of outputs
Use of citation information UOAs for which ‘robust’ data is available will make use of citation information Panels will be provided with citation information about submitted outputs, and with appropriate benchmarks Citation information will be used to inform and supplement the review of outputs Decisions will not be made solely on the basis of citation information All outputs will be treated equally, whether or not there is citation information available for them.
REF Outcomes For each UoA submitted the outcomes will be presented as: a sub-profile for each of outputs, impact and environment to show the proportion of submitted work meeting each level in a five-point scale an overall excellence profile, which combines the three sub-profiles.
The influence of research assessment “... for the clear majority of researchers, the RAE is the dominant concern when it comes to assessing their research.” RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads
RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads
RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.38 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads Citation behaviour Major influence on citation are the perceived authority of the publication and author Accessibility is a major influence on what is cited Citation practice related to length of experience Citation practices are clearly infuenced by disciplinary norms and the policies of individual journals Emphasis on bibliometrics is likely to change citation practice.
Beating the REF Increase author self citation Publish with US authors because they overcite their own papers Publish controversial papers Make citation arrangements Cite collaborators work more often
RAE Rules and Institutional Policies Understanding of institutional policies has an important part in determining behaviour Institutions are perceived to have pressured researchers to publish in high impact journals Institutions adopt strategies to maximise their performance in research assessments; this may constrain researchers autonomy
Open Access The use of bibliometrics in research assessment will lead to more researchers making there work available on open access The major research funders in the UK have open access mandates There are now 11 institutional mandates in the UK and at least 5 more are being discussed Institutions are having to building better publication databases in preparation for REF Some using this as an opportunity to populate their repository Others are embedding the repository into the RMS
Conclusions “Researchers are not fools. Whenever targets have changed in the past, academic’s behaviour has adjusted to the target. So I expect publishing behaviour to be changed to align with the requirements of the REF.”
This work is licensed under a Creative Commons Licence. Thank you for listening Any Questions?