• Save
From RAE to REF
Upcoming SlideShare
Loading in...5

From RAE to REF






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial LicenseCC Attribution-NonCommercial License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Sub panels produced a quality profile the research environmentindicators of esteem & impact
  • .
  • Hefce says: "The main incentive for researchers will be to publish work that is recognised as high quality by peers and becomes highly cited by the international academic community. Short-term game-playing is far less likely to achieve this than responsible strategies to nurture the talent ... and for publishing work that enhances the group's international reputation.“In the technical work on citation analysis commissioned by HEFCE, the Leiden group addressed game playing, something they believe will have little effect on the outcomes.The RIN found that researchers say they will cite their collaborators’ work more often but there is little evidence that they will cite competitors’ work less often.

From RAE to REF From RAE to REF Presentation Transcript

  • Research Assessment
    David Clay
    Playing the game of Bibliometrics, DIT
    20th November 2009
  • Outline
    Research Funding in England
    Research Assessment Exercise (RAE)
    Research Evaluation Framework
    Impact of the RAE/REF on research outputs
    Open access
  • RIN (2008) Making sense of research funding in UK higher education. London: RIN. http://www.rin.ac.uk/resources/factsheets/making-sense-research-funding-uk
  • Research Assessment Exercise 2008
    The purpose of the RAE is:
    “to provide authoritative and comprehensive quality ratings for research in all disciplines carried out in universities and colleges across the UK, to inform UK higher education funding bodies’ allocation of grant for research.”
  • RAE 2008
    HEIs made submissions to the UOAs in which they specialised
    All submissions were reviewed by expert panels, who made judgements against a pre-defined quality assessment.
    Each panel issued a statement detailing its criteria and working methods.
    Each HEI was provided with a quality profile for each of the UOAs to which they submitted
  • Elements of assessment
    Research outputs: 4 per person (70%)
    environment (20%)
    Esteem & impact indicators (10%)
    Quality profile
    Research income
    PhD students
    Staff development
    Weighted and aggregated across each submission
    Source: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt
  • Selection of staff
    HEFCE (2009) Selection of staff for inclusion in RAE2008. London: HEFCE. http://www.hefce.ac.uk/pubs/hefce/2009/09_34/09_34.pdf
  • RAE 2008 quality criteria for outputs
    * To knowledge, theory, policy or practice
    Source: Thorpe, R (2009) Reflections on RAE 2008. http://www.ufhrd.co.uk/wordpress/wp-content/.../reflections-on-rae-2008.ppt
  • RAE 2008 Quality Profile
  • From RAE to Research Evaluation Framework (REF)
    In 2006 DfES proposed moving to a metrics-based system of allocating research funding
    HEFCE developed proposals for a system based on:
    a bibliometric indicator of quality, research income and research student data for STEM subjects
    light-touch peer review informed by metrics for the other subjects
    2007-08: consultation on bibliometrics
    2009: Announced a more unified framework, combining metrics and expert review across all subjects
    2009/10: consultation on Impact
  • Outputs (60%)
    REF Framework____________________________________________________________________________________________________________
    Impact (25%)
    Environment (15%)
    Quality of all types of research
    Economic, social, cultural and quality of life benefits
    Quality and sustainability of the research environment
    Expert review of selected outputs (informed by citation information in appropriate UoAs)
    Narrative statement and case studies, supported by indicators
    Narrative supported by indicators
    Langlands, A (2009) Research Funding and Assessment: The Future. HEPI Conference. 14th October. http://www.hepi.ac.uk/files/Alan%20Langlands%20Research%20Funding%20and%20Assessment.ppt
  • Impact
    Rosenberg, G (2009) The REF – taking account of the impact of research. AURIL Conference, 10 October. http://www.auril.org.uk/modules/download_gallery/dlc.php?file=66%20-
  • Outputs
    HEIs will select staff and outputs (3) to submit for assessment
    The criteria for assessing outputs will be ‘originality, rigour and significance’
    Sub-panels will assess outputs through a process of expert review
    Some sub-panels will also make use of citation information to inform their assessment of outputs
  • Use of citation information
    UOAs for which ‘robust’ data is available will make use of citation information
    Panels will be provided with citation information about submitted outputs, and with appropriate benchmarks
    Citation information will be used to inform and supplement the review of outputs
    Decisions will not be made solely on the basis of citation information
    All outputs will be treated equally, whether or not there is citation information available for them.
  • REF Outcomes
    For each UoA submitted the outcomes will be presented as:
    a sub-profile for each of outputs, impact and environment to show the proportion of submitted work meeting each level in a five-point scale
    an overall excellence profile, which combines the three sub-profiles.
  • The influence of research assessment
    “... for the clear majority of researchers, the RAE is the dominant concern when it comes to assessing their research.”
    RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads
  • RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.33 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads
  • RIN (2009) Communicating knowledge: How & why UK researchers publish & disseminate their findings. London: JISC. p.38 http://www.jisc.ac.uk/publications/documents/communicatingknowledgereport.aspx#downloads
    Citation behaviour
    Major influence on citation are the perceived authority of the publication and author
    Accessibility is a major influence on what is cited
    Citation practice related to length of experience
    Citation practices are clearly infuenced by disciplinary norms and the policies of individual journals
    Emphasis on bibliometrics is likely to change citation practice.
  • Beating the REF
    Increase author self citation
    Publish with US authors because they overcite their own papers
    Publish controversial papers
    Make citation arrangements
    Cite collaborators work more often
  • RAE Rules and Institutional Policies
    Understanding of institutional policies has an important part in determining behaviour
    Institutions are perceived to have pressured researchers to publish in high impact journals
    Institutions adopt strategies to maximise their performance in research assessments; this may constrain researchers autonomy
  • Open Access
    The use of bibliometrics in research assessment will lead to more researchers making there work available on open access
    The major research funders in the UK have open access mandates
    There are now 11 institutional mandates in the UK and at least 5 more are being discussed
    Institutions are having to building better publication databases in preparation for REF
    Some using this as an opportunity to populate their repository
    Others are embedding the repository into the RMS
  • Conclusions
    “Researchers are not fools. Whenever targets have changed in the past, academic’s behaviour has adjusted to the target. So I expect publishing behaviour to be changed to align with the requirements of the REF.”
  • This work is licensed under a Creative Commons Licence.
    Thank you for listening
    Any Questions?