The Research Excellence Framework (REF) is a process used to assess research in UK higher education institutions. It replaces the Research Assessment Exercise and will inform £2 billion in annual research funding. The REF assesses three elements: outputs, impact, and environment. Institutions submit evidence of staff, outputs, impact case studies, and environment data to expert panels who will assess submissions in 2014 based on published criteria. Guidance documents provide details on the submission and assessment process.

1. The Research Excellence
Framework
Anna Dickinson
REF Team
DARTS
June 2012

2. Presentation outline
• Overview
• Staff
• Outputs
• Impact
• Environment
• REF panels
• Submissions

3. Overview:
Purpose of the REF
• The REF is a process of expert review
• It replaces the RAE as the UK-wide framework for
assessing research in all disciplines
• Its purpose is:
- To inform research funding allocations by the four UK
HE funding bodies (approximately £2 billion per year)
- Provide accountability for public funding of research
and demonstrate its benefits
- To provide benchmarks and reputational yardsticks

4. Overview:
The assessment framework
Overall quality
Outputs Impact Environment
Maximum of 4 outputs Impact template and Environment data and
per researcher case studies template
65% 20% 15%

5. Overview:
The REF process
Criteria phase Submissions phase Assessment phase
2011 2012-13 2014
• Develop and publish • HEIs submit Codes of • Panels assess
Guidance on submissions practice submissions
(Jul)
• Launch the REF • Publish outcomes
• Develop, consult on and submissions system Dec 2014
publish Panel criteria
(Jan 2012) • Submission deadline
29 Nov 2013

6. Overview:
Guidance and criteria
Comprehensive information and guidance is set out in:
• Assessment framework and guidance on
submissions (July 2011):
- Sets out the information required in submissions and
the definitions used
• Panel criteria and working methods (Jan 2012):
- Sets out how panels will assess submissions
The above documents set out the official guidelines for the REF.
These slides provide a summary of key points but do not provide or
replace the official guidelines.

7. Overview:
Submissions
• Each HEI may submit in any or all of the 36 units of
assessment (UOAs)
• Each submission in a UOA provides evidence about the
activity and achievements of a ‘submitted unit’ including:
- Staff details (REF1a/b/c)
- Research outputs (REF2)
- Impact template and case studies (REF3a/b)
- Environment data (REF4a/b/c)
- Environment template (REF5)
• A submitted unit may, but need not, comprise staff who
work within a single ‘department’ or organisational unit

8. Overview:
Assessment
• Submissions will be assessed by 36 sub-panels
working under the guidance of 4 main panels
• Panels will carry out the assessment according to the
published criteria and working methods
Sub-panel responsibilities Main panel responsibilities
• Contributing to the panel • Developing the panel criteria
criteria and working methods and working methods
• Assessing submissions and • Ensuring adherence to the
recommending the outcomes criteria/procedures and
consistent application of the
overall assessment
standards
• Signing off the outcomes

9. Staff

10. Staff:
Staff selection
• HEIs are responsible for selecting eligible staff whose
outputs are to be included in their REF submissions
• Each HEI is required to develop, document and apply a
code of practice on the fair selection of staff:
- Demonstrating principles of
transparency, consistency, accountability and inclusivity
- Guidance on developing the codes has been based on
good practice found in the 2008 RAE
- The code must be submitted to the REF team, will be
examined for adherence to the published guidance, and
will be published at the end of the exercise

11. Staff:
Individual staff circumstances
• Up to four outputs must be listed against each
individual
• The number of outputs can be reduced without penalty
where an individual’s circumstances have constrained
their ability to work productively or produce four outputs
in the REF period
• We have sought to make these arrangements as clear
and consistent as possible, with due regard to
confidentiality

12. Staff:
Clearly defined circumstances
• These are circumstances involving a
- Early Career clear ‘absence’ from work
researchers
• ‘Tariffs’ define the number of outputs that
- Part-time working, may be reduced without penalty
career breaks and • These will be applied consistently by all
secondments REF sub-panels
outside of HE
• Circumstances can be combined up to a
maximum reduction of three outputs
- Periods of maternity,
adoption and • Where an individual has a combination of
additional paternity clearly defined and complex
leave circumstances, these should be
submitted collectively as ‘complex’

13. Staff:
Complex circumstances
- Disability
- Ill health or injury
- Mental health • For these circumstances a judgement is
conditions needed about the appropriate reduction
• The EDAP will consider all these cases
- Additional
on a consistent and confidential
constraints related basis, and recommend the appropriate
to bringing a child reductions to the Main Panel Chairs
into the family
• Sub-panels will be informed of the
- Other caring decisions and will not have access to
responsibilities further details
- Gender • ECU has published worked examples
reassignment (www.ecu.ac.uk)
- Other circumstances
related to legislation

14. Outputs

15. Outputs:
Research outputs
• Panels will assess the quality of research outputs
through a process of expert review
• All forms of output that embody research will be
assessed on an equal footing
• Panels will assess the quality of outputs, not the
contribution of individual researchers
• A co-authored output may be listed against one or
more individuals that made a substantial research
contribution to it (no more than twice within the same
submission)
• Institutions may request ‘double-weighting’ for outputs
of extended scale and scope

16. Outputs:
Additional information
• Several sub-panels will make use of citation data as a
minor component to inform peer-review
• HEIs will be provided access to the Scopus citation data
(in the relevant UOAs) through the REF submission
system
• Several sub-panels invite additional information from the
HEI to inform judgements
• Panels will not use journal impact factors, rankings or
lists or the perceived standing of the publisher

17. Outputs:
Assessment criteria
The criteria for assessing the quality of outputs are
originality, significance and rigour*
Four star Quality that is world-leading in terms of originality, significance
and rigour
Quality that is internationally excellent in terms of originality,
Three star significance and rigour but which falls short of the highest
standards of excellence
Two star Quality that is recognised internationally in terms of originality,
significance and rigour
One star Quality that is recognised nationally in terms of originality,
significance and rigour
Quality that falls below the standard of nationally recognised
Unclassified work. Or work which does not meet the published definition of
research for the purposes of this assessment
* Each main panel provides descriptive account of the criteria

18. Impact

19. Impact:
Definition of impact
• Impact is defined broadly for the REF:
an effect on, change or benefit to the
economy, society, culture, public policy or
services, health, the environment or quality of
life, beyond academia
• Panels recognise that impacts can be manifest in a
wide variety of ways, may take many forms and occur
in a wide range of spheres, in any geographic location
• Panels provide examples of impact relevant to their
disciplines, intended to stimulate ideas - not as
exhaustive or prescriptive lists

20. Impact:
Some examples of impact
Enhanced professional
Improved health or standards, ethics, guidelines Improved risk
welfare outcomes or training management
Public debate has
been shaped or
Improved More effective Improved business informed by research
quality, accessibility or management or performance
efficiency of a public service workplace practices
A social enterprise
Research has enabled initiative has been
Changes to the Production costs have
stakeholders to challenge created
design or delivery of reduced
conventional wisdom
the school curriculum Improved forensic
Enhanced
Improved access to methods or expert
Policy debate or decisions preservation, conservation or
justice, employment systems
have been influenced or presentation of cultural
or education
shaped by research heritage
Improved management or
Research has informed conservation of natural
Organisations have Jobs have been public
created or protected resources
adapted to changing understanding, values, attitud
cultural values es or behaviours
Enhanced corporate The policies or activities of Changes to
Levels of waste have
social responsibility NGOs or charities have been legislation or
reduced
policies informed by research regulations
New forms of artistic Changes in Enhanced technical
A new product has
expression or changes to professional practice standards or
been commercialised
creative practice protocols

21. Impact:
Submission requirements
• Sets out the submitted unit’s general
Impact template approach to supporting impact from
(REF3a) its research:
• Approach to supporting impact during
20% of the the period 2008 to 2013
impact
sub-profile • Forward strategy and plans
• Specific examples of impacts already
achieved, that were underpinned by
Case studies
the submitted unit’s research:
(REF3b)
• 1 case study per 10 FTE staff
80% of the submitted (plus 1 extra)
impact • Impacts during 2008 to 2013;
sub-profile underpinned by research since 1993

22. Impact:
Case studies
• Each case study should:
- Clearly describe the underpinning research, who undertook
it and when
- Provide references to the research and evidence of quality
- Explain how the research led/contributed to the impact
- Clearly identify the beneficiaries and define the impact
- Provide evidence/indicators of the impact
- Provide independent sources of corroboration
• All the material required to make a judgement should be
included in the case study
• Submitted case studies need not be representative of
activity across the unit: pick the strongest examples

23. Impact:
Assessment criteria
The criteria for assessing impacts are reach and significance*
Four star Outstanding impacts in terms of their reach and significance
Three star Very considerable impacts in terms of their reach and
significance
Two star Considerable impacts in terms of their reach and significance
One star Recognised but modest impacts in terms of their reach and
significance
The impact is of little or no reach and significance; or the impact
Unclassified was not eligible; or the impact was not underpinned by excellent
research produced by the submitted unit
* Each main panel provides descriptive account of the criteria

24. Environment

25. Environment:
Environment template
• Each submission to include a completed template:
- Overview
- Research strategy
- People, including:
- staffing strategy and staff development
- research students
- Income, infrastructure and facilities
- Collaboration and contribution to the discipline or research
base
• The ‘panel criteria’ request specific types of evidence
under each heading, and indicate how much weight they
will attach to each component

26. Environment:
Environment data
• All submissions to include data on:
- Research doctoral degrees awarded (REF4a)
- Research income (REF4b)
- Research income in-kind (REF4c)
• Definitions are aligned with HESA returns; the data
relate to the ‘whole unit’ - not just submitted staff
• Some sub-panels request specific additional data, to
be included within the environment template (REF5)
• Data will be considered by panels alongside the
narrative information provided in the relevant section
of the environment template

27. Environment:
Assessment criteria
The criteria for assessing the environment are
vitality and sustainability*
Four star An environment that is conducive to producing research of
world-leading quality, in terms of its vitality and sustainability
An environment that is conducive to producing research of
Three star internationally excellent quality, in terms of its vitality and
sustainability
An environment that is conducive to producing research of
Two star internationally recognised quality, in terms of its vitality and
sustainability
An environment that is conducive to producing research of
One star nationally recognised quality, in terms of its vitality and
sustainability
Unclassified An environment that is not conducive to producing research of
nationally recognised quality
* Each main panel provides a descriptive account of the criteria

28. Submissions

29. Overview:
Multiple and joint submissions
• Institutions will normally make one submission in each
UOA they elect to submit in
• Joint submissions are encouraged where this is an
appropriate way of describing collaborative research
• An institution may make multiple submissions in a UOA
only by exception and with prior permission

30. Pre-submission
• Codes of practice
Submission by Response from funding body by
27 April 2012 6 July 2012
31 July 2012 12 October 2012
• Requests for multiple submissions or case studies requiring
security clearance
Request by Response from REF team by
27 April 2012 8 June 2012
28 September 2012 9 November 2012
7 December 2012 18 January 2013
• Survey of submission intentions
Invitation and guidance Online survey open
July 2012 Early Oct – early Dec 2012

31. Overview:
The submission system
• All submissions must be made through the REF
submission system:
 Pilot available to all HEIs: Sep 2012
 Open for submissions: Jan–Nov 2013
• Each HEI to set up system users and user permissions
• All data may be entered onto the system and/or bulk
imported
• HEI contacts have seen a demonstration of the system
• User guidance and support will be provided

32. Access to REF4 data
• We will provide HESA data on research income and
doctoral degrees awarded in stages:
- May 2012: Data for 2008-09, 2009-10 and 2010-11
- Apr 2013: Data for 2011-12
- Doctoral degrees data for 2012-13 will be available from
HESA when the student record is submitted
• Data on research income-in-kind will be provided by
the Research Councils and health research funders
• Institutions will need to allocate these data to the
appropriate UOAs; or use their own sources
• The submission system will validate submitted data
against the data we provided, at HEI level
• stem

33. Further information
www.ref.ac.uk
(includes all relevant documents)
Enquiries from staff at HEIs should be directed to
their nominated institutional contact
(see www.ref.ac.uk for a list)
Other enquiries to info@ref.ac.uk