The document discusses key points about what makes research outputs eligible for submission to the UK's Research Excellence Framework (REF) assessment. It dispels common myths, such as the importance of journal impact factors or citation counts. The core REF assessment criteria of originality, significance, and rigor are emphasized. The peer review process is described as independent and ensuring anonymity. Tips are provided for writing REF-eligible papers, such as using appropriate guidelines and conveying messages clearly. The timing of research, publication, and submission processes is outlined in preparation for the 2022 REF.

1. What makes a REF paper
REF-able?
Roger Watson
Professor of Nursing
Editor-in-Chief, Journal of Advanced Nursing
Editor, Nursing Open

2. REF myths
• Outputs have to be related to one another
• Impact factors of journals are important
• Outputs have to be highly cited
• Order of authorship matters
• The scoring of my outputs will be published
• There’s no information available about REF processes
• If only we could see examples of environment & impact

3. REF 2020
• Assessment criteria for outputs unlikely to change
• Possibly only 4* research will be funded
• Peer review will be the basis of the assessment

4. 191,150 outputs!
10,358 to
sub-panel 3

5. Sub-panel working methods
• Sub-panels will review their expertise to ensure
appropriate coverage
• Work will be allocated to members/assessors with
appropriate expertise
• Each sub-panel will run calibration exercises for
outputs and impacts, guided by the main panels
• All outputs will be examined in sufficient detail to
contribute to the formation of the outputs sub-profiles
• Each case study will normally be assessed by at least
one academic and one user
• Graduated sub-profiles will be formed for each aspect
of submissions
REF panels:

6. Research outputs
• Outputs may include but are not limited to: printed or
electronic publications, materials, devices, images,
artefacts, products, buildings, confidential or technical
reports, patents, performances, exhibits or events
• All types of research and all forms of research output
shall be assessed on a fair and equal basis
• Panels will assess outputs through a process of expert
review. Where stated in the ‘panel criteria’, panels will
take account of additional information and/or citation
data to inform judgements based on expert review
• Panels will not use journal impact factors, rankings or
lists or the perceived standing of the publisher
Outputs:

7. Co-authorship
• A co-authored output may be listed against one or more
individuals that made a substantial research contribution
to it
• It may be listed against any or all such co-authors
returned in different submissions; and a maximum of
two such co-authors within the same submission
• In very specific situations the panels require information
to confirm that the author made a substantial research
contribution
• Once this is accepted, panels will assess the quality of
the output, not the individual author’s contribution
Outputs:

8. Assessment criteria
• The criteria for assessing the quality of outputs are
originality, significance and rigour
• Each panel provides further explanation of how they will
interpret these criteria
• Panels will assess the quality of outputs, not the
contribution of individual researchers to the submission
• They will examine all outputs in sufficient detail to
contribute to the formation of a robust outputs sub-profile
that represents all the outputs listed in a submission
Outputs:

9. Assessment criteria
The criteria for assessing the quality of outputs are
originality, significance and rigour*
Four star
Quality that is world-leading in terms of originality, significance and
rigour
Three star
Quality that is internationally excellent in terms of originality,
significance and rigour but which falls short of the highest standards of
excellence
Two star Quality that is recognised internationally in terms of originality,
significance and rigour
One star
Quality that is recognised nationally in terms of originality, significance
and rigour
Unclassified
Quality that falls below the standard of nationally recognised work. Or
work which does not meet the published definition of research for the
purposes of this assessment
* Each main panel provides descriptive account of the criteria
Outputs:

10. How it works
• Independent peer review of outputs
• Entered separately on REF system
• Anomalies discussed and scores agreed
• Scores are aggregated and individual scores are obliterated
from the system
• No paper trail is maintained
• Highly reliable system

11. Assessors
• ~ 300 outputs each as first assessor; plus
• ~ 300 as second assessor
• ~ 6 months to complete assessment
• How long can an assessor spend on an output?

12. Anomalous outputs
• Books
• Chapters
• Double counts
• Double claims
• Local reports
DON’T!

13. Criteria
• Originality
• Significance
• Rigour
• Not a ‘tick-box’ exercise
• Not a peer-review of the article
• It is a peer assessment of the research

14. Citations and journal rankings
• Citation data available
• Journal rankings never enter discussions

15. Writing REF-able papers 1
• Remember – it is the research that is being judged
• The output is only the vehicle
• However, the vehicle should be in ‘good shape’

16. Writing REF-able papers 2
• Use the words ‘originality’, ‘significance’ and ‘rigour’
• Use the words ‘international’, ‘global’
• Good journals will tend to help you write good papers
• Use journal facilities such as summaries to convey the main
message of the output
• Don’t use predatory journals

17. Writing REF-able papers 3
• Use Equator guidelines to present papers, eg:
• CONSORT for RCTs
• PRISMA for Systematic reviews

18. Processes leading up to REF
• Internal peer review prior to publication
• Internal REF review
• External REF review (former sub-panel members)
• Selection – will depend on university strategy

19. Timing for 2022
• Publication date (ie with doi) 31 December 2020
• Submission to publication can take 12 months
• Writing can take 12 months (possible including rejections)
• So:
– 2021 becomes 2020 (assessment period)
– 2020 becomes 2019 (final acceptance period)
– 2019 becomes 2018 (final submission period)
– 2018 becomes 2017 (final writing period)
Open Access – be systematic in using ‘green route’
repositories

20. REF myths
• Outputs have to be related to one another
• Impact factors of journals are important
• Outputs have to be highly cited
• Order of authorship matters
• The scoring of my outputs will be published
• There’s no information available about REF processes
• If only we could see examples of environment & impact

21. r.watson@hull.ac.uk
0000-0001-8040-7625
@rwatson1955