Summary of key findings from research by Bella Reichard and colleagues analysing high versus low scoring case studies from REF2014. View full slide deck here: https://www.slideshare.net/MarkReed11/language-in-ref2014-impact-case-studies-what-might-it-mean-for-ref2021. Read the paper here: https://www.nature.com/articles/s41599-020-0394-7

2. 1. Case study fields remain, but with metadata and 5 pages
2. Underpinning research must be 2* quality and published >2000
3. Impacts must occur between Aug 2013 and July 2020
4. Minimum 2 case studies up to 20 FTE, then 1 per 15 FTE
5. Impacts from a body of research (e.g. collaborative project) must
demonstrate substantive contribution from institution
6. Public engagement is a pathway, not an impact
7. Underpinning research 2* as a body with quality justification
8. All evidence submitted with case studies and independently
verifiable (testimonials should be based on evidence, not opinion)
9. Continuation case studies have 1) no significant new underpinning
research, AND 2) similar impacts and beneficiaries to those in 2014
10. A researcher’s outputs & impact can be submitted to different UoAs
11. Panel A will treat qualitative evidence and continuation case studies
without prejudice but consultation revealed disciplinary biases
Impact in REF2021: a summary

3. What made a 4*case study in REF2014?
Based on PhD research by Bella Reichard @BellaReichard based
on quantitative analysis of 217 and qualitative analysis of 180 of
the highest and lowest scoring cases, spread across Panels A, B,
C and D

5. Quantitative linguistic analysis
1. Highly-rated case studies provided specific,
high-magnitude and well-evidenced
articulations of significance and reach
• 84% of high-scoring cases articulated significant and far-reaching
benefits, compared to 32% of low-scoring cases, which typically focused
on pathway
Phrases more common in high-
scoring:
• Significance and reach (specific and
high): in England and, in the US, the
UK’s, millions of, long-term, the
government’s, the department of,
the House of Commons, for the first
time, prime minister, select
committee
Phrases more common in low-scoring:
• Significance and reach (non-specific or low): a number
of, a range, nationally and internationally, in local, of
local, the north, city council, policy and practice, an
impact on, impact on the, the impact
• Pathways to impact: has been disseminated, disseminated
through, dissemination of, been disseminated, and
workshops, the event, the book
• Beneficiaries (not benefits): and community, practitioners
and, group of, members of the
Qualitative analysis

6. • 97% of high-scoring cases clearly linked the
underpinning research to claimed impacts,
compared to 50% of low-scoring case studies
• 42% high-scoring policy cases described policy and
implementation, compared to 17% in low
Phrases more common in high-scoring
• Attribution between research and
impact: cited in the, (was) used to
inform, to improve the, led to the,
resulting in, showing that, was
subsequently, produced by, reported in,
evidence for, cited in, led by
2. Highly-rated case studies established
links between research (cause) and
impact (effect) convincingly
Phrases more common in low-scoring
• Research outputs/process: the paper, peer-
reviewed, journal of, et al, research project, this
research has, by Dr, of Dr, research team
• Attribution between research and pathways:
work has, has informed, through the
0
10
20
30
40
50
60
70
80
Generally high
quality
corroborating
evidence
Some
questionable
quality evidence
Vague and/or not
clearly linked to
impacts
Numberofcasestudies
High-scoring Low-scoring
Quantitative linguistic analysis
Qualitative analysis
• Coh-metrix analysis shows higher-scoring cases had more explicit causal connections between
ideas and more logical connective words (and, or, but) than low-scoring cases

7. 3. Highly-rated case studies were
easy to understand and well
written
• Low-scoring cases more likely to have
academic phrasing: in relation to, in
terms of, the way(s) in which
• Flesch Reading Ease score, out of 100,
was 30.0 on average for 4* and 27.5 on
average for 1*/2* (all “college-graduate”
difficulty). Panels C & D high-scoring
case studies significantly easier to read
than low-scorers
• High scoring cases had more sub-
headings (especially pronounced when
comparing high to low cases in Panel D)
• High-scoring cases used more direct, plain language, had fewer
expressions of uncertainty or hedging statements, and were less likely to
contain unsubstantiated or vague use of adjectives to describe impacts
Qualitative analysis
Quantitative linguistic analysis

8. www.fasttrackimpact.com/resources

9. https://blogs.ncl.ac.uk/breichard2/