The document discusses incorporating impact assessments into the Research Excellence Framework (REF) evaluation process. It summarizes the results of an impact pilot project that tested assessing the societal and economic impacts of research through expert review of case studies. The pilot found it is possible to assess research impacts and that a generic approach can work across disciplines, with refinements needed. Sub-panels provided feedback that bibliometrics could help inform peer review but not be used formulaically due to coverage differences across fields. Next steps include procuring citation data and a July decision on bibliometrics in the REF.

2. Incorporating impact in the REF______________________________________________________________________________________________________________________________
 The aim is to identify and reward the contribution that high quality
research has made to the economy and society:
– Making these explicit to the government and wider society
– Creating a level playing field
– Encouraging institutions to achieve the full potential
contribution of their research in future

3. Impact: initial consultations______________________________________________________________________________________________________________________________
 Widespread acceptance of the principle of incorporating impact in
the REF, and agreement that the impact assessment should:
- Be based on expert review
- Review historical impacts, not predict future impact
- Focus on the impact of submitted units’ research, not individual
researchers
- Be underpinned by high quality research
- Take a wide view of impact, inclusive of all disciplines

4. The impact pilot exercise______________________________________________________________________________________________________________________________
 Tested and developed a case study approach to assessing the
impact of research
 Five units of assessment (UOAs)
 29 UK higher education institutions each submitting to 2 UOAs
 Each submission included:
- An ‘impact statement’ for the submitted unit as a whole
- Case studies illustrating examples of impacts achieved (a total of one
case study per 10 research staff)
 Impacts that occurred during 2005-09, underpinned by
research since 1993

5. The pilot panels______________________________________________________________________________________________________________________________
 Membership drawn from academia and research users from
the private, public and third sectors
 The panels tested the methodology by:
- Assessing the case studies in terms of ‘reach and significance’ of
the impacts
- Considering the wider ‘impact statements’
- Producing impact profiles
- Reflecting on the process, identifying issues and making
recommendations on how to improve the process

6. Key findings______________________________________________________________________________________________________________________________
 The process makes explicit the benefits that research in each
discipline brings to society

7. Benefits of research______________________________________________________________________________________________________________________________

8. Key findings______________________________________________________________________________________________________________________________
 The process makes explicit the benefits that research in each
discipline brings to society
 It is possible to assess the impact of research, through expert
review of case studies
 A number of refinements are needed for full implementation
 A generic approach is workable, with scope for REF panels to tailor
the criteria as appropriate to their disciplines
 The weighting should be significant to be taken seriously by all
stakeholders, and needs careful consideration

9. Decisions on impact______________________________________________________________________________________________________________________________
 Published decisions available online at: www.ref.ac.uk
 Weightings of the three elements: outputs, impact and
environment
 Broad framework for the assessment of impact

10. The REF Framework______________________________________________________________________________________________________________________________

11. Broad framework______________________________________________________________________________________________________________________________
 Based on expert review of case studies (1, + 1 in 10)
 May include any social, economic or cultural impact or benefit
beyond academia, arising from excellent research, that has taken
place during the assessment period
 Also include information about how the unit has supported and
enabled impact during the assessment period
 Assessment of reach and significance
 Involvement of research-users
 Further guidance: July 2011 and January 2012

12. BIBLIOMETRICS IN THE REF
Royal Society of Medicine
University Health & Medical Librarians’ Group
7 March

13.  Background to bibliometrics
 Bibliometrics pilot
 Preliminary feedback from sub-panels
 Next steps
Contents______________________________________________________________________________________________________________________________

14. Background to bibliometrics______________________________________________________________________________________________________________________________
 Count the number of citations gained by each item under
assessment
 Time dependence
 Subject dependence
 Benchmarking of raw counts

15.  22 UK HEIs
 All UOAs with reasonable coverage in the bibliometric
databases
 Trialled several approaches
 Address based model
 Author based models
 Just RAE submitted people
 Selected papers
Bibliometrics pilot______________________________________________________________________________________________________________________________

16.  Fully inclusive models difficult
 Institutional data verification
 Defining subject areas
 Cannot use formulaically - though significant potential to
support assessment process
 Equalities implications
Conclusions from the pilot______________________________________________________________________________________________________________________________

17.  Interest from main panels A and B
 Little interest from C and D
 Range of options
 Simple counts and guidance on use
 Fully normalised / benchmarked
REF Sub-panel feedback______________________________________________________________________________________________________________________________

18.  Not for use formulaically
 As a means of informing peer review
 Cross-check/tie break
 Reflects academic significance only
 Supporting evidence only
 Concerns about:
 Differential coverage
 Discouraging applied/translational work
 Timeliness of data
 Clarity of guidance on submissions
 Normalisation of scores
 Time lag on indicators
REF Sub-panel feedback______________________________________________________________________________________________________________________________

19.  Procurement of citation data
 Discussions with several suppliers
 REF steering group will take a decision informed by
 Procurement discussions
 Panel feedback
 Decision published in July
Next steps______________________________________________________________________________________________________________________________

20. Thank you______________________________________________________________________________________________________________________________
Kim Hackett
Policy Adviser
HEFCE
0117 931 7267
k.hackett@hefce.ac.uk
David Mawdsley
Analyst
HEFCE
0117 931 7365
d.mawdsley@hefce.ac.uk