Impact, bibliometrics and the REF

2. Incorporating impact in the REF______________________________________________________________________________________________________________________________
 The aim is to identify and reward the contribution that high quality
research has made to the economy and society:
– Making these explicit to the government and wider society
– Creating a level playing field
– Encouraging institutions to achieve the full potential
contribution of their research in future

3. Impact: initial consultations______________________________________________________________________________________________________________________________
 Widespread acceptance of the principle of incorporating impact in
the REF, and agreement that the impact assessment should:
- Be based on expert review
- Review historical impacts, not predict future impact
- Focus on the impact of submitted units’ research, not individual
researchers
- Be underpinned by high quality research
- Take a wide view of impact, inclusive of all disciplines

4. The impact pilot exercise______________________________________________________________________________________________________________________________
 Tested and developed a case study approach to assessing the
impact of research
 Five units of assessment (UOAs)
 29 UK higher education institutions each submitting to 2 UOAs
 Each submission included:
- An ‘impact statement’ for the submitted unit as a whole
- Case studies illustrating examples of impacts achieved (a total of one
case study per 10 research staff)
 Impacts that occurred during 2005-09, underpinned by
research since 1993

5. The pilot panels______________________________________________________________________________________________________________________________
 Membership drawn from academia and research users from
the private, public and third sectors
 The panels tested the methodology by:
- Assessing the case studies in terms of ‘reach and significance’ of
the impacts
- Considering the wider ‘impact statements’
- Producing impact profiles
- Reflecting on the process, identifying issues and making
recommendations on how to improve the process

6. Key findings______________________________________________________________________________________________________________________________
 The process makes explicit the benefits that research in each
discipline brings to society

7. Benefits of research______________________________________________________________________________________________________________________________

8. Key findings______________________________________________________________________________________________________________________________
 The process makes explicit the benefits that research in each
discipline brings to society
 It is possible to assess the impact of research, through expert
review of case studies
 A number of refinements are needed for full implementation
 A generic approach is workable, with scope for REF panels to tailor
the criteria as appropriate to their disciplines
 The weighting should be significant to be taken seriously by all
stakeholders, and needs careful consideration

9. Decisions on impact______________________________________________________________________________________________________________________________
 Published decisions available online at: www.ref.ac.uk
 Weightings of the three elements: outputs, impact and
environment
 Broad framework for the assessment of impact

10. The REF Framework______________________________________________________________________________________________________________________________

11. Broad framework______________________________________________________________________________________________________________________________
 Based on expert review of case studies (1, + 1 in 10)
 May include any social, economic or cultural impact or benefit
beyond academia, arising from excellent research, that has taken
place during the assessment period
 Also include information about how the unit has supported and
enabled impact during the assessment period
 Assessment of reach and significance
 Involvement of research-users
 Further guidance: July 2011 and January 2012

12. BIBLIOMETRICS IN THE REF
Royal Society of Medicine
University Health & Medical Librarians’ Group
7 March

13.  Background to bibliometrics
 Bibliometrics pilot
 Preliminary feedback from sub-panels
 Next steps
Contents______________________________________________________________________________________________________________________________

14. Background to bibliometrics______________________________________________________________________________________________________________________________
 Count the number of citations gained by each item under
assessment
 Time dependence
 Subject dependence
 Benchmarking of raw counts

15.  22 UK HEIs
 All UOAs with reasonable coverage in the bibliometric
databases
 Trialled several approaches
 Address based model
 Author based models
 Just RAE submitted people
 Selected papers
Bibliometrics pilot______________________________________________________________________________________________________________________________

16.  Fully inclusive models difficult
 Institutional data verification
 Defining subject areas
 Cannot use formulaically - though significant potential to
support assessment process
 Equalities implications
Conclusions from the pilot______________________________________________________________________________________________________________________________

17.  Interest from main panels A and B
 Little interest from C and D
 Range of options
 Simple counts and guidance on use
 Fully normalised / benchmarked
REF Sub-panel feedback______________________________________________________________________________________________________________________________

18.  Not for use formulaically
 As a means of informing peer review
 Cross-check/tie break
 Reflects academic significance only
 Supporting evidence only
 Concerns about:
 Differential coverage
 Discouraging applied/translational work
 Timeliness of data
 Clarity of guidance on submissions
 Normalisation of scores
 Time lag on indicators
REF Sub-panel feedback______________________________________________________________________________________________________________________________

19.  Procurement of citation data
 Discussions with several suppliers
 REF steering group will take a decision informed by
 Procurement discussions
 Panel feedback
 Decision published in July
Next steps______________________________________________________________________________________________________________________________

20. Thank you______________________________________________________________________________________________________________________________
Kim Hackett
Policy Adviser
HEFCE
0117 931 7267
k.hackett@hefce.ac.uk
David Mawdsley
Analyst
HEFCE
0117 931 7365
d.mawdsley@hefce.ac.uk