The document discusses incorporating impact assessments into the Research Excellence Framework (REF) evaluation process. It summarizes the results of an impact pilot project that tested assessing the societal and economic impacts of research through expert review of case studies. The pilot found it is possible to assess research impacts and that a generic approach can work across disciplines, with refinements needed. Sub-panels provided feedback that bibliometrics could help inform peer review but not be used formulaically due to coverage differences across fields. Next steps include procuring citation data and a July decision on bibliometrics in the REF.
Incorporating impact in the REF: Key findings and decisions
1.
2. Incorporating impact in the REF______________________________________________________________________________________________________________________________
The aim is to identify and reward the contribution that high quality
research has made to the economy and society:
– Making these explicit to the government and wider society
– Creating a level playing field
– Encouraging institutions to achieve the full potential
contribution of their research in future
4. The impact pilot exercise______________________________________________________________________________________________________________________________
Tested and developed a case study approach to assessing the
impact of research
Five units of assessment (UOAs)
29 UK higher education institutions each submitting to 2 UOAs
Each submission included:
- An ‘impact statement’ for the submitted unit as a whole
- Case studies illustrating examples of impacts achieved (a total of one
case study per 10 research staff)
Impacts that occurred during 2005-09, underpinned by
research since 1993
8. Key findings______________________________________________________________________________________________________________________________
The process makes explicit the benefits that research in each
discipline brings to society
It is possible to assess the impact of research, through expert
review of case studies
A number of refinements are needed for full implementation
A generic approach is workable, with scope for REF panels to tailor
the criteria as appropriate to their disciplines
The weighting should be significant to be taken seriously by all
stakeholders, and needs careful consideration
12. BIBLIOMETRICS IN THE REF
Royal Society of Medicine
University Health & Medical Librarians’ Group
7 March
13. Background to bibliometrics
Bibliometrics pilot
Preliminary feedback from sub-panels
Next steps
Contents______________________________________________________________________________________________________________________________
15. 22 UK HEIs
All UOAs with reasonable coverage in the bibliometric
databases
Trialled several approaches
Address based model
Author based models
Just RAE submitted people
Selected papers
Bibliometrics pilot______________________________________________________________________________________________________________________________
16. Fully inclusive models difficult
Institutional data verification
Defining subject areas
Cannot use formulaically - though significant potential to
support assessment process
Equalities implications
Conclusions from the pilot______________________________________________________________________________________________________________________________
17. Interest from main panels A and B
Little interest from C and D
Range of options
Simple counts and guidance on use
Fully normalised / benchmarked
REF Sub-panel feedback______________________________________________________________________________________________________________________________
18. Not for use formulaically
As a means of informing peer review
Cross-check/tie break
Reflects academic significance only
Supporting evidence only
Concerns about:
Differential coverage
Discouraging applied/translational work
Timeliness of data
Clarity of guidance on submissions
Normalisation of scores
Time lag on indicators
REF Sub-panel feedback______________________________________________________________________________________________________________________________
19. Procurement of citation data
Discussions with several suppliers
REF steering group will take a decision informed by
Procurement discussions
Panel feedback
Decision published in July
Next steps______________________________________________________________________________________________________________________________