Elizabeth Gadd, Research Policy Manager - Loughborough University Journal publications currently play a significant role in research assessment: in rankings, REF and recruitment. They are assessed by quantity, citedness, the journal they appear in, or peer review. However, assessing journal publications can be difficult and can drive wider system failures such as questionable research practices, rising publication costs, and delays. Increasingly there are calls to evaluate a broader diversity of roles, inputs, processes, and outputs through our research assessments but it’s not clear to what extent these approaches truly value a wider range of research 'qualities'. This participative session will explore the appropriate place of journal publications in research and researcher assessment with the assistance of the INORMS SCOPE framework.

1. Investigating the appropriate role for
publications in research evaluation
Using the INORMS ‘SCOPE’ framework
Dr Elizabeth Gadd
@INORMS_REWG @lizziegadd

2. Overview
 The current role of publications in research
assessment
 How publications are assessed
 The problems of publication-based assessment
 Calls for broader diversity in research assessment
 What is the role of publications, if any, in the
assessment of research & researchers?
 Introducing the SCOPE framework
 Exercise

3. The current role of publications in:
 Individual researcher evaluation
 …and the impact on research project evaluation
 Institutional evaluation

4. EUA ‘Research
Assessment in the
transition to open
science’ report.
https://eua.eu/downloads/publications/research%20assessment
%20in%20the%20transition%20to%20open%20science.pdf

5. The research evaluation food chain
University
rankings
Governments
Funders
Universities
Researchers
Data
vendors

7. Ranking % score based on
publications
Academic Ranking of
World Universities
(Shanghai)
60%
Times Higher
Education
World University
Ranking
38.5%
QS World Ranking 20%

8. How publications are assessed:
 How good?
 How many?
 Where published?
 How cited?
 How internationally/industrially co-authored?

9. How publications are used to make
OTHER forms of assessment
 How interdisciplinary?
 How open?
 Pre-registration
 Data availability
 Code sharing
 How gender balanced?
 Author contribution (First, Last & Corresponding authors)
 AI-informed…
 Peer reviewer selection
 Citation sentiment assessment

10. The challenges of publication-based
research assessment
Technical challenges
Systemic impacts

11. Technical challenges
 Quantitative
 Database coverage
 Hyper-authorship
 Mapping author contribution
 Qualitative
 Getting peer reviewers
 Peer review quality,
 Reproducibility
 Transparency
 Equity
 Efficiency

12. Geographical coverage
Arianna
Becerril-García,
Responsible
Research
Assessment
Conference,
GRC,
November
2020

13. Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and
OpenCitations’ COCI: a multidisciplinary comparison of coverage via citations.
Martin Martin et al. (2019)
Disciplinary coverage

14. Hyper-authorship – what does it
mean to be an author?

15. Mapping author contribution

16. Finding peer reviewers

17. Publication delays

19. Systemic impacts
 Publication bias
 Citation bias
 Publication costs
 Delays
 Paper mills
 Publication ethics (gift, ghost & guest authorship)

20. Skewing the scholarly record
From Professor Dorothy Bishop presentation to King’s Open Research Conference
June 2020

21. From Professor Dorothy Bishop presentation to King’s Open Research Conference June
2020
…which distorts scholarship

22. Citation biases

23. …and intersectionality

24. Publication costs
Zhang, L., Wei, Y., Huang, Y. et al. Should open access lead to closed research? The trends towards paying to perform
research. Scientometrics 127, 7653–7679 (2022). https://doi.org/10.1007/s11192-022-04407-5

25. Heather Morrison et al, 2021, https://sustainingknowledgecommons.org/2021/06/24/open-
access-article-processing-charges-2011-2021/
A link between JIFs and APCs

26. Accelerated publishing – for a fee

27. Paper mills & peer review rings
“All journals regardless of
discipline see up to 2% of
papers submitted that are
identified as potential
fake papers.”
https://publicationethics.org/sites/default/files/paper-mills-cope-stm-research-report.pdf &
https://retractionwatch.com/2022/09/28/exclusive-hindawi-and-wiley-to-retract-over-500-papers-linked-to-peer-review-
rings/

28. Guest, ghost & gift authorship
“For 14 survey
studies, a meta-
analysis showed a
pooled weighted
average of 29% (95%
CI 24% to 35%)
researchers
reporting their own
or others' experience
with misuse of
authorship.”

29. https://scholarlykitchen.sspnet.org/2020/09/03/articles
-are-the-fundamental-unit-of-data-sharing/
Embedding journal articles as the
accounting unit of scholarship.

30. Calls for more diversity in research
assessment

31. 4 CoARA core commitments
1. Recognise the diversity of contributions to, and careers in, research,
in accordance with the needs and the nature of the research
2. Base research assessment primarily on qualitative evaluation for which
peer-review is central, supported by responsible use of quantitative indicators
3. Abandon the inappropriate uses in research assessment of journal-
and publication-based metrics, in particular the inappropriate uses of
journal impact factor (JIF) and h-index
4. Avoid the use of rankings of research organisations
in research assessment
31

32. Introducing the INORMS SCOPE
framework

34. Three
principles of
SCOPE
Evaluate only where necessary
Evaluate with the evaluated
Draw on evaluation expertise

35. START with what you value

36. Context: Why and what are you
measuring?

37. Use of FWCI in measuring to understand

38. Use of FWCI to identify staff for redundancy…

39. Understand who & why you’re
evaluating

40. Options: you have them!
 Is your indicator a suitable proxy for what you are
evaluating?
 Quantitative measures are for quantifiable things…
 Citations, publications, money, students
 Qualitative measures for qualifiable things…
 Quality, excellence, value
 Be careful if using quantitative indicators as a proxy for
qualitative things
 Citations ≠ quality
 Ranking position ≠ excellence

41. Alan Dix
University of Birmingham and Talis
http://alandix.com/ref2014/
Doing metrics responsibly
ARMA Liverpool 2017
metrics are
rubbish
but …
people are worse
(far)

42. PROBE

43. PROBE
1. Who might this discriminate against?
2. What might the unintended
consequences be?
3. How might this be gamed?
4. What is the cost-benefit?

44. EVALUATE

45. Workshop
START with what we value about researchers
 Menti.com 7856 1435
 What makes a good researcher?

46. CONTEXT - Recruitment

47. OPTIONS – To what extent might
publications indicate what we value
about researchers?
 Menti.com 7856 1435
 To what extent might publications indicate what we
value about researchers?
 And how?

48. PROBE – If we used publications as an
indicator what might the consequences
be?
 Menti.com 7856 1435
1. Who might this discriminate against?
2. What might the unintended consequences be?
3. How might this be gamed?
4. What is the cost-benefit?

49. Thank you for listening
 Dr Elizabeth Gadd, Loughborough University, UK
 Chair, INORMS Research Evaluation Group
 Vice-Chair, Coalition on Advancing Research Assessment (CoARA)
 @LizzieGadd
 @INORMS_REWG
 @CoARAssessment