The document discusses recommendations from a review of HEFCE metrics in 2015, including using indicators instead of metrics, ensuring peer review is not replaced by metrics, being transparent about how institutions use data, and dialoging with staff. It also notes that journal impact factors do not accurately reflect the variation in citations individual papers receive and that distributions of citations are highly skewed across journals. The document calls for redefining good behaviors in research and incentivizing behaviors like research quality, open and efficient publishing practices, and contributions to mentoring and peer review.

1. HEFCE Metrics Review 2015: Main Recommendations
Responsible metrics
• Mind your language: indicators, not metrics
• Metrics inform but do not replace peer review
• Institutions need to be transparent about use
• Clearly state of principles for assessment
• Do not delegate measures of excellence to league tables or journals
• Dialogue with staff
• Data should to be transparent (challenge to providers)
• Build on DORA/Leiden Manifesto
Prof. Stephen Curry
Imperial College London

2. Words are not enough: metrics are a deep cultural problem (reified by the REF)
“In some cases the culture of scientific research does not
support or encourage scientists’ goals and the activities
they believe to be important for… high quality science."
http://nuffieldbioethics.org/project/research-culture/
“The most common complaint
from reviewers is that authors are
overselling their work.”
Jan 2015
Aug 2014
Aug 2012

3. 0
5
10
15
20
0-4 25-29 50-54 75-79 100-104 125-129 150-154 175-179 200-204
%ofPapers
Citations per paper in 2010
Data for Nature Materials, 2010
Impact factor = 29.897
Mode = 7
Range (>2 orders of magnitude)
Distribution highly skewed:
Highest cited 15% of papers account for 50% of citations
Two-thirds of papers perform less well than the JIF
http://www.nature.com/doifinder/10.1038/
Evidence: JIFs mask real variation in citation performance

4. Kravitz, D. J., & Baker, C. I. (2011). Frontiers Comp. Neurosci. doi: 10.3389/fncom.2011.00055
Distribution highly skewed:
Highest cited 15% of papers account for 50% of citations
Two-thirds of papers perform less well than the JIF
Evidence: Distributions are similar: all journals have highly & lowly cited papers
See also:
https://quantixed.wordpress.com/2016/01/05/t
he-great-curve-ii-citation-distributions-and-
reverse-engineering-the-jif/
All journals
should publish
their citation
distributions

5. Evidence: Correlations between JIF and article citations for individual scientists are poor
“…authors do not necessarily
publish their most citable work
in journals of the highest
impact, nor do their articles
necessarily match the impact
of the journals they appear in.”
Seglen, P. O. (1997).
BMJ, 314, 498–502.
r=0.05 r=0.27
r=0.63r=0.44
Journal Impact Factor
Noofcitationsperarticle
A B
DC

6. Data may not be enough: re-define good behaviour and incentivise it
Ron Vale (2012) Mol. Biol. Cell, 23, 3286-3289
“As stewards of our profession, academic scientists
have a collective responsibility to consider how to
disseminate knowledge through publications and
how to advance (careers)…”
• Research quality and reproducibility (not just novelty)
• Publishing openly and efficiently (preprints, open access, sharing data & reagents)
• Mentoring and citizenship (e.g. contributions to peer review)
Do we still remember what
universities are for?