Assessing research impact beyond citations

1. CBMR-SciComm
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 1
Altmetrics
Assessing research impact beyond citations
Daniel Noesgaard
Center for Basic Metabolic Research
Science Communication

2. What is Impact?
CBMR-SciComm
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 2

3. What is impact and how do we measure it?
Classical assessment of research
• Results of research is published as article in journal
• No of citations
Journal impact factor
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 3
CBMR-SciComm

4. The Journal Impact Factor
Definition :
The average number of citations received per paper published
in that journal during the two preceding years
The IF for 2012 is based on citations made in 2012 to papers
published in 2010 and 2011.
Formula:
•Eugene Garfield, 1955
•Originally created as a tool to help librarians select which
journals to purchase
•Calculated and published by Thomson Reuters
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 4
CBMR-SciComm
citations in 2012 to papers from 2010 & 2011
papers from 2010 & 2011

5. CBMR-SciComm
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 5

6. DORA - Declaration On Research Assessment
• Annual meeting - The American Society for Cell Biology
(ASCB) December 2012, San Francisco
• “Putting science into the assessment of research”1
Problems with Journal Impact Factor
• Citation distributions highly skewed
• Can be manipulated by editorial policy
• Calculations are neither transparent, publically available
nor reproducible
Examples
• Editors requests further references2
• Attempts to reproduce IF consistently failed3
• 89% of citations in Nature generated by just 25% of
papers4
1
DORA, http://am.ascb.org/dora/
2
Editorial, PLoS Med 3(6): e291
3
Rossner et al, Journal of Cell Biology 179 (6): 1091–2.
2
Editorial, Nature 435, 1003-1004
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 6
CBMR-SciComm

7. DORA (continued)
Signed by editors
Elizabeth M. Adler Executive Editor, The Journal of General Physiology
Sharon Ahmad Executive Editor, Journal of Cell Science
Kurt H. Albertine Editor-in-Chief, The Anatomical Record
Bruce Alberts Editor-in-Chief, Science
David Botstein Founding Editor-in-Chief of Molecular Biology of the Cell; Director Lewis-Sigler Institute for Integrative Genomics, Princeton versity
Julio E. Celis Editor-in-Chief, Molecular Oncology
Ana Maria Cuervo co-Editor-in-Chief of Aging Cell; Professor, Albert Einstein College of Medicine
Tracey DePellegrin Executive Editor, GENETICS and G3: Genes|Genomes|Genetics
David Drubin Editor-in-Chief, Molecular Biology of the Cell; Professor, University of California, Berkeley
Sir Alan Fersht, FRS Associate Editor, PNAS
Bruce L. Goode Editor, Cytoskeleton; Professor, Biology Rosenstiel Basic Medical Sciences Research Center, Brandeis University
Sharona Gordon Incoming Editor, Journal of General Physiology
Peter Gunning President, Australian Society for Biochemistry and Molecular Biology; Editor-In-Chief, BioArchitecture, University of New South Wales
Richard W. Hartel Editor-in-Chief, Journal of the American Oil Chemists' Society
Dennis W. Hess Editor, ECS Journal of Solid State Science & Technology and ECS Solid State Letters
Steve Humphries Editor-in-Chief, Atherosclerosi, Official Journal of the European Atherosclerosis Society
Howy Jacobs Chief Editor, EMBO Reports
Mark Marsh Co-editor, Traffic; Director, Medical Research Council Laboratory for Molecular Cell Biology
Marc A. Marti-Renom Associate Editor at PLOS Computational Biology; National Center for Genomic Analysis and Centre for Genomic Regulation, Barcelona, Spain
Tom Misteli Editor-in-Chief, The Journal of Cell Biology
Eric Murphy Editor-in-Chief, Lipids, a Journal of the American Oil Chemists' Society
Richard N. Perham Editor-in-Chief, FEBS Journal
Alberto Prestininzi Editor-in -Chief, Italian Journal of Engineering Geology and Environment
Edward N. Pugh, Jr. Editor, Journal of General Physiology
Bernd Pulverer Chief Editor, The EMBO Journal; Head of Scientific Publications, EMBO
Jordan Raff President, British Society of Cell Biology; Editor-in-Chief, Biology Open; Professor, Cancer Cell Biology, University of Oxford.
Jean-Louis Salager Editor-in-Chief, Journal of Surfactants and Detergents
Randy Schekman Editor-in-Chief, eLife
Trina Schroer Co-editor, Traffic; Professor, Johns Hopkins University
Jörg Schulz Editor-in-Chief, Journal of Neurochemistry; Chair and Full Professor, Department of Neurology, RWTH Aachen University, Germany
Tom Stevens Co-editor, Traffic; Professor, University of Oregon
Marlowe Tessmer Senior Editor, The Journal of Experimental Medicine
Inder Verma Editor-in-Chief, Proceedings of The National Academy Of Sciences (PNAS)
Michael Way Editor-in-Chief, Journal of Cell Science
Felix Wieland Managing Editor, FEBS Letters
Mitsuhiro Yanagida Editor-in-Chief, Genes to Cells
Wellcome Trust
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 7
CBMR-SciComm

8. Editor comments about impacts factors
Bruce Alberts, Editor-in-Chief, Science, 17 May 2013
“The impact factor, (a number calculated annually for each scientific journal based on
the average number of times its articles have been referenced in other articles),
was never intended to be used to evaluate individual scientists, but rather as a
measure of journal quality. However, it has been increasingly misused in this
way, with scientists now being ranked by weighting each of their publications
according to the impact factor of the journal in which it appeared. For this reason,
I have seen CVs in which a scientist annotates each of his or her publications
with its journal impact factor listed to three significant decimal places (for
example, 11.345). And in some nations, publication in a journal with an
impact factor below 5.0 is officially of zero value. As frequently pointed out
by leading scientists, this impact factor mania makes no sense.”1
Tom Misteli, Editor-in-Chief, Journal of Cell Biology, 20 May 2013
“The IF is pervasive in the scientific community. Scientists refer to it casually in
conversation to convince colleagues of the importance of their own papers, or
they wonder how a paper ended up in “a journal with such a high Impact Factor.”
Students and postdocs want to publish only in “high Impact Factor” journals, and
the IF is frequently used in recruitment, tenure, and granting decisions when a
candidate’s past publication performance is assessed…The IF was never meant to
be used in that way!”2
1
Alberts, Science, 10.1126/science.1240319
2
Mistelli, JCB, 10.1083/jcb.201304162
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 8
CBMR-SciComm

9. DORA recommendations
• Do not use journal-based metrics, such as Journal Impact
Factors, as a surrogate measure of the quality of individual
research articles, to assess an individual scientist’s
contributions, or in hiring, promotion, or funding decisions.
• Specific recommendations for funding agencies, institutions,
publishers, and researchers:
• When involved in committees making decisions about funding, hiring,
tenure, or promotion, make assessments based on scientific content
rather than publication metrics.
• Wherever appropriate, cite primary literature in which observations are
first reported rather than reviews in order to give credit where credit is due.
• Use a range of article metrics and indicators on personal/supporting
statements, as evidence of the impact of individual published articles and
other research outputs
• Challenge research assessment practices that rely inappropriately on
Journal Impact Factors and promote and teach best practice that focuses on
the value and influence of specific research outputs.
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 9
CBMR-SciComm

10. What is Impact?
CBMR-SciComm
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 10

11. Impact?
• Paper on sexually-sustained injury downloaded 40,000 times
– impact?
• Randomized trial which proves that cheap antibiotic can
prevent the spread of opportunistic infections – and
ultimately, deaths – in children with HIV – impact?
• Review cited hundreds of times – impact?
• Arsenic life paper picked up and vigorously discussed among
bloggers – impact?
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 11
CBMR-SciComm

12. What is Altmetrics?
CBMR-SciComm
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 12

13. Altmetrics
• Term coined by Jason Priem, UNC Chapel Hill, in 20101
• A form of article-level metrics (<- journal-level metrics)
• Measuring things other than traditional citations
• “Creation and study of new metrics based on the Social Web
for analyzing, and informing scholarship”2
• Track the impact of research output by looking at
conversations, discussions, mentions, etc. generated based
on a specific research paper
• Fast! Appears in days/week rather than years
• Diverse! Scholarly communications across different
products, platforms and audiences
1 https://twitter.com/#/jasonpriem/status/25844968813
2
Priem et al, Altmetrics: A manifesto, (v.1.0), 26 October 2010. http://altmetrics.org/manifesto
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 13
CBMR-SciComm

14. Altmetrics - continued
• PLOS + another example
• Nature + another example
• Altmetric.com – metrics provider
• Cell
• Bookmarklet for others
• Predictions – Twitter: Highly tweeted articles 11 times more
likely to be highly cited than less-tweeted articles1
• ImpactStory
• PlumX
1
Eysenbach, J Med Internet Res 2011;13(4):e123
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 14
CBMR-SciComm

15. Why should I care / what can I do?
• Why do you publish? To ensure the next grant?
• “If you are a scientist, you care about about letting people
read your science. If you don't, you're an alchemist!”
• Altmetrics will allow funding agencies and employers to
making more informed decisions - rather than relying on gut
feelings, antiquated systems
• Track your work!
• Discover new “flavors” of impact beyond citation
• Take part in discussions around your work
• Make a live CV!
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 15
CBMR-SciComm

16. More information
• Websites
• altmetrics.org
• StartPublicering.nu
• Papers
• The Altmetrics Collections (PLOS One)
• Riding the crest of the altmetrics wave
• Impact factor: a valid measure of journal quality?
• Altmetrics: Rethinking the Way We Measure
• Blogs
• Martin Fenner, MD (PLOS)
• Jason Priem
• Ask your colleagues in SciComm! 
CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 16
CBMR-SciComm

17. CBMR Summer Retreat, Pharmakon, 23 May 2013
Slide 17
CBMR-SciComm