Published on

Assessing research impact beyond citations

Published in: Technology, Health & Medicine
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 1AltmetricsAssessing research impact beyond citationsDaniel NoesgaardCenter for Basic Metabolic ResearchScience Communication
  2. 2. What is Impact?CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 2
  3. 3. What is impact and how do we measure it?Classical assessment of research• Results of research is published as article in journal• No of citationsJournal impact factorCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 3CBMR-SciComm
  4. 4. The Journal Impact FactorDefinition :The average number of citations received per paper publishedin that journal during the two preceding yearsThe IF for 2012 is based on citations made in 2012 to paperspublished in 2010 and 2011.Formula:•Eugene Garfield, 1955•Originally created as a tool to help librarians select whichjournals to purchase•Calculated and published by Thomson ReutersCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 4CBMR-SciCommcitations in 2012 to papers from 2010 & 2011papers from 2010 & 2011
  5. 5. CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 5
  6. 6. DORA - Declaration On Research Assessment• Annual meeting - The American Society for Cell Biology(ASCB) December 2012, San Francisco• “Putting science into the assessment of research”1Problems with Journal Impact Factor• Citation distributions highly skewed• Can be manipulated by editorial policy• Calculations are neither transparent, publically availablenor reproducibleExamples• Editors requests further references2• Attempts to reproduce IF consistently failed3• 89% of citations in Nature generated by just 25% ofpapers41DORA,, PLoS Med 3(6): e2913Rossner et al, Journal of Cell Biology 179 (6): 1091–2.2Editorial, Nature 435, 1003-1004CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 6CBMR-SciComm
  7. 7. DORA (continued)Signed by editorsElizabeth M. Adler Executive Editor, The Journal of General PhysiologySharon Ahmad Executive Editor, Journal of Cell ScienceKurt H. Albertine Editor-in-Chief, The Anatomical RecordBruce Alberts Editor-in-Chief, ScienceDavid Botstein Founding Editor-in-Chief of Molecular Biology of the Cell; Director Lewis-Sigler Institute for Integrative Genomics, Princeton versityJulio E. Celis Editor-in-Chief, Molecular OncologyAna Maria Cuervo co-Editor-in-Chief of Aging Cell; Professor, Albert Einstein College of MedicineTracey DePellegrin Executive Editor, GENETICS and G3: Genes|Genomes|GeneticsDavid Drubin Editor-in-Chief, Molecular Biology of the Cell; Professor, University of California, BerkeleySir Alan Fersht, FRS Associate Editor, PNASBruce L. Goode Editor, Cytoskeleton; Professor, Biology Rosenstiel Basic Medical Sciences Research Center, Brandeis UniversitySharona Gordon Incoming Editor, Journal of General PhysiologyPeter Gunning President, Australian Society for Biochemistry and Molecular Biology; Editor-In-Chief, BioArchitecture, University of New South WalesRichard W. Hartel Editor-in-Chief, Journal of the American Oil Chemists SocietyDennis W. Hess Editor, ECS Journal of Solid State Science & Technology and ECS Solid State LettersSteve Humphries Editor-in-Chief, Atherosclerosi, Official Journal of the European Atherosclerosis SocietyHowy Jacobs Chief Editor, EMBO ReportsMark Marsh Co-editor, Traffic; Director, Medical Research Council Laboratory for Molecular Cell BiologyMarc A. Marti-Renom Associate Editor at PLOS Computational Biology; National Center for Genomic Analysis and Centre for Genomic Regulation, Barcelona, SpainTom Misteli Editor-in-Chief, The Journal of Cell BiologyEric Murphy Editor-in-Chief, Lipids, a Journal of the American Oil Chemists SocietyRichard N. Perham Editor-in-Chief, FEBS JournalAlberto Prestininzi Editor-in -Chief, Italian Journal of Engineering Geology and EnvironmentEdward N. Pugh, Jr. Editor, Journal of General PhysiologyBernd Pulverer Chief Editor, The EMBO Journal; Head of Scientific Publications, EMBOJordan Raff President, British Society of Cell Biology; Editor-in-Chief, Biology Open; Professor, Cancer Cell Biology, University of Oxford.Jean-Louis Salager Editor-in-Chief, Journal of Surfactants and DetergentsRandy Schekman Editor-in-Chief, eLifeTrina Schroer Co-editor, Traffic; Professor, Johns Hopkins UniversityJörg Schulz Editor-in-Chief, Journal of Neurochemistry; Chair and Full Professor, Department of Neurology, RWTH Aachen University, GermanyTom Stevens Co-editor, Traffic; Professor, University of OregonMarlowe Tessmer Senior Editor, The Journal of Experimental MedicineInder Verma Editor-in-Chief, Proceedings of The National Academy Of Sciences (PNAS)Michael Way Editor-in-Chief, Journal of Cell ScienceFelix Wieland Managing Editor, FEBS LettersMitsuhiro Yanagida Editor-in-Chief, Genes to CellsWellcome TrustCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 7CBMR-SciComm
  8. 8. Editor comments about impacts factorsBruce Alberts, Editor-in-Chief, Science, 17 May 2013“The impact factor, (a number calculated annually for each scientific journal based onthe average number of times its articles have been referenced in other articles),was never intended to be used to evaluate individual scientists, but rather as ameasure of journal quality. However, it has been increasingly misused in thisway, with scientists now being ranked by weighting each of their publicationsaccording to the impact factor of the journal in which it appeared. For this reason,I have seen CVs in which a scientist annotates each of his or her publicationswith its journal impact factor listed to three significant decimal places (forexample, 11.345). And in some nations, publication in a journal with animpact factor below 5.0 is officially of zero value. As frequently pointed outby leading scientists, this impact factor mania makes no sense.”1Tom Misteli, Editor-in-Chief, Journal of Cell Biology, 20 May 2013“The IF is pervasive in the scientific community. Scientists refer to it casually inconversation to convince colleagues of the importance of their own papers, orthey wonder how a paper ended up in “a journal with such a high Impact Factor.”Students and postdocs want to publish only in “high Impact Factor” journals, andthe IF is frequently used in recruitment, tenure, and granting decisions when acandidate’s past publication performance is assessed…The IF was never meant tobe used in that way!”21Alberts, Science, 10.1126/science.12403192Mistelli, JCB, 10.1083/jcb.201304162CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 8CBMR-SciComm
  9. 9. DORA recommendations• Do not use journal-based metrics, such as Journal ImpactFactors, as a surrogate measure of the quality of individualresearch articles, to assess an individual scientist’scontributions, or in hiring, promotion, or funding decisions.• Specific recommendations for funding agencies, institutions,publishers, and researchers:• When involved in committees making decisions about funding, hiring,tenure, or promotion, make assessments based on scientific contentrather than publication metrics.• Wherever appropriate, cite primary literature in which observations arefirst reported rather than reviews in order to give credit where credit is due.• Use a range of article metrics and indicators on personal/supportingstatements, as evidence of the impact of individual published articles andother research outputs• Challenge research assessment practices that rely inappropriately onJournal Impact Factors and promote and teach best practice that focuses onthe value and influence of specific research outputs.CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 9CBMR-SciComm
  10. 10. What is Impact?CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 10
  11. 11. Impact?• Paper on sexually-sustained injury downloaded 40,000 times– impact?• Randomized trial which proves that cheap antibiotic canprevent the spread of opportunistic infections – andultimately, deaths – in children with HIV – impact?• Review cited hundreds of times – impact?• Arsenic life paper picked up and vigorously discussed amongbloggers – impact?CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 11CBMR-SciComm
  12. 12. What is Altmetrics?CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 12
  13. 13. Altmetrics• Term coined by Jason Priem, UNC Chapel Hill, in 20101• A form of article-level metrics (<- journal-level metrics)• Measuring things other than traditional citations• “Creation and study of new metrics based on the Social Webfor analyzing, and informing scholarship”2• Track the impact of research output by looking atconversations, discussions, mentions, etc. generated basedon a specific research paper• Fast! Appears in days/week rather than years• Diverse! Scholarly communications across differentproducts, platforms and audiences1 et al, Altmetrics: A manifesto, (v.1.0), 26 October 2010. Summer Retreat, Pharmakon, 23 May 2013Slide 13CBMR-SciComm
  14. 14. Altmetrics - continued• PLOS + another example• Nature + another example• – metrics provider• Cell• Bookmarklet for others• Predictions – Twitter: Highly tweeted articles 11 times morelikely to be highly cited than less-tweeted articles1• ImpactStory• PlumX1Eysenbach, J Med Internet Res 2011;13(4):e123CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 14CBMR-SciComm
  15. 15. Why should I care / what can I do?• Why do you publish? To ensure the next grant?• “If you are a scientist, you care about about letting peopleread your science. If you dont, youre an alchemist!”• Altmetrics will allow funding agencies and employers tomaking more informed decisions - rather than relying on gutfeelings, antiquated systems• Track your work!• Discover new “flavors” of impact beyond citation• Take part in discussions around your work• Make a live CV!CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 15CBMR-SciComm
  16. 16. More information• Websites••• Papers• The Altmetrics Collections (PLOS One)• Riding the crest of the altmetrics wave• Impact factor: a valid measure of journal quality?• Altmetrics: Rethinking the Way We Measure• Blogs• Martin Fenner, MD (PLOS)• Jason Priem• Ask your colleagues in SciComm! CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 16CBMR-SciComm
  17. 17. CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 17CBMR-SciComm