• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content




Assessing research impact beyond citations

Assessing research impact beyond citations



Total Views
Views on SlideShare
Embed Views



1 Embed 8

https://twitter.com 8


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Altmetrics Altmetrics Presentation Transcript

    • CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 1AltmetricsAssessing research impact beyond citationsDaniel NoesgaardCenter for Basic Metabolic ResearchScience Communication
    • What is Impact?CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 2
    • What is impact and how do we measure it?Classical assessment of research• Results of research is published as article in journal• No of citationsJournal impact factorCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 3CBMR-SciComm
    • The Journal Impact FactorDefinition :The average number of citations received per paper publishedin that journal during the two preceding yearsThe IF for 2012 is based on citations made in 2012 to paperspublished in 2010 and 2011.Formula:•Eugene Garfield, 1955•Originally created as a tool to help librarians select whichjournals to purchase•Calculated and published by Thomson ReutersCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 4CBMR-SciCommcitations in 2012 to papers from 2010 & 2011papers from 2010 & 2011
    • CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 5
    • DORA - Declaration On Research Assessment• Annual meeting - The American Society for Cell Biology(ASCB) December 2012, San Francisco• “Putting science into the assessment of research”1Problems with Journal Impact Factor• Citation distributions highly skewed• Can be manipulated by editorial policy• Calculations are neither transparent, publically availablenor reproducibleExamples• Editors requests further references2• Attempts to reproduce IF consistently failed3• 89% of citations in Nature generated by just 25% ofpapers41DORA, http://am.ascb.org/dora/2Editorial, PLoS Med 3(6): e2913Rossner et al, Journal of Cell Biology 179 (6): 1091–2.2Editorial, Nature 435, 1003-1004CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 6CBMR-SciComm
    • DORA (continued)Signed by editorsElizabeth M. Adler Executive Editor, The Journal of General PhysiologySharon Ahmad Executive Editor, Journal of Cell ScienceKurt H. Albertine Editor-in-Chief, The Anatomical RecordBruce Alberts Editor-in-Chief, ScienceDavid Botstein Founding Editor-in-Chief of Molecular Biology of the Cell; Director Lewis-Sigler Institute for Integrative Genomics, Princeton versityJulio E. Celis Editor-in-Chief, Molecular OncologyAna Maria Cuervo co-Editor-in-Chief of Aging Cell; Professor, Albert Einstein College of MedicineTracey DePellegrin Executive Editor, GENETICS and G3: Genes|Genomes|GeneticsDavid Drubin Editor-in-Chief, Molecular Biology of the Cell; Professor, University of California, BerkeleySir Alan Fersht, FRS Associate Editor, PNASBruce L. Goode Editor, Cytoskeleton; Professor, Biology Rosenstiel Basic Medical Sciences Research Center, Brandeis UniversitySharona Gordon Incoming Editor, Journal of General PhysiologyPeter Gunning President, Australian Society for Biochemistry and Molecular Biology; Editor-In-Chief, BioArchitecture, University of New South WalesRichard W. Hartel Editor-in-Chief, Journal of the American Oil Chemists SocietyDennis W. Hess Editor, ECS Journal of Solid State Science & Technology and ECS Solid State LettersSteve Humphries Editor-in-Chief, Atherosclerosi, Official Journal of the European Atherosclerosis SocietyHowy Jacobs Chief Editor, EMBO ReportsMark Marsh Co-editor, Traffic; Director, Medical Research Council Laboratory for Molecular Cell BiologyMarc A. Marti-Renom Associate Editor at PLOS Computational Biology; National Center for Genomic Analysis and Centre for Genomic Regulation, Barcelona, SpainTom Misteli Editor-in-Chief, The Journal of Cell BiologyEric Murphy Editor-in-Chief, Lipids, a Journal of the American Oil Chemists SocietyRichard N. Perham Editor-in-Chief, FEBS JournalAlberto Prestininzi Editor-in -Chief, Italian Journal of Engineering Geology and EnvironmentEdward N. Pugh, Jr. Editor, Journal of General PhysiologyBernd Pulverer Chief Editor, The EMBO Journal; Head of Scientific Publications, EMBOJordan Raff President, British Society of Cell Biology; Editor-in-Chief, Biology Open; Professor, Cancer Cell Biology, University of Oxford.Jean-Louis Salager Editor-in-Chief, Journal of Surfactants and DetergentsRandy Schekman Editor-in-Chief, eLifeTrina Schroer Co-editor, Traffic; Professor, Johns Hopkins UniversityJörg Schulz Editor-in-Chief, Journal of Neurochemistry; Chair and Full Professor, Department of Neurology, RWTH Aachen University, GermanyTom Stevens Co-editor, Traffic; Professor, University of OregonMarlowe Tessmer Senior Editor, The Journal of Experimental MedicineInder Verma Editor-in-Chief, Proceedings of The National Academy Of Sciences (PNAS)Michael Way Editor-in-Chief, Journal of Cell ScienceFelix Wieland Managing Editor, FEBS LettersMitsuhiro Yanagida Editor-in-Chief, Genes to CellsWellcome TrustCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 7CBMR-SciComm
    • Editor comments about impacts factorsBruce Alberts, Editor-in-Chief, Science, 17 May 2013“The impact factor, (a number calculated annually for each scientific journal based onthe average number of times its articles have been referenced in other articles),was never intended to be used to evaluate individual scientists, but rather as ameasure of journal quality. However, it has been increasingly misused in thisway, with scientists now being ranked by weighting each of their publicationsaccording to the impact factor of the journal in which it appeared. For this reason,I have seen CVs in which a scientist annotates each of his or her publicationswith its journal impact factor listed to three significant decimal places (forexample, 11.345). And in some nations, publication in a journal with animpact factor below 5.0 is officially of zero value. As frequently pointed outby leading scientists, this impact factor mania makes no sense.”1Tom Misteli, Editor-in-Chief, Journal of Cell Biology, 20 May 2013“The IF is pervasive in the scientific community. Scientists refer to it casually inconversation to convince colleagues of the importance of their own papers, orthey wonder how a paper ended up in “a journal with such a high Impact Factor.”Students and postdocs want to publish only in “high Impact Factor” journals, andthe IF is frequently used in recruitment, tenure, and granting decisions when acandidate’s past publication performance is assessed…The IF was never meant tobe used in that way!”21Alberts, Science, 10.1126/science.12403192Mistelli, JCB, 10.1083/jcb.201304162CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 8CBMR-SciComm
    • DORA recommendations• Do not use journal-based metrics, such as Journal ImpactFactors, as a surrogate measure of the quality of individualresearch articles, to assess an individual scientist’scontributions, or in hiring, promotion, or funding decisions.• Specific recommendations for funding agencies, institutions,publishers, and researchers:• When involved in committees making decisions about funding, hiring,tenure, or promotion, make assessments based on scientific contentrather than publication metrics.• Wherever appropriate, cite primary literature in which observations arefirst reported rather than reviews in order to give credit where credit is due.• Use a range of article metrics and indicators on personal/supportingstatements, as evidence of the impact of individual published articles andother research outputs• Challenge research assessment practices that rely inappropriately onJournal Impact Factors and promote and teach best practice that focuses onthe value and influence of specific research outputs.CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 9CBMR-SciComm
    • What is Impact?CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 10
    • Impact?• Paper on sexually-sustained injury downloaded 40,000 times– impact?• Randomized trial which proves that cheap antibiotic canprevent the spread of opportunistic infections – andultimately, deaths – in children with HIV – impact?• Review cited hundreds of times – impact?• Arsenic life paper picked up and vigorously discussed amongbloggers – impact?CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 11CBMR-SciComm
    • What is Altmetrics?CBMR-SciCommCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 12
    • Altmetrics• Term coined by Jason Priem, UNC Chapel Hill, in 20101• A form of article-level metrics (<- journal-level metrics)• Measuring things other than traditional citations• “Creation and study of new metrics based on the Social Webfor analyzing, and informing scholarship”2• Track the impact of research output by looking atconversations, discussions, mentions, etc. generated basedon a specific research paper• Fast! Appears in days/week rather than years• Diverse! Scholarly communications across differentproducts, platforms and audiences1 https://twitter.com/#/jasonpriem/status/258449688132Priem et al, Altmetrics: A manifesto, (v.1.0), 26 October 2010. http://altmetrics.org/manifestoCBMR Summer Retreat, Pharmakon, 23 May 2013Slide 13CBMR-SciComm
    • Altmetrics - continued• PLOS + another example• Nature + another example• Altmetric.com – metrics provider• Cell• Bookmarklet for others• Predictions – Twitter: Highly tweeted articles 11 times morelikely to be highly cited than less-tweeted articles1• ImpactStory• PlumX1Eysenbach, J Med Internet Res 2011;13(4):e123CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 14CBMR-SciComm
    • Why should I care / what can I do?• Why do you publish? To ensure the next grant?• “If you are a scientist, you care about about letting peopleread your science. If you dont, youre an alchemist!”• Altmetrics will allow funding agencies and employers tomaking more informed decisions - rather than relying on gutfeelings, antiquated systems• Track your work!• Discover new “flavors” of impact beyond citation• Take part in discussions around your work• Make a live CV!CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 15CBMR-SciComm
    • More information• Websites• altmetrics.org• StartPublicering.nu• Papers• The Altmetrics Collections (PLOS One)• Riding the crest of the altmetrics wave• Impact factor: a valid measure of journal quality?• Altmetrics: Rethinking the Way We Measure• Blogs• Martin Fenner, MD (PLOS)• Jason Priem• Ask your colleagues in SciComm! CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 16CBMR-SciComm
    • CBMR Summer Retreat, Pharmakon, 23 May 2013Slide 17CBMR-SciComm