Research metrics Apr2013


Published on

Research Metrics Tools: Measuring Scientific Output

Published in: Education, Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Research metrics Apr2013

  1. 1. Research Metrics: MeasuringScientific OutputNazi TorabiResearch & Instructional LibrarianAllyn & Betty Taylor LibraryApril 25, 2013
  2. 2. 1. To provide an overview of variousbibliometrics products/tools2. Have a good understanding of limitation ofeach product3. How to calculate your own h-indexObjectives2
  3. 3. AgendaJournal Ranking•Journal Citaiton Report- Impact Factor•Eigenfactor•Scopus (SNIP)•SCImago Journal Rank (SJR)Article Ranking•Web of Science•F1000•Google: POP•ScopusAuthor Metrics: H-index•Web of Science•Scopus•Google•and H-index variantsAltmetrics• Use of social Media to evaluatescientific impactTools for Researchers Profile• Google Citation Profile• Researcher ID• ORCID3
  4. 4. The Metrics Principles• Publication count VS. Publication quality (impact)• Citation countso Total number of citation counto Total number of citation count in particular time spano Total number of citation per papero Number of papers cited more than n timeso Number of citations to the n most cited papers• Normalized citation countso Number of authors for each papero discipline-normalized score4
  5. 5. Journal Ranking• Journal Citation Report– Impact Factor– journal Immediacy Index• Eigenfactor– Eiginfactor Score– Article Influence Score• Scopus (SNIP)• SCImago Journal Rank (SJR)5Key Metrics/ unitKey Metrics/ unit
  6. 6. •“IF of a journal is the average number of citations receivedper paper published in that journal during the two precedingyears.”Source•IF provides Journal ranking by discipline•JCR provides citing pattern analysis in discipline6
  7. 7. Impact FactorNumber of times articles published in 2008 and 2009were cited by indexed journals during 2010Total number of "citable items" published by thatjournal in 2008 and 20092010 Journal IF =i.e. An Impact Factor of 1.0 in 2013 for ajournal means that, on average, the articlespublished in 2011 and 2012 in that journalhave been cited one time.•Self-citation is included.•Citable items are usually articles, reviews, proceedings, or notes; noteditorials or Letters-to-the-Editor. 7
  8. 8. Journal Immediacy Indexi.e. The journal Immediacy Index indicateshow quickly articles in a journal are cited.Number of times articles published in 2010 were citedby indexed journals during 2010Total number of "citable items" published by thatjournal in 20102010 JournalImmediacy =IndexJCR8
  9. 9. Journal: CHEMBIOCHEM9JCR Video on how to find IF
  10. 10. For 2011, the journal CHEMBIOCHEMhas an Impact Factor of 3.944.Category NameTotal Journalsin CategoryJournal Rankin CategoryQuartilein CategoryBIOCHEMISTRY &MOLECULARBIOLOGY290 84 Q2CHEMISTRY,MEDICINAL59 6 Q110
  11. 11. IF & its shortcomings• Citation pattern is discipline specific– Time span– Citation frequency– differences in citation norms• Sleeping beauties• Review Journals• Self-Citation!• Negative citations• Title and journal format changes• A questionable validity11
  12. 12. Probability density function for journals in subjectcategories of ENERGY & FUELS – data from JCR-5 -4 -3 -2 -1 0 1 2 3 4 5Normal distributionPareto distribution12IF is based on thearithmetic meannumber of citationsper paperThe reality ofcitation patterns isa Paretodistribution
  13. 13. Articles on validity of IF• Dissecting our impact factor. Nature Materials10, 645 (2011) doi:10.1038/nmat3114• Bollen, Johan, et al. "A principal componentanalysis of 39 scientific impact measures."PloS one 4.6 (2009): e6022.• Priem, Jason, and Bradely H. Hemminger."Scientometrics 2.0: New metrics of scholarlyimpact on the social Web." First Monday 15.7(2010).13
  14. 14. Eigenfactor.org14
  15. 15. Citation Networks & Network Effects15
  16. 16. Scholarship is theflow of idea.16
  17. 17. High School DatingNetworkBearman, Peter S., James Moody, and Katherine Stovel."Chains of Affection: The Structure of AdolescentRomantic and Sexual Networks." American Journal ofSociology 110, no. 1 (July 2004): 44-91.17
  18. 18. Which network you would rather bepart of?West, J (2010). The EigenfactorTM Metrics: a network approach toassessing scholarly journals. B
  19. 19. The Rich gets Richer•Journals are ranked according to the citation network.•Journals are influential when they are cited by other influentialjournals.•“The Eigenfactor™ score of a journal is an estimate of thepercentage of time that researchers spend with that journal.”19
  20. 20.
  21. 21. Article Influence™ ScoreJournal’s Eigenfactor Score (5yrs)Number of articles in the journalArticle Influence™ =Score21
  22. 22. IF vs. Eigenfactor22
  23. 23. e.g. of Article Influence™ Score23
  24. 24. More on Journal RankingSJR• SCImago Journal Rank isweighted by the prestige ofa journal. Subject field,quality and reputation ofthe journal have a directeffect on the value of acitation.• SJR normalizes fordifferences in citationbehavior between subjectfields.SNIP• Source Normalized Impactper Paper measurescontextual citation impactby weighting citations basedon the total number ofcitations in a subject field.• It corrects for differences inthe frequency of citationacross research fields.244 years of data is required to calculate SJR and SNIP
  25. 25. 25
  26. 26. Journal Ranking SummaryIF Eigenfactor®ScoreSNIP SJR indicatorSources Database JCR Eigenfactor Scopus/CWTS Scopus/SCImagoCitation timeframe2 years 5 years 4 years 4 yearsJournal self-citationIncluded Excluded Limited LimitedCitation Value Unweighted Weighted Weighted WeightedConnections NA Normalized by thenumber of identifiedreferences in thecitingJournalCorrects fordifferences in thefrequency ofcitation acrossresearch fieldsNormalized bythe totalnumber ofreferences inthe citing journalAdapted and modified from: González-Pereir , B. et al. “The SJR indicator: A new indicator ofjournals scientific prestige”. 26
  27. 27. Articles Ranking27
  28. 28. •Key metric: F1000 Article Factors (FFa)•Medicine and Biology specific28
  29. 29. F1000 factorsThe F1000 Article Factor (FFa) is calculated from the highest ratingawarded by a Faculty Member (FM) plus an increment for eachadditional rating from other FMs.For example, a single article that has been evaluated by three FacultyMembers, who rated it ‘Recommended’, ‘Must Read’ and ‘Must Read’,will have its FFa calculated thus:8 (highest rating ‘Must Read’) + 2 (increment for ‘Must Read’) + 1 (increment for ‘Recommended’) = 11.Rating Value IncrementExceptional 10 3Must Read 8 2Recommended 6 129
  30. 30. 30
  31. 31. 31
  32. 32. Citation overview from Scopus• The Citation Overview Results page listsdocuments that cited the document you selectedfrom the Citation Overview.• You can include up to 5000 documents in aCitation Tracker. If you select more than 5000documents only the first 5000 are included.• More
  33. 33. 33
  34. 34. 34
  35. 35. 35
  36. 36. Harzings Publish or PerishPublish or Perish is a software program that retrieves andanalyzes academic citations. It uses Google Scholar toobtain the raw citations, then analyzes these andpresents the following statistics:o Total number of paperso Total number of citationso Average number of citations per papero Average number of citations per authoro Average number of papers per authoro Average number of citations per yearo h-index and related parameterso An analysis of the number of authors per paper.36Download and install Publish or Perish at
  37. 37. 37
  38. 38. Google Scholar vs. WOS dataset• Transparency• Coverageo Better coverage for unique citationso Not limited to journal articles published in ISI listedjournalso Greater geographical coverage for non-US/UK basedjournalso Better coverage of non-English sources• Includes more comprehensive data for junioracademics38
  39. 39. Google Scholar vs. WOS dataset cont’• Accounts for minor citation errors• Does not account for duplications• Includes non-scholarly citations such asstudent handbooks, library guides or editorialnotes.• Unclear coverageo uneven coverage across different fields of studyo older materials are not well covered39
  40. 40. A researcher’s H-index is the largest possible number nfor which n of the researcher’s publications have beencited at least n times.i.e. If a researcher has an h-index of 25 if 25 of theirpapers have been cited at least 25 times (See the graphin the handout).40
  41. 41. h-Index exampleResearcher A Researcher B Researcher C Researcher DCitations for most-citedarticle3 5 14 8Citations for second most-cited article3 5 1 3Citations for third most-citedarticle3 5 0 2Citations for fourth most-cited article3 0 0 1Citations for fifth most-citedarticle3 0 0 1Average citations per paper 3 3 3 3H-index 3 3 1 241Jarvey, P. (2012). Making Research Count: Analyzing Canadian Academic Publishing Cultures
  42. 42. Citation Report from WOSThe Citation Report provides aggregate citationstatistics for a set of search results. Thesestatistics include:o The total number of times all items have been citedo The average number of times an item has been citedo The number of times an item has been cited each yearo The average number of times an item has been citedin a yearo More info42
  43. 43. 43
  44. 44. Finding H-Index• Scopus• Google44
  45. 45. Why h-Index• It considers both productivity and impact.• It is not influenced by a small number of verysuccessful articles, which may not berepresentative of a researcher’s career output.The H-index similarly discounts the value ofpapers that are not influential.• The H-index is simpler and easier tounderstand than many other compoundbibliometric metrics.45
  46. 46. h-index shortcomings• Early career researchers• Number of authors• Order of authorship• Sleeping beauties• the time since publication of individual articles• Databases are not error free• Inconsistency between different metric tools (Seethe handout).• The balance between number of publications vs.number of citations each receives46Adapted from Jarvey, P. (2012). Making Research Count: Analyzing Canadian Academic Publishing Cultures
  47. 47. number of publications vs. number ofcitations each receives47Researcher A : 5 articles published – each cited 5 timesResearcher B : 2 articles published – each cited 20 timesResearcher A (5, 5, 5, 5, 5 ) = h-index = 5Researcher B (20, 20) = h-index = 20
  48. 48. Selected h-index Variants• g-index: more weight is given to the highest citedpapers• individual h-index: accounts for co-authorship incalculating impact by giving less weight to such papers• contemporary h-index: less weight to older citedpapers• i10 index : from Google Scholars, indicates the numberof papers which have at least ten citations• age-weighted citation rate: accounts for the age ofpapersTarma Software Research (2010). Publish or Perish User’s Manual
  49. 49. • “altmetrics is the creation and study ofnew metrics based on the Social Webfor analyzing, and informingscholarship.” Impact Explorer
  50. 50. ImpactStory• Open Source• Metrics are computed based on the data sourcesaggregated from various social media tools• The first 1000 items you register are free• Covers diverse products including slides, articles,software, dataset, etc• It does not provide comprehensive impact• It is in Beta format• e.g.
  51. 51. 51Usage - Downloads, views, book holdings, ILL, document deliveryCaptures - Favorites, bookmarks, saves, readers, groups, watchersMentions - blog posts, news stories, Wikipedia articles, comments, reviewsSocial media - Tweets, +1s, likes, shares, ratingsCitations - PubMed, Scopus, patents
  52. 52. 52
  53. 53. 53
  54. 54. Google Scholar Researchers Profiles
  55. 55. 55
  56. 56. Open Researcher and Contributor ID56• Open Researcher ID - is an initiativeto provide researchers and scholars with apersistent, unique identifier.
  57. 57. WOS Scopus CWTS/ScopusJournalindicatorPOP JCR EigenFactorSCImagoArticleAnalysisYes Yes YesAuthorAnalysisYes Yes YesJournalRankingYes Yes Yes Yes Yes Yes YesCountryRankingYesWhich metric tool?Adapted form
  58. 58. More info:• Guide• MyRI Bibliometrics Toolkit• Research Trends
  59. 59. QuestionsNeed library related assistance?Please don’t hesitate to contact us:Nazi Torabi519-661-2111 x88992ntorabi@uwo.ca59