Research Metrics Tools: Measuring Scientific Output

1. Research Metrics: Measuring
Scientific Output
Nazi Torabi
Research & Instructional Librarian
Allyn & Betty Taylor Library
April 25, 2013
http://www.slideshare.net/ntorabi/
1

2. 1. To provide an overview of various
bibliometrics products/tools
2. Have a good understanding of limitation of
each product
3. How to calculate your own h-index
Objectives
2

3. Agenda
Journal Ranking
•Journal Citaiton Report- Impact Factor
•Eigenfactor
•Scopus (SNIP)
•SCImago Journal Rank (SJR)
Article Ranking
•Web of Science
•F1000
•Google: POP
•Scopus
Author Metrics: H-index
•Web of Science
•Scopus
•Google
•and H-index variants
Altmetrics
• Use of social Media to evaluate
scientific impact
Tools for Researchers Profile
• Google Citation Profile
• Researcher ID
• ORCID
3

4. The Metrics Principles
• Publication count VS. Publication quality (impact)
• Citation counts
o Total number of citation count
o Total number of citation count in particular time span
o Total number of citation per paper
o Number of papers cited more than n times
o Number of citations to the n most cited papers
• Normalized citation counts
o Number of authors for each paper
o discipline-normalized score
4

5. Journal Ranking
• Journal Citation Report
– Impact Factor
– journal Immediacy Index
• Eigenfactor
– Eiginfactor Score
– Article Influence Score
• Scopus (SNIP)
• SCImago Journal Rank (SJR)
5
Key Metrics/ unit
Key Metrics/ unit

6. •“IF of a journal is the average number of citations received
per paper published in that journal during the two preceding
years.”Source
•IF provides Journal ranking by discipline
•JCR provides citing pattern analysis in discipline
6

7. Impact Factor
Number of times articles published in 2008 and 2009
were cited by indexed journals during 2010
Total number of "citable items" published by that
journal in 2008 and 2009
2010 Journal IF =
i.e. An Impact Factor of 1.0 in 2013 for a
journal means that, on average, the articles
published in 2011 and 2012 in that journal
have been cited one time.
•Self-citation is included.
•Citable items are usually articles, reviews, proceedings, or notes; not
editorials or Letters-to-the-Editor. 7

8. Journal Immediacy Index
i.e. The journal Immediacy Index indicates
how quickly articles in a journal are cited.
Number of times articles published in 2010 were cited
by indexed journals during 2010
Total number of "citable items" published by that
journal in 2010
2010 Journal
Immediacy =
Index
JCR
8

9. Journal: CHEMBIOCHEM
9
JCR Video on how to find IF

10. For 2011, the journal CHEMBIOCHEM
has an Impact Factor of 3.944.
Category Name
Total Journals
in Category
Journal Rank
in Category
Quartile
in Category
BIOCHEMISTRY &
MOLECULAR
BIOLOGY
290 84 Q2
CHEMISTRY,
MEDICINAL
59 6 Q1
10

11. IF & its shortcomings
• Citation pattern is discipline specific
– Time span
– Citation frequency
– differences in citation norms
• Sleeping beauties
• Review Journals
• Self-Citation!
• Negative citations
• Title and journal format changes
• A questionable validity
11

12. Probability density function for journals in subject
categories of ENERGY & FUELS – data from JCR
-5 -4 -3 -2 -1 0 1 2 3 4 5
Normal distribution
Pareto distribution
12
IF is based on the
arithmetic mean
number of citations
per paperThe reality of
citation patterns is
a Pareto
distribution

13. Articles on validity of IF
• Dissecting our impact factor. Nature Materials
10, 645 (2011) doi:10.1038/nmat3114
• Bollen, Johan, et al. "A principal component
analysis of 39 scientific impact measures."
PloS one 4.6 (2009): e6022.
• Priem, Jason, and Bradely H. Hemminger.
"Scientometrics 2.0: New metrics of scholarly
impact on the social Web." First Monday 15.7
(2010).
13

14. Eigenfactor.org
14

15. Citation Networks & Network Effects
15

16. Scholarship is the
flow of idea.
16

17. High School Dating
Network
Bearman, Peter S., James Moody, and Katherine Stovel.
"Chains of Affection: The Structure of Adolescent
Romantic and Sexual Networks." American Journal of
Sociology 110, no. 1 (July 2004): 44-91.
17

18. Which network you would rather be
part of?
West, J (2010). The EigenfactorTM Metrics: a network approach to
assessing scholarly journals.
http://mediazone.brighttalk.com/comm/ReedElsevier/ad0fb0ef1d-
21688-4348-23066
18
A B

19. The Rich gets Richer
•Journals are ranked according to the citation network.
•Journals are influential when they are cited by other influential
journals.
•“The Eigenfactor™ score of a journal is an estimate of the
percentage of time that researchers spend with that journal.”
19

20. http://well-formed.eigenfactor.org/radial.html
20

21. Article Influence™ Score
Journal’s Eigenfactor Score (5yrs)
Number of articles in the journal
Article Influence™ =
Score
21

22. IF vs. Eigenfactor
22

23. e.g. of Article Influence™ Score
23
http://www.eigenfactor.org/

24. More on Journal Ranking
SJR
• SCImago Journal Rank is
weighted by the prestige of
a journal. Subject field,
quality and reputation of
the journal have a direct
effect on the value of a
citation.
• SJR normalizes for
differences in citation
behavior between subject
fields.
SNIP
• Source Normalized Impact
per Paper measures
contextual citation impact
by weighting citations based
on the total number of
citations in a subject field.
• It corrects for differences in
the frequency of citation
across research fields.
24
4 years of data is required to calculate SJR and SNIP
http://www.youtube.com/watch?v=ZQQIYBBMiUg&list=UUtm20B-NV4ShpVGj_5A8ukQ&index=8
Scopus

25. 25

26. Journal Ranking Summary
IF Eigenfactor®
Score
SNIP SJR indicator
Sources Database JCR Eigenfactor Scopus/CWTS Scopus/SCImago
Citation time
frame
2 years 5 years 4 years 4 years
Journal self-
citation
Included Excluded Limited Limited
Citation Value Unweighted Weighted Weighted Weighted
Connections NA Normalized by the
number of identified
references in the
citing
Journal
Corrects for
differences in the
frequency of
citation across
research fields
Normalized by
the total
number of
references in
the citing journal
Adapted and modified from: González-Pereir , B. et al. “The SJR indicator: A new indicator of
journals' scientific prestige”. http://arxiv.org/ftp/arxiv/papers/0912/0912.4141.pdf 26

27. Articles Ranking
27

28. •Key metric: F1000 Article Factors (FFa)
•Medicine and Biology specific
28

29. F1000 factors
The F1000 Article Factor (FFa) is calculated from the highest rating
awarded by a Faculty Member (FM) plus an increment for each
additional rating from other FMs.
For example, a single article that has been evaluated by three Faculty
Members, who rated it ‘Recommended’, ‘Must Read’ and ‘Must Read’,
will have its FFa calculated thus:
8 (highest rating ‘Must Read’) + 2 (increment for ‘Must Read’) + 1 (increment for ‘Recommended’) = 11.
Rating Value Increment
Exceptional 10 3
Must Read 8 2
Recommended 6 1
29

30. 30

31. 31

32. Citation overview from Scopus
• The Citation Overview Results page lists
documents that cited the document you selected
from the Citation Overview.
• You can include up to 5000 documents in a
Citation Tracker. If you select more than 5000
documents only the first 5000 are included.
• More
http://trainingdesk.elsevier.com/videos/scopus-
analyze-results-tool
32

33. 33

34. 34

35. 35

36. Harzing's Publish or Perish
Publish or Perish is a software program that retrieves and
analyzes academic citations. It uses Google Scholar to
obtain the raw citations, then analyzes these and
presents the following statistics:
o Total number of papers
o Total number of citations
o Average number of citations per paper
o Average number of citations per author
o Average number of papers per author
o Average number of citations per year
o h-index and related parameters
o An analysis of the number of authors per paper.
36Download and install Publish or Perish at http://www.harzing.com/pop.htm#download

37. 37

38. Google Scholar vs. WOS dataset
• Transparency
• Coverage
o Better coverage for unique citations
o Not limited to journal articles published in ISI listed
journals
o Greater geographical coverage for non-US/UK based
journals
o Better coverage of non-English sources
• Includes more comprehensive data for junior
academics
38

39. Google Scholar vs. WOS dataset cont’
• Accounts for minor citation errors
• Does not account for duplications
• Includes non-scholarly citations such as
student handbooks, library guides or editorial
notes.
• Unclear coverage
o uneven coverage across different fields of study
o older materials are not well covered
39

40. A researcher’s H-index is the largest possible number n
for which n of the researcher’s publications have been
cited at least n times.
i.e. If a researcher has an h-index of 25 if 25 of their
papers have been cited at least 25 times (See the graph
in the handout).
40

41. h-Index example
Researcher A Researcher B Researcher C Researcher D
Citations for most-cited
article
3 5 14 8
Citations for second most-
cited article
3 5 1 3
Citations for third most-cited
article
3 5 0 2
Citations for fourth most-
cited article
3 0 0 1
Citations for fifth most-cited
article
3 0 0 1
Average citations per paper 3 3 3 3
H-index 3 3 1 2
41
Jarvey, P. (2012). Making Research Count: Analyzing Canadian Academic Publishing Cultures

42. Citation Report from WOS
The Citation Report provides aggregate citation
statistics for a set of search results. These
statistics include:
o The total number of times all items have been cited
o The average number of times an item has been cited
o The number of times an item has been cited each year
o The average number of times an item has been cited
in a year
o More info
42

43. 43

44. Finding H-Index
• Scopus
• Google
44

45. Why h-Index
• It considers both productivity and impact.
• It is not influenced by a small number of very
successful articles, which may not be
representative of a researcher’s career output.
The H-index similarly discounts the value of
papers that are not influential.
• The H-index is simpler and easier to
understand than many other compound
bibliometric metrics.
45

46. h-index shortcomings
• Early career researchers
• Number of authors
• Order of authorship
• Sleeping beauties
• the time since publication of individual articles
• Databases are not error free
• Inconsistency between different metric tools (See
the handout).
• The balance between number of publications vs.
number of citations each receives
46
Adapted from Jarvey, P. (2012). Making Research Count: Analyzing Canadian Academic Publishing Cultures

47. number of publications vs. number of
citations each receives
47
Researcher A : 5 articles published – each cited 5 times
Researcher B : 2 articles published – each cited 20 times
Researcher A (5, 5, 5, 5, 5 ) = h-index = 5
Researcher B (20, 20) = h-index = 20

48. Selected h-index Variants
• g-index: more weight is given to the highest cited
papers
• individual h-index: accounts for co-authorship in
calculating impact by giving less weight to such papers
• contemporary h-index: less weight to older cited
papers
• i10 index : from Google Scholars, indicates the number
of papers which have at least ten citations
• age-weighted citation rate: accounts for the age of
papers
Tarma Software Research (2010). Publish or Perish User’s Manual
http://www.harzing.com/pophelp/metrics.htm#gindex
48

49. • “altmetrics is the creation and study of
new metrics based on the Social Web
for analyzing, and informing
scholarship.”
http://altmetrics.org/manifesto/
PLoS Impact Explorer

50. ImpactStory
• Open Source
• Metrics are computed based on the data sources
aggregated from various social media tools
• The first 1000 items you register are free
• Covers diverse products including slides, articles,
software, dataset, etc
• It does not provide comprehensive impact
• It is in Beta format
• e.g. http://impactstory.org/collection/fdtpk4

51. 51
Usage - Downloads, views, book holdings, ILL, document delivery
Captures - Favorites, bookmarks, saves, readers, groups, watchers
Mentions - blog posts, news stories, Wikipedia articles, comments, reviews
Social media - Tweets, +1's, likes, shares, ratings
Citations - PubMed, Scopus, patents

52. 52

53. 53

54. Google Scholar Researchers Profiles
http://www.google.ca/intl/en/scholar/citations.html

55. 55

56. Open Researcher and Contributor ID
56
• Open Researcher ID - is an initiative
to provide researchers and scholars with a
persistent, unique identifier.
http://about.orcid.org/
http://trainingdesk.elsevier.com/videos/scopus-
creating-an-orcid-record-from-your-scopus-
author-id

57. WOS Scopus CWTS/Scopus
Journal
indicator
POP JCR Eigen
Factor
SCImago
Article
Analysis
Yes Yes Yes
Author
Analysis
Yes Yes Yes
Journal
Ranking
Yes Yes Yes Yes Yes Yes Yes
Country
Ranking
Yes
Which metric tool?
Adapted form https://dspace.ndlr.ie/jspui/handle/10633/27390
57

58. More info:
• Guide
http://researchguides.library.syr.edu/content.php?p
id=194034&sid=3337810
• MyRI Bibliometrics Toolkit
http://www.youtube.com/user/NDLRMyRI/videos
http://www.ndlr.ie/myri/
• Research Trends
http://www.researchtrends.com/?s=bibliometric&x
=6&y=7&cat=-1
58

59. Questions
Need library related assistance?
Please don’t hesitate to contact us:
Nazi Torabi
519-661-2111 x88992
ntorabi@uwo.ca
59
http://www.lib.uwo.ca/chat/