ULS/ISchool workshop delivered at Hillman Library, University of Pittsburgh on 11 November 2016

1. WHAT IS YOUR H-INDEX AND
OTHER MEASURES OF
ACADEMIC IMPACT
Berenika M. Webster, PhD
ULS/ISchool workshop
11 November 216
1

2. Metrics are everywhere
2

3. They are used to measure/demonstrate
impact of…
• National science systems
• Research-producing institutions
• Research groups
• Individual researchers
In order to…
• Understand impact of investment
• Award funding
• Employment and promotion decisions
• Apply for funding, job, promotion
• Identify experts, collaborators, etc.
3

4. Learning outcomes
• At the end of sessions participants will
• be able to identify paper-level and author-level indicators of impact
• be able to extract these indicators from Scopus, WoS and SciVal
• Understand how individual researchers may use these indicators to
demonstrate impact
4

5. “Citizen” bibliometrics…
• Productivity (publications counts)
• leads to “salami slicing”, maybe
• quantity vs quality
• Impact (citation counts)
• but what impact?
• Impact factor
• Speaks to prestige of outlet, not quality of individual paper
• 20% of papers in Nature get 80% of all citations (averages do not
work when distributions are skewed)
• h-index
• arbitrary, simplistic and lacks consistency
• it’s always highest in Google Scholar
5

6. (BIBLIOMETRIC) STORY
OF ONE PUBLICATION
aka article-level metrics
6

7. Story of one publication
7
Kato, M., Han, TW., Xie, S., Shi, K., Du, X., Wu, LC., Mirzaei, H.,
Goldsmith, EJ., Longgood, J., Pei,J., and Grishin, N.V. and Frantz, DE.,
Schneider, JW., Chen, S., Li, L., Sawaya, MR., Eisenberg, D, Tycko, R.
and S. McKnight. (2012) “Cell-free Formation of RNA Granules: Low
Complexity Sequence Domains Form Dynamic Fibers within
Hydrogels.”, CELL, 149 (4): 753-67. (citations 325, IF 34.242)

8. Story of one publication
• Journal characteristics
• JIF or SNIP or another indicator of impact in context
• Acceptance rates (e.g. from Cabell's Directory of Publishing Opportunities)
• Co-author characteristics
• Names, institutions, countries
• Citations
• Number of citations in context (normalised baselines)
• Ratio
• Percentile distribution
• Characteristics of citing papers
• authors, institutions, countries, subject fields, journals
• Altmetrics
• Predictor of citation impact (???)
• Indicator of attention
• Indicator of impact outside academic realm (e.g. citations in policy documents or clinical
guidelines)
8

9. Story of one publication: WoS
9

10. Story of one publication: Scopus
10

11. Story of one publication: altmetrics
11

12. Story of one publication: more Altmetrics
12

13. Baselines
• Web of Science (WoS) (at Pitt publication and citation data going back to 1980)
• Highly Cited?; Hot?, Impact Factor value
• Essential Science Indicators (ESI) (publication and citation data going back 10 years +
current year)
• across 22 broad disciplines
• Baselines (Averages and Percentiles)
• Journal Citation Reports (JCR)
• IF values of thousands of journals and much more (citation networks, rates of
obsolescence, etc.)
• Scopus (complete citation data going back to 1996, project to extend to 1970 under
way to be completed by end of 2016)
• discipline adjusted citation rates (more granular disciplinary and sub-disciplinary
divisions), also adjusted for all doc. types
• SciVal (complete publication and citation data going back to 1996. Based on
SCOPUS-indexed publications)
• Over 20 indicators of productivity and impact
13

14. Story of one publication
14
My paper has been published in Cell, 2nd top ranked journal in the field of
biochemistry and mol. biology and 3nd top title in the filed of cell biology (JCR,
2015 ed.)
To date it was cited 325 times, which places it in the top 1% of all world’s 2012
biochem. and mol. biology papers. The paper has an international and
multidisciplinary reach. Citations come from authors from 34 different countries
and from across 95 different journals in 35 different fields (including 7 citations
from Nature and 6 from Science). (WoS, 20 Sept. 2016).
My paper also attracted attention from non-scholarly audiences, …
Kato, M., Han, TW., Xie, S., Shi, K., Du, X., Wu, LC., Mirzaei, H., Goldsmith, EJ.,
Longgood, J., Pei,J., and Grishin, N.V. and Frantz, DE., Schneider, JW., Chen, S., Li, L.,
Sawaya, MR., Eisenberg, D, Tycko, R. and S. McKnight. (2012) “Cell-free Formation of
RNA Granules: Low Complexity Sequence Domains Form Dynamic Fibers within
Hydrogels.”, CELL, 149 (4): 753-67. (citations 325, IF 34.242)

15. Exercise
• Murphy, S.V. and A. Atala (2014) “3D bioprinting of tissues
and organs” NATURE BIOTECHNOLOGY; 32(8):773-785.
DOI: 10.1038/nbt.2958
• Describe it using bibliometric indicators discussed today
• Citation count
• Normalised citation count
• Percentile position
• Impact of publishing journal
• Characteristics of citing publications
• Use WoS or SCOPUS and Altmetric.com
15

16. Web of Science and Essential
Science Indicators
• Murphy, S.V. and A. Atala (2014) “3D bioprinting of
tissues and organs” NATURE BIOTECHNOLOGY;
32(8):773-785.
• 337 citations
• 337/5.31 = 63.45 (normalised citation impact)
• In top 1% of all 2014 Biology and Biochemistry
publications (HICI paper)
16

17. 17

18. 18

19. (BIBLIOMETRIC) STORY
OF ONE AUTHOR
aka aruthor-level metrics
19

20. Story of one author
• Source of information about publications
• CV
• Pitt Faculty Info System (Elements)
• ORCID
• Google Scholar profile
• How much output is captured in databases
• Scopus Author Identifier
• ResearcherID (WoS)
20

21. Scholarly communication practices
21
Subject area Books and book chapters Conference papers Journal articles
History 45.6 3.8 50.6
Politics and Policy 43.1 10.8 46.1
Language 40.5 7.6 51.8
Human Society 31.3 5.6 63
Philosophy 29.8 5.4 64.8
Economics 27.4 8 64.5
Law 26.2 1.9 71.9
The Arts 25.2 20.3 54.5
Education 21.8 23.6 54.5
Architecture 20.8 43.6 35.6
Psychology 18.9 4.9 76.2
Journalism, library 18.6 24.2 57.2
Management 13 34 52.9
Earth Sciences 8.6 9.2 82.2
Medical & Health Sci 6.6 2.9 90.5
Biological Sciences 6.6 2.7 90.7
Agriculture 6.3 14.7 79
Computing 5 62.3 32.8
Mathematical Sciences 5 11.2 83.8
Engineering 2.9 45.1 52
Physical Sciences 2.7 7.3 90
Chemical Sciences 2.3 1.9 95.7
L. Butler, 2006
And what about
all other
research
outputs?

22. Author metrics
• Overview of productivity and impact
• Should use size-dependent variables as these take into
consideration volume of outputs
• Publications (other outputs) counts and characteristics
• Total citation counts in context (discipline, age, output type)
• Excellence measures (top 1, 5, 10% of distributions)
• ESI’s Highly Cited author?
• h-index (if you must) – though it is inconsistent (example later)
22

23. Story of one author
23

24. SCOPUS view
24
Citation per publication rate for my 200 publications
indexed in Scopus is 91.2, only 5 of my publications (2.5%)
were not cited o date. (Scopus, 8 June 2016)
My research is multidisciplinary spanning across
biochemistry, genetics and molecular biology,
physics chemistry, materials science and
engineering.
I have been publishing consistently since 1982,
averaging 7 publications per year in high quality, peer
reviewed outlets. These publications, collectively receive
upwards of 1,000 citations per year. (Scopus, 8 June 2016)

25. Story of one author
25
Citations to my publications in the last 10 years placed
me in the group of top 1% of biochemistry researchers
in the world and 5 of these publications placed in the
top 1% of publications in their field. (ESI, 20 Sept
2016)

26. SciVal view - overview
26
In the last 5 years, 68% of my
publications placed in the top
10% of all similar publications
(by field and age), and 32% of
them were published in the
top 10% of scientific journals.
(SciVal, 8 June 2016)

27. My publications in biochemistry consistency outperform these of NIH and USA as a whole in
relation of impact and percentage of publications in top 1% in the world. (SciVal, 8 June 2016)
27

28. Whose h-index? One author – 3 different
values
28
137 vs. 143 vs. 160

29. Author 1 Author 2 Author
3
15 150 15
10 100 10
10 50 10
5 25 5
5 5 5
4 1 0
4 0 0
3 0 0
3 0 0
1 0 0
Does not account
for:
•insensitive to highly-cited
publications
•citation characteristics of
publication outlets
•citation characteristics of fields
of science
•age of publications
•type of publications
•co-authorship
•self-citations
•scientific age of author
29

30. More problems with h-index
30
Ludo Waltman and Nees Jan van Eck, JASIST 2012
If two scientists achieve the same absolute performance improvement, their
ranking relative to each other should remain unchanged.
Scientist X Scientist Y
Publication Citations
1 5 6
2 5 6
3 5 6
4 5 6
5 5 3
6 2 3
7 2 3
Scientist X Scientist Y
Publication Citations
1 8 8
2 8 8
3 5 6
4 5 6
5 5 6
6 5 6
7 5 3
8 2 3
9 2 3
Scientist X h-index 5
Scientist Y 4
Scientist X – 5
Scientist Y - 6

31. Exercise: create a bibliometric profile of a
researcher
• Ludo Waltman, Leiden U
• Krzysztof Matyjaszewski, CMU
• Ervin Sejdic, Pitt
• Anne-Wil Harzing, U Melbourne
• Try SCOUPS, WOS and SciVal to find out
• Number of publications
• Total number of citations
• Normalised citations (e.g. within a discipline)
• Subject distributions
• Characteristics/impact of citing publications
• For an entire career span/for last 5 years
31