Presentation at the ACA European Policy Seminar “12 years with global university rankings”. Brussels, October 15, 2015.

1. Research-only rankings of HEIs:
Is it possible to measure scientific
performance?
Ludo Waltman
Centre for Science and Technology Studies (CWTS), Leiden University
ACA European Policy Seminar “12 years with global university rankings”
Brussels, October 15, 2015

2. University rankings
1

3. Outline
• What do we mean by scientific performance?
• Measuring scientific performance in the CWTS Leiden
Ranking
• Is it really possible to measure scientific performance?
2

4. Indicators of scientific performance
• Publications:
– Total
– Per faculty
– Per student
– Interdisciplinary
– International collaboration
– Nature and Science
• Citations:
– Total
– Per publication
– Per faculty
– Highly cited researchers
• Reputation survey
• Others:
– Nobel Prizes/Field Medals
– PhDs awarded
– PhDs awarded per faculty
– Post-doc positions
– Research income
3

5. What do we mean by scientific
performance?
4
Size-dependent concept of scientific performance:
• Overall contribution of a university to science
• Total number of ‘performance points’ (e.g., publications,
citations, expert recommendations, awards)
Size-independent concept of scientific performance:
• Contribution of a university to science relative to
available resources
• Number of ‘performance points’ divided by available
resources (e.g., number of faculty, research budget)

6. Indicators of scientific performance
• Publications:
– Total
– Per faculty
– Per student
– Interdisciplinary
– International collaboration
– Nature and Science
• Citations:
– Total
– Per publication
– Per faculty
– Highly cited researchers
• Reputation survey
• Others:
– Nobel Prizes/Field Medals
– PhDs awarded
– PhDs awarded per faculty
– Post-doc positions
– Research income
6
Size-dependent indicators Size-independent indicators
Rankings based
on composite
indicators

7. Mixing up different concepts of scientific
performance
• Shanghai, THE, QS, and US News use composite
indicators
• These composite indicators combine size-dependent
and size-independent indicators
7
It is unclear which concept of scientific performance is
measured by Shanghai, THE, QS, and US News

8. CWTS Leiden Ranking
• Focused completely on measuring scientific performance
• Purely based on bibliometric indicators
• No composite indicators
• Separate indicators of size-dependent and size-
independent scientific performance
8

9. 9

10. Main indicators
• Size-dependent:
– P: Number of publications of a university
– P(top 10%): Number of publications belonging to the top 10% most cited
of their field
• Size-independent:
– PP(top 10%): Proportion of publications belonging to the top 10% most
cited of their field
10
PP top 10% =
P(top 10%)
P

11. Advanced bibliometric methodology
• Field classification system
• Counting citations vs. counting highly cited publications
• Full counting vs. fractional counting
• Bibliographic database
11

12. About 4000 fields of science in the
Leiden Ranking
12
Social sciences
and humanities
Biomedical and
health sciences
Life and earth
sciences
Physical
sciences and
engineering
Mathematics and
computer science

13. Why count highly cited publications?
• Leiden Ranking counts number of highly cited
publications (top 10%)
• THE, QS, and US News count number of citations
• Effect of counting number of citations:
13

14. Why count highly cited publications?
14

15. Why count highly cited publications?
15
Counting citations Counting highly cited publications
Leaving out Göttingen’s
most cited publication

16. How to handle publications co-authored
by multiple institutions?
• THE, QS, and US News:
– Co-authored publications are fully assigned to each co-authoring
institution (full counting)
• Leiden Ranking:
– Co-authored publications are fractionally assigned to each co-authoring
institution (fractional counting)
16
This publication is
assigned to
Enschede, Twente,
and Leiden with a
weight of 1/3 each

17. Why use fractional counting?
17
Full counting is biased in favor of universities with a
strong biomedical focus

18. Choice of bibliographic database:
Is more data always better?
18
Database 1:
• Restricted to international scientific journals
• University A: P = 2000; P(top 10%) = 200; PP(top 10%) = 10%
• University B: P = 1000; P(top 10%) = 100; PP(top 10%) = 10%
Database 2:
• Also includes a lot of national scientific journals, trade journals, popular
magazines, etc.
• University A: P = 2000; P(top 10%) = 220; PP(top 10%) = 11%
• University B: P = 1500; P(top 10%) = 135; PP(top 10%) = 9%
US university Chinese university

19. Choice of bibliographic database:
Is more data always better?
• Universities from China, Russia, France, Germany, etc.
may not benefit at all from having more data
• Indicators should be based on a restricted database of
publications
19
Leiden Ranking uses Web of Science, but excludes
national scientific journals, trade journals, and popular
magazines

20. How much difference does it make?
Comparing LR and THE citation scores
• Weak correlation between size-independent citation
scores in Leiden Ranking and THE
• Leiden Ranking score of 10% corresponds with THE
scores between 30 and 85 20

21. Fundamental problem of size-
independent bibliometric indicators
• Same resources as Univ. B
• P = 1000
• P(top 10%) = 200
• PP(top 10%) = 20%
• Same resources as Univ. A
• P = 2000
• P(top 10%) = 300
• PP(top 10%) = 15%
21
Univ. A Univ. B
• Taking into account that both universities have the
same resources, it is clear that university B has
performed better
• However, according to the PP(top 10%) indicator,
university A has performed better

22. Conclusions
• Is it really possible to measure scientific performance?
– Size-dependent concept of scientific performance:
• Reasonable bibliometric measurements are possible
– Size-independent concept of scientific performance:
• Purely bibliometric measurements are problematic
• Do not combine size-dependent and size-independent
indicators of scientific performance
• Bibliometric indicators should:
– Be normalized using a sufficiently large number of fields
– Count the number of highly cited publications, not the number of citations
– Use fractional counting, not full counting
– Be based on a restricted database of publications
22

23. Thank you!
23