The document summarizes Ludo Waltman's presentation on responsible university ranking at the 1st Moscow International Symposium in Scientometrics, Higher Education and Law Research. It discusses CWTS, Leiden University's work in developing quantitative analyses of science, including the Leiden Ranking of universities and the Leiden Manifesto providing principles for responsible use of metrics. The presentation emphasizes the need for transparent, multi-dimensional rankings that consider different university missions and account for uncertainty. It also introduces the new Quantitative Science Studies journal.

1. Ranking universities responsibly
Ludo Waltman
Centre for Science and Technology Studies (CWTS), Leiden University
1st Moscow International Symposium in Scientometrics, Higher Education and Law Research
RUDN University, Moscow
November 20, 2019

2. Outline
• Centre for Science and Technology Studies (CWTS), Leiden University
• Ranking universities responsibly
• Quantitative Science Studies (QSS) journal
1

3. Centre for Science and
Technology Studies
(CWTS), Leiden
University
2

4. Centre for Science and Technology Studies (CWTS)
• Research center in science and technology studies
• Located at Leiden University in the Netherlands
• Our mission: Making the science system better!
3

5. CWTS Leiden Ranking
4

6. 5

7. Open access publishing and gender balance
6
0% 20% 40% 60% 80% 100%
PP(OA) (2006-2009)
0%
20%
40%
60%
80%
100%
PP(OA)(2014-2017)
0% 10% 20% 30% 40% 50% 60%
PA(F|MF) (2006-2009)
0%
10%
20%
30%
40%
50%
60%
PA(F|MF)(2014-2017)

8. Leiden Manifesto
• Leiden Manifesto provides 10
principles for responsible use of
scientometrics in research
assessments
• Used by research institutions to
develop internal policies for use
of scientometrics
7

9. More diverse journal indicators
• Call for more diverse journal
indicators covering the different
functions of journals
8

10. VOSviewer
9

11. Citation network of journals
10

12. Leiden algorithm
11

13. Landscape of science
12
Social sciences
and humanities
Biomedical and
health sciences
Physical
sciences and
engineering
Mathematics and
computer science
Life and earth
sciences

14. Research on research in the landscape of science
13
Social sciences
and humanities
Biomedical and
health sciences
Mathematics and
computer sciencePhysical
sciences
Life and earth
sciences
scientometrics
science and
technology
studies
innovation
studies
science of
science
higher
education
data sharing
metaresearch
(research quality)
metaresearch
(team science and
research funding)
health
policy
metaresearch
(reproducibility)
metaresearch
(reproducibility)

15. Research on Research Institute (RoRI)
14

16. Ranking universities
responsibly
15

17. Project 5-100
16

18. Ten rules for ranking universities
17
Source: www.researchresearch.com/news/article/?articleId=1368350
Source: https://vimeo.com/279712695

19. Ten rules for ranking universities
1. One size doesn’t fit all
2. Separate the relative from
the absolute
3. Be explicit about the
definition of a university
4. Be transparent
5. Compare and contrast
6. Acknowledge uncertainty
7. Look at the underlying data
8. Not everything that counts
can be counted
9. Know your level
10. Handle with care, but don’t
discard
18

20. One size doesn’t fit all
19

21. Separate the relative from the absolute
What is the world’s wealthiest country?
20
GDP per capita GDP

22. Be transparent
21Weak correlation between citation scores
in Leiden Ranking and Times Higher Education ranking

23. Be explicit about the definition of a university
22

24. Criteria for fair and responsible university ranking
23

25. Quantitative Science
Studies (QSS) journal
24

26. Quantitative Science Studies (QSS)
• Official journal of the International Society for
Scientometrics and Informetrics
• Published by MIT Press
• Launched in early 2019
• Managed by the former editorial board of Journal of Informetrics (JOI)
• Openness and transparency:
– Open access (APC of USD 600/800; covered by a subsidy from TIB in Germany)
– Open data (whenever possible)
– Open metadata (available in Crossref)
– Reporting of competing interests
– APC transparency
– Peer review transparency (pilot project in 2020) 25

27. Initiative for Open Citations (I4OC)
26

28. JOI/QSS flip
27

29. Thank you for your attention!
28