Downloaded 10 times
Uploaded byNees Jan van Eck
A scientometric perspective on university ranking
This document discusses responsible use of university rankings. It summarizes a presentation given by Nees Jan van Eck of CWTS about their Leiden university ranking methodology. The presentation outlines principles for responsible ranking design, interpretation, and use. It emphasizes using transparent, field-normalized bibliometric indicators to measure research impact rather than composite scores. Comparisons should consider size and subject differences between universities. Ranks are less important than underlying indicator values. Non-research metrics are also important to consider.
More Related Content
Similar to A scientometric perspective on university ranking
A scientometric perspective on university ranking
- 1. A scientometric perspectiveon university ranking Nees Jan van Eck Centre for Science and Technology Studies (CWTS), Leiden University Wuhan University Wuhan, China, April 15, 2019
- 2. Centre for Scienceand Technology Studies (CWTS) • Research center at Leiden University in the Netherlands • Science and technology studies, with a strong emphasis on scientometrics • About 50 staff members • Our mission: Making the science system better! • Research groups: – Quantitative Science Studies – Science and Evaluation Studies – Science, Technology, and Innovation Studies • Commissioned research for research institutions, funders, governments, companies, etc. 1
- 3. About myself • Masterin computer science • PhD thesis on bibliometric mapping of science • Senior researcher at CWTS – Bibliometric network analysis and visualization – Bibliometric data sources – Bibliometric indicators • Head of ICT 2
- 4.
- 5. Outline • CWTS LeidenRanking • Responsible use of university rankings 4
- 6.
- 7.
- 8. CWTS Leiden Ranking •Provides bibliometric indicators of: – Scientific impact – Scientific collaboration • Calculated based on Clarivate Analytics Web of Science data 7
- 9. Selection of universities(2018 edition) • All universities worldwide with ≥1000 Web of Science publications in period 2013–2016 • 938 universities from 55 countries 8
- 10. Indicators • Size-dependent andsize-independent indicators • Scientific output: – P 9 • Scientific impact: – P(top 1%) and PP(top 1%) – P(top 5%) and PP(top 5%) – P(top 10%) and PP(top 10%) – P(top 50%) and PP(top 50%) – TCS and MCS – TNCS and MNCS • Scientific collaboration: – P(collab) and PP(collab) – P(int collab) and PP(int collab) – P(industry) and PP(industry) – P(<100 km) and PP(<100 km) – P(>5000 km) and PP(>5000 km) PP X = P(X) P
- 11. Differences with otheruniversity rankings • No composite indicators • Focused on research, not on teaching • Based purely on bibliometric indicators; no survey data or data provided by universities • High-quality bibliometric methodology • Multiple views, not just a simple list 10
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19. Advanced bibliometric methodology •Field classification system • Counting citations vs. counting highly cited publications • Full counting vs. fractional counting • Bibliographic database 18
- 20. 19 About 4000 fieldsof science in the Leiden Ranking Social sciences and humanities Biomedical and health sciences Life and earth sciences Mathematics and computer science Physical sciences and engineering
- 21. Why count highlycited publications? • Leiden Ranking counts number of highly cited publications (top 10%) • THE, QS, and US News count number of citations • Effect of counting number of citations: 20
- 22. Why count highlycited publications? 21
- 23. Why count highlycited publications? 22 Counting citations Counting highly cited publications Leaving out Göttingen’s most cited publication
- 24. How to handlepublications co-authored by multiple institutions? • THE, QS, and US News: – Co-authored publications are fully assigned to each co-authoring institution (full counting) • Leiden Ranking: – Co-authored publications are fractionally assigned to each co-authoring institution (fractional counting) 23 This publication is assigned to Leiden with a weight of 2/4
- 25. Why use fractionalcounting? • Full counting is biased in favor of universities with a strong biomedical focus 24
- 26. Choice of bibliographicdatabase: Is more data always better? • Universities from China, Russia, France, Germany, etc. may not benefit at all from having more data • Indicators should be based on a restricted database of publications • Leiden Ranking uses Web of Science, but excludes national scientific journals, trade journals, and popular magazines 25
- 27. Responsible use of universityrankings 26
- 28. Responsible use ofuniversity rankings • Ten principles for responsible use of rankings: – Design of rankings – Interpretation of rankings – Use of rankings • Covers university rankings in general, not only the Leiden Ranking 27 Source: www.cwts.nl/blog?article=n-r2q274
- 29. Responsible use ofuniversity rankings 28 Source: www.researchresearch.com/news/article/?articleId=1368350 Source: https://vimeo.com/279712695
- 30. Design of rankings 1.A generic concept of university performance should not be used 2. A clear distinction should be made between size-dependent and size- independent indicators 3. Universities should be defined in a consistent way 4. University rankings should be sufficiently transparent 29
- 31. Design of rankings 1.A generic concept of university performance should not be used 2. A clear distinction should be made between size-dependent and size- independent indicators 3. Universities should be defined in a consistent way 4. University rankings should be sufficiently transparent 30
- 32. Do not usea generic concept of university performance 31 Composite indicator
- 33. Do not usea generic concept of university performance 32
- 34. Do not usea generic concept of university performance 33
- 35. Design of rankings 1.A generic concept of university performance should not be used 2. A clear distinction should be made between size-dependent and size- independent indicators 3. Universities should be defined in a consistent way 4. University rankings should be sufficiently transparent 34
- 36. Distinguish between size-dependentand size- independent indicators What are the wealthiest countries in the world? 35 GDP per capita GDP
- 37. Distinguish between size-dependentand size- independent indicators • Shanghai, THE, QS, and US News use composite indicators • These composite indicators combine size-dependent and size-independent indicators • It is unclear which concept of scientific performance is measured 36
- 38. Distinguish between size-dependentand size- independent indicators 37
- 39. Design of rankings 1.A generic concept of university performance should not be used 2. A clear distinction should be made between size-dependent and size- independent indicators 3. Universities should be defined in a consistent way 4. University rankings should be sufficiently transparent 38
- 40. Define universities consistently •leiden univ • leiden state univ • state univ leiden • leiden univ hosp • state univ leiden hosp • univ leiden hosp • univ hosp leiden • lumc • univ leiden • leiden univ med ctr • leiden state univ hosp • leiden observ • sterrewacht leiden • acad hosp leiden • rijksuniv leiden • rijksherbarium • gorlaeus labs • leiden inst brain & cognit • leiden inst chem • sylvius labs • acad ziekenhuis leiden • leiden cytol & pathol lab • rijksherbarium hortus bot • ... 39
- 41. Design of rankings 1.A generic concept of university performance should not be used 2. A clear distinction should be made between size-dependent and size- independent indicators 3. Universities should be defined in a consistent way 4. University rankings should be sufficiently transparent 40
- 42. University rankings shouldbe sufficiently transparent 41www.issi-society.org/open-citations-letter/
- 43. Interpretation of rankings 5.Comparisons between universities should be made keeping in mind differences between universities 6. Uncertainty in university rankings should be acknowledged 7. An exclusive focus on ranks of universities should be avoided; values of underlying indicators should be taken into account 42
- 44. Interpretation of rankings 5.Comparisons between universities should be made keeping in mind differences between universities 6. Uncertainty in university rankings should be acknowledged 7. An exclusive focus on ranks of universities should be avoided; values of underlying indicators should be taken into account 43
- 45. Interpretation of rankings 5.Comparisons between universities should be made keeping in mind differences between universities 6. Uncertainty in university rankings should be acknowledged 7. An exclusive focus on ranks of universities should be avoided; values of underlying indicators should be taken into account 44
- 46. Interpretation of rankings 5.Comparisons between universities should be made keeping in mind differences between universities 6. Uncertainty in university rankings should be acknowledged 7. An exclusive focus on ranks of universities should be avoided; values of underlying indicators should be taken into account 45
- 47. Take into accountvalues of indicators 46 Rockefeller University Rank 1 (PPtop 10% = 28.2%) Queen Mary University London Rank 50 (PPtop 10% = 14.8%) University of Bari Aldo Moro Rank 550 (PPtop 10% = 8.0%) Difference in PP(top 10%) is two times larger for universities at ranks 1 and 50 than for universities at ranks 50 and 500
- 48. Use of rankings 8.Dimensions of university performance not covered by university rankings should not be overlooked 9. Performance criteria relevant at university level should not automatically be assumed to have same relevance at department of research group level 10.University rankings should be handled cautiously, but they should not be dismissed as being completely useless 47
- 49. Simplistic use ofrankings 48
- 50. Proper use ofrankings 49
- 51.
- 52. Conclusions • Rankings providevaluable information... • …but only when designed, interpreted, and used in a proper manner • Ranking producers, universities, governments, and news media need to work toward shared principles for responsible university ranking 51
- 53. CWTS Leiden Ranking2019 • Two new types of indicators, in addition to impact and collaboration indicators • Open access indicators (based on Unpaywall data) – P(OA) – P(gold) – P(green) – P(unknown) • Gender indicators – A(male) – A(female) – A(unknown) 52
- 54. Thank you foryour attention! 53