This document summarizes a presentation on scientometric approaches to classification. It discusses: - Bibliographic databases like Web of Science and Scopus and their coverage. - Types of classification systems for scientific literature including mono-disciplinary vs multidisciplinary and journal-level vs publication-level classifications. - The CWTS publication-level classification system which uses a fully algorithmic approach to cluster over 21 million publications into a hierarchical structure of disciplines, fields, and subfields. - Applications of the CWTS classification system including field normalization, field delineation, research strength analysis, and identification of interdisciplinary areas. - Studies that have evaluated aspects of the quality and accuracy of classification systems.

1. Scientometric approaches to classification
Nees Jan van Eck
Centre for Science and Technology Studies (CWTS), Leiden University
Colloquium Research Information Systems and Science Classifications: Revisiting the NARCIS Classification
Museum Meermanno, The Hague, The Netherlands
September 28, 2018

2. Outline
• Bibliographic databases
• Classification systems of scientific literature
• CWTS publication-level classification system of science
– Methodology
– Structure
– Applications
• Quality of classification systems
1

3. Bibliographic
databases
2

4. Bibliographic databases
3

5. Bibliographic databases
4
Web of Science Scopus
Journals 20,000 24,000
Publications 55 million 45 million
Citations 1.2 billion 1.2 billion

6. Classification systems
of scientific literature
5

7. Classification systems of scientific literature
• Mono-disciplinary vs. multidisciplinary
• Journal-level vs. publication-level
• Manual vs. algorithmic
6

8. Classification systems of scientific literature
• Mono-disciplinary:
– Chemical Abstracts: 80 different sections and 5 broad headings
– EconLit: Journal of Economic Literature (JEL) classification system
– PubMed: Medical Subject Headings (MeSH)
• Multidisciplinary:
– Web of Science: 250 categories
– Scopus (ASJC): bottom level has 304 categories and top level includes 27 categories
– Science-Metrix: 176 categories
– National Science Foundation (NSF): 125 categories
– University of California, San Diego (UCSD): more than 500 categories
– Australian and New Zealand Standard Research Classification (FoR): 3 hierarchical levels
7

9. CWTS publication-
level classification
system of science
8

10. Algorithmic classification system of science
• First version created in 2012
• Publications (not journals) are clustered into research areas based on citation
relations
• Research areas are defined at different levels of granularity and are
organized hierarchically
• Clustering is performed using the smart local moving algorithm (improved
Louvain algorithm; Waltman & Van Eck, 2013)
9

11. Objectives
To create a classification system
• in a fully algorithmic manner
• covering all sciences and social sciences
• at the level of individual publications
• with a hierarchical structure
• using transparent, freely available algorithms
• without excessive computational requirements
10

12. Main challenges
• Dealing with huge volumes of data
• Avoiding disciplinary biases
• Reaching a high level of accuracy
• Being flexible in terms of number of hierarchical levels and size of research
areas
• Obtaining proper labels for the research areas
• Keeping the methodology reasonably simple and transparent
11

13. Dealing with huge volumes of data
• Linking publications based on direct citations only; no co-citations,
bibliographic coupling, or word co-occurrences
• Efficient clustering algorithm based on ideas taken from:
– Newman (2004): Modularity-based clustering
– Blondel et al. (2008): ‘Louvain method’
– Waltman et al. (2010): VOS clustering technique
– Rotta & Noack (2011): Multilevel local search algorithms
12

14. Avoiding disciplinary biases
• cij: Relatedness of publications i and j, i.e., 1 if there is a direct citation
relation between i and j, 0 otherwise
• aij: Normalized relatedness of publications i and j, defined as
• Similar to fractional citation counting (Small & Sweeney, 1985)


k ik
ij
ij
c
c
a
13

15. Reaching a high level of accuracy
• Clustering technique based on maximization of a quality function:
• xi denotes the cluster (research area) to which publication i is assigned
• (xi, xj) = 1 if xi = xj and 0 otherwise
• r denotes a resolution parameter
• Quality function is maximized with respect to x1, ..., xn
 
i j
ijji raxx ))(,(
14

16. Being flexible in terms of number of hierarchical levels
and size of research areas
• Three types of parameters:
– Number of hierarchical levels
– Each level’s resolution parameter
– Each level’s minimum number of publications per research area
15

17. Obtaining proper labels for the research areas
1. Identification of terms in titles and abstracts of articles using part-of-speech
tagging
2. Calculation of term relevance scores based on a combination of a term’s
absolute and relative frequency of occurrence
3. Selection of the most relevant terms based on term relevance scores
combined with a filter for removing similar terms
16

18. CWTS publication-level classification system of
science
• 21.2 million publications from the period 2000–2017 indexed in Web of
Science
• 374.1 million citation relations
• Classification system of 3 hierarchical levels:
– 22 broad disciplines
– 868 fields
– 4,047 subfields
• Computational performance: less than 2 hours
17

19. 18
Breakdown of scientific literature into 22 broad
disciplines
Social sciences
and humanities
Biomedical and
health sciences
Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering

20. 22 broad disciplines
19

21. 20
Breakdown of scientific literature into 868 fields
Social sciences
and humanities
Biomedical and
health sciences
Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering

22. 21
Breakdown of scientific literature into 4,047 subfields
Social sciences
and humanities
Biomedical and
health sciences Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering

23. 22
Breakdown of scientific literature into 4,047 subfields
Social sciences
and humanities
Biomedical and
health sciences Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering
Scientometrics

24. Summary of scientometrics subfield
23
Cluster: 145
No. publications: 16,312
Top 5 terms No. pubs
bibliometric analysis 852
impact factor 495
h index 264
peer review 515
citation 642
Top 5 publications No. cits
hirsch, je (2005). an index to quantify an individual's scientific research output. p natl acad sci usa, 102(46), 16569-16572. 2,635
wuchty, s; et al. (2007). the increasing dominance of teams in production of knowledge. science, 316(5827), 1036-1039. 699
egghe, l (2006). theory and practise of the g-index. scientometrics, 69(1), 131-152. 609
king, da (2004). the scientific impact of nations. nature, 430(6997), 311-316. 496
newman, mej (2004). coauthorship networks and patterns of scientific collaboration. p natl acad sci usa, 101, 5200-5205. 488
Top 5 authors No. pubs Top 5 journals No. pubs
bornmann, l 221 scientometrics 2,865
thelwall, m 202 journal of informetrics 700
leydesdorff, l 175 journal of the american society for information science and technology 613
rousseau, r 161 plos one 339
egghe, l 133 research evaluation 324
Top 5 institutes No. pubs Top 5 departments No. pubs
univ granada 316 sch lib & informat sci (indiana univ) 106
kathol univ leuven 256 amsterdam sch commun res ascor (univ amsterdam) 97
leiden univ 249 ctr sci & technol studies (leiden univ) 90
indiana univ 246 sch publ policy (georgia inst technol - atlanta) 88
univ wolverhampton 216 trend res ctr (asia univ) 84
0
200
400
600
800
1,000
1,200
1,400
1,600
2000 2002 2004 2006 2008 2010 2012 2014 2016
No.publications

25. Publications in scientometrics subfield
24

26. 25
Term map of scientometrics subfield
Peer review,
OA, careers,
and gender
CollaborationScientometric
indicators and
networks
Medical research
Country-level
analyses

27. 26
Time-line map of highly cited scientometrics
publications

28. 27
Overlay visualizations
Social sciences
and humanities
Biomedical and
health sciences Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering

29. Time trend
28
Social sciences
and humanities
Biomedical and
health sciences Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering

30. Time trend
29
MicroRNA Graphene

31. Summary of graphene subfield
30
Cluster: 9
No. publications: 27,771
Top 5 terms No. pubs
bilayer graphene 836
epitaxial graphene 491
silicene 401
graphene nanoribbon 1,035
graphene field effect transistor 207
Top 5 publications No. cits
novoselov, ks; et al. (2004). electric field effect in atomically thin carbon films. science, 306(5696), 666-669. 27,743
geim, ak; et al. (2007). the rise of graphene. nat mater, 6(3), 183-191. 20,073
novoselov, ks; et al. (2005). two-dimensional gas of massless dirac fermions in graphene. nature, 438(7065), 197-200. 11,359
castro neto, ah; et al. (2009). the electronic properties of graphene. rev mod phys, 81(1), 109-162. 11,368
zhang, yb; et al. (2005). experimental observation of the quantum hall effect and berry's phase in graphene. nature, 438(7065), 201-204. 8,110
Top 5 authors No. pubs Top 5 journals No. pubs
watanabe, k 249 physical review b 4,013
taniguchi, t 240 applied physics letters 1,834
peeters, fm 233 carbon 994
lin, mf 178 nano letters 906
katsnelson, mi 177 journal of applied physics 841
Top 5 institutes No. pubs Top 5 departments No. pubs
chinese acad sci 1,394 dept phys (natl univ singapore) 257
russian acad sci 778 inst phys (chinese acad sci) 226
peking univ 557 inst mol & mat (radboud univ nijmegen) 216
natl univ singapore 482 dept phys (mit) 209
tsing hua univ 458 dept phys (univ calif berkeley and berkeley national lab) 206
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
2000 2002 2004 2006 2008 2010 2012 2014 2016
No.publications

32. Open access
31
Social sciences
and humanities
Biomedical and
health sciences Life and earth
sciences
Mathematics and
computer science
Physical
sciences and
engineering

33. University profiles
32
Delft University of TechnologyLeiden University

34. Applications
• Field normalization
– CWTS Leiden Ranking/U-Multirank
– Dutch University Medical Centers
• Field delineation
– European research funders
• High-resolution research strengths analysis
– European universities
– European research funders
• Identification of interdisciplinary and emerging research areas
– UK Engineering and Physical Sciences Research Council
33

35. Adopters and potential adopters
• Adopters:
– CWTS
– SciTech Strategies (e.g. SciVal)
– Royal School of Technology (KTH) Stockholm
• Potential adopters:
– Chinese Academy of Sciences
– European Research Council
– Max Planck
34

36. Quality of
classification systems
35

37. Empirical micro study using papers on overall water
splitting
• Haunschild et al. (2018)
• Case study comparing CWTS classification to
journal-based and manually constructed
classifications
• Ability of CWTS classification to distinguish
between fields is questioned
36

38. Accuracy of the journal classification systems of Web
of Science and Scopus
• Wang and Waltman (2016)
• Two criteria to identify journals with questionable
classifications:
– journals that have weak connections with their assigned
categories
– journals that are not assigned to categories with which they
have strong connections
• Web of Science performs significantly better than
Scopus
37

39. Field classification of publications in Dimensions
• Bornmann (2018)
• Field classification in Dimensions:
– Based on Fields of Research (FOR) from Australian and New
Zealand Standard Research Classification (ANZSRC)
– Machine learning approach
– Each publication is assigned to at least one field
• Based on Bornmann’s own publications
• Questions reliability and validity of Dimensions
classification
38

40. Response from Dimensions
• Herzog and Lunn (2018)
• Implementation at launch was first step and
requires improvements:
– Improvement of training sets
– Adding new subcategories to FOR system
39

41. Large-scale system to organize publications into
hierarchical concept structure
• Shen et al. (2018)
• Core component in Microsoft Academic
• Iterative approach to:
– concept discovery (Wikipedia)
– concept tagging to publications (both textual data and graph
structure are considered)
– concept hierarchy construction
• Based on 2000 initial seed concepts, over 228K
concepts have been identified
• Concepts are organized in six-level hierarchy
• 1 billion publication-concept relations
40

42. Conclusions
41

43. Conclusions
• Algorithmic approaches can be used to construct large-scale classifications
• Algorithmic classifications at the level of publications gain popularity
• Algorithmic possibilities depend on data availability
• Algorithmic classifications may have the disadvantage of mixing up different
principles for classifying items (e.g., research topic, research method,
scientific community, theoretical tradition, basic vs. applied)
42

44. Thank you for your attention!
43