This document compares how different disciplinary classifications influence bibliometric representations of research in social sciences and humanities. It analyzes journal article data from Flanders and Norway classified according to several systems, finding some agreements but also differences. While disciplinary structure is largely unaffected by classification, publication counts can vary substantially between classifications for some disciplines. Classifications reflect their local contexts, so no single system adequately represents all contexts. Future work should analyze more countries and classifications as well as classify articles directly.

1. Comparison of classification-related differences
in the distribution of journal articles across
academic disciplines
The case of social sciences and humanities
in Flanders and Norway (2006-2015)
Linda Sīle, Raf Guns, Frédéric Vandermoere, and Tim C. E. Engels
ISSI/STI 2019 Rome, 5 September 2019
@SileLinda
@RafG

2. Central question
How does the choice of disciplinary classification influence
bibliometric representations of research?
especially for study of social sciences and humanities in national
databases

3. Classification
• Disciplinary classifications enable:
• delineating document sets for analysis
• normalising indicators
• insights into research profile of institution, country…
• Operationalisation of ‘academic discipline’ is not straightforward
(Sugimoto & Weingart, 2015)
• How to classify set of articles into disciplines?
• content-based classifications < librarianship
• citation-based approaches
• text-based approaches
• journal classifications

4. National databases
• Often use local classifications (developed in specific local
context)
• Classifications are artifacts that carry traces of social/cognitive
context they originated in
Set of publications 1 Set of publications 2
Classified set 1 Classiffied set 2
classification A classification B
comparable?

5. National databases
• Often use local classifications (developed in specific local
context)
• Classifications are artifacts that carry traces of social/cognitive
context they originated in
Set of publications 1 Set of publications 2
Classified set 1 Classiffied set 2
classification A classification B
comparable?

6. Data
• Journal articles, peer-reviewed, from SSH
• Source: comprehensive bibliographic databases
• Flanders, Belgium:VABB-SHW
• Norway: CRISTIN
• Time period: 2006-2015
• Journal-based classifications:
• VABB (for Flemish data, based on OECD)
• NPU classification in Norwegian List of Scholarly Journals
• WoS subject categories
• Science-Metrix classification

7. Classifications
VABB
(Flanders)
NPU
(Norway)
SM
(Science-Metrix)
WOS
Number of SSH
categories in level 1
2 2 2 -
Number of SSH
categories in level 2
17 35 6 -
Number of SSH
categories in level 3
- - 43 76
Multiple
categories?
Yes No No Yes

8. Classifications
VABB
(Flanders)
NPU
(Norway)
SM
(Science-Metrix)
WOS OECD-
FORD
Number of SSH
categories in level 1
2 2 2 - 2
Number of SSH
categories in level 2
17 35 6 - 14
Number of SSH
categories in level 3
- - 43 76 -
Multiple
categories?
Yes No No Yes -
mapped to

9. Data
Set A Set B
[WOS only]
Set C
[SM only]
Flanders
# of publications
# of journals
30,915
5,797
13,513
2,377
12,313
1,994
Norway
# of publications
# of journals
26,007
5,172
9,451
2,089
10,399
1,974
B
WOS
A
C
SM

10. Results

11. Agreement in journal
classification
VABB –
WOS
NPU –
WOS
VABB –
SM
NPU –
SM
All journals 58% 54% 60% 66%
Journals assigned
to one category
94% 79% 73% -
Overall, higher agreement with Science-Metrix than with WOS
Large variation at discipline level, e.g.:
• Educational Sciences: 70–86%
• Sociology: 44–56%

12. Differences in share of journal articles across disciplines – WOS

13. Differences in share of journal articles across disciplines – WOS
NPU: Other social sciences
WOS: Social and economic geography

14. Differences in share of journal articles across disciplines – SM

15. Differences in share of journal articles across disciplines – SM
VABB: Psychology
SM: Sociology

16. Agreement in
classification of
articles • Number of SSH articles is greater when using local
classifications
• Differences at article level: smaller than at journal level
• Higher agreement with WOS than with SM (<> journals)
 Percentage differences:
Flanders WOS SM
Social sciences 1% 14%
Humanities 4% 10%
Norway WOS SM
Social sciences 7% 21%
Humanities 7% 17%

17. Differences in
disciplinary structure
• Disciplinary structure = relative size of each discipline
• Cosine similarity between distributions > 0.98
(all datasets and classifications)
 Disciplinary structure remains largely unaffected!

18. Discussion and
conclusion
Choice of classification:
• has only minor influence on disciplinary structure
• can substantially affect publication counts per discipline
For some disciplines (e.g., Languages and literature) there is more
agreement between classifications.
For others (e.g., Media and communications), differences can be
considerable.

19. Discussion and
conclusion
Classifications carry traces from their context.
• E.g., SSH research is understood more broadly in Norway
than internationally (difference NPU-SM = 17%)
• Unlikely that one classification is fit for all contexts
Future work
• Expand to other countries and classifications
• Article-level classifications

20. Thank you!
Linda Sīle, Raf Guns, FrédéricVandermoere, and Tim C. E. Engels
linda.sile@uantwerpen.be, raf.guns@uantwerpen.be, frederic.vandermoere@uantwerpen.be, tim.engels@uantwerpen.be
@SileLinda, @RafG