We address the question how citation-based bibliometric indicators can best be normalized to ensure fair comparisons between publications from different scientific fields and different years. In a systematic large-scale empirical analysis, we compare a traditional normalization approach based on a field classification system with three source normalization approaches. We pay special attention to the selection of the publications included in the analysis. Publications in national scientific journals, popular scientific magazines, and trade magazines are not included. Unlike earlier studies, we use algorithmically constructed classification systems to evaluate the different normalization approaches. Our analysis shows that a source normalization approach based on the recently introduced idea of fractional citation counting does not perform well. Two other source normalization approaches generally outperform the classification-system-based normalization approach that we study. Our analysis therefore offers considerable support for the use of source-normalized bibliometric indicators.

1. A systematic empirical comparison
of different approaches for
normalizing citation impact
indicators
Ludo Waltman and Nees Jan van Eck
Centre for Science and Technology Studies (CWTS), Leiden University
14th ISSI conference, Vienna, Austria
July 16, 2013

2. 2
Introduction
• Citation-based indicators need to be normalized for
differences in citation practices between fields
• Traditional normalization based on WoS subject
categories is problematic because many subject
categories are heterogeneous in terms of citation
practices

3. 3
Clinical Neurology: Citation density
Visualization produced using VOSviewer
(Van Eck et al., PLoS ONE, 2012)

4. 4
Clinical Neurology: Reference density
Density of references instead of citations
Notice the similar patterns in the two visualizations!

5. 5
Normalization approaches
• Normalization based on a classification system (‘cited-
side normalization’)
• Source normalization (‘citing-side normalization’)

6. 6
Normalization based on classification system
• Requires a field classification system in which
publications are assigned to fields
• Following common practice, we use the WoS journal
subject categories
• Citations are compared to the field average
e
c
NCS
Number of citations of a publication
Average number of citations of all
publications in the same field

7. 7
Source normalization (1)
• No field classification system is needed
• Citations are weighted differently depending on the
number of references in the citing publication or the
citing journal
• Instead of giving equal weight to each citation, equal
weight is given to each citing publication

8. 8
Source normalization (2)
• Three source normalization variants:
– Audience factor (Zitt & Small, JASIST, 2008):
– Fractional citation counting (Leydesdorff and colleagues):
– Revised SNIP (Waltman et al., JOI, 2013):
• Only ‘active references’ should be considered!


c
i ia1
)1( 1
SNCS


c
i ir1
)2( 1
SNCS


c
i ii rp1
)3( 1
SNCS
Average number of references per publication
in citing journal
Number of references in citing publication
Proportion of publications in citing journal
without references

9. 9
Evaluation
• To what degree does each normalization approach:
– correct for field differences?
– correct for differences in the age of publications?
• Using the same classification system both in the
implementation and in the evaluation of a normalization
approach gives biased results (Sirtes, JOI, 2012)
• For evaluation purposes, we use four classification
systems:
– WoS journal subject categories
– Algorithmically constructed classification systems A, B, and C

10. • 3.8 million WoS publications from the period 2007–2010
• Classification systems constructed using large-scale
clustering approach (Waltman &Van Eck, JASIST, 2012)
• Clusters defined at the level of individual publications
rather than at the journal level
• Number of clusters (research areas) per classification
system:
– Classification system A: 21
– Classification system B: 161
– Classification system C: 1334
Algorithmically constructed classification
systems
10

11. Average score per normalization
approach and per publication year
11
0
2
4
6
8
10
12
CS NCS SNCS(1) SNCS(2) SNCS(3)
Averagescore
2007
2008
2009
2010

12. Similarity of normalized citation
distributions
• Let’s now look beyond averages
• To what degree do fields have identical normalized
citation distributions?
12

13. Results based on WoS subject categories (1)
• Similarity of normalized citation distributions in different
fields
13
235 WoS subject categories; publication year 2007

14. Inequality index
• How to summarize the degree to which citation
distributions coincide?
• We use the methodological framework of Crespo et al.
(PLoS ONE, 2013)
• Citation distributions are compared percentile-by-
percentile using the Theil inequality index
14

15. Results based on WoS subject categories (2)
15

16. Results based on classification system A
16

17. Results based on classification system B
17

18. Results based on classification system C
18

19. 19
Citation density before
normalization
Citation density after
normalization using SNCS(3)

20. 20
Citation density before normalization
Citation density after normalization using SNCS(3)

21. Conclusions
• Using the same classification system both in the
implementation and in the evaluation of a normalization
approach should be avoided
• NCS (subject-category-based normalization) and SNCS(2)
(‘fractional citation counting’) do not perform so well
• SNCS(1) (‘audience factor’) and SNCS(3) (‘revised
SNIP’) have a good performance
• Need for more practical experience with SNCS(1),
SNCS(3), and alternative normalization approaches, in
particular percentile-rank normalization
21

22. 2222
Thank you for your attention!

23. Problem of using the same classification
system both in the implementation and in the
evaluation of the NCS approach
Publication Field No. of citations NCS
1 A 1 0.50
2 A 3 1.50
3 B 2 0.50
4 B 6 1.50
23
Publication Field No. of citations NCS
1 X 1 0.67
2 Y 3 0.67
3 X 2 1.33
4 Y 6 1.33
• Results based on the correct assignment of publications to fields
• Results based on an incorrect assignment of publications to fields

24. 24
Practical issues
• World average
• Document types
• Citation windows
• PPtop indicators
• ‘Trade journal problem’

25. 25
A related problem (1)

26. 26
A related problem (2)
298 articles in 2008
40 citations

27. 27
A related problem (3)
Resp. 266 and 486 articles in 2008
Resp. 6% and 15% of all articles and reviews in WoS
categories Business and Business, finance
Resp. 29 and 8 citations

28. 28
National journals
Effect of excluding national journals on MNCS
See Waltman & Van Eck (2012)

29. 29
National journals
Effect of excluding national journals on MSNCS3