This document presents a novel method called Citation Classes (CSS) for classifying scientific publications based on their citation rates. CSS assigns publications to four performance classes in a self-adjusting manner without arbitrary thresholds. The document demonstrates how CSS can be applied at the level of institutions, countries, and individual researchers to generate detailed profiles of research performance. CSS was shown to be robust, flexible, and able to identify various types of highly-cited publications that traditional metrics cannot fully capture.

1. citation classes: a novel indicator base to
classify scientific output
Wolfgang Glänzel, Koenraad Debackere, Bart Thijs
Centre for R&D Monitoring and Dept. MSI, KU Leuven, Belgium

2. Structure of presentation
1. introduction
2. characteristic scores and scales (css)
3. profiling excellence in practice
3.1 CSS in institutional research evaluation
3.2 CSS in research assessment of individuals
4. discussion and conclusions
Glänzel et al., Citation Classes, Ghent, 2016 2/19

3.  Introduction 
The need for multi-level profiling of excellence
• Continuous debate about normalisation of bibliometric indicators
• Insufficiency of using a single indicator to depict research
performance
• Extreme values might bias scientometric indicators
• The disproportion between standard models of “regular” and
“outstanding” performance
• Limited applicability of traditional indicators to bibliometric author
profiling
Glänzel et al., Citation Classes, Ghent, 2016 3/19

4. Introduction
Extreme citation rate received by one paper till July 2016
The phenomenon of this “outlier” publication has been pointed out by
Waltman et al. (2012). The extreme citation rate of this paper even
biases the mean citation impact of the university.
Glänzel et al., Citation Classes, Ghent, 2016 4/19

5. Introduction
Desired properties of alternative bibliometric solutions:
• Provides parameter-free performance classes
• Applicable at different levels of aggregation
• Replaces the conception of “linearly structured” all-in-one indicators
by performance profiles
• Ensures the seamless integration of measures of outstanding
performance into the standard tools of scientometric performance
assessment
We propose an method that proceeds from a baseline model, which
usually consists of a complete bibliographic database.
Here we use Thomson Reuters Web of Science Core Collection but we
stress that Elsevier’s Scopus could be used as well.
Glänzel et al., Citation Classes, Ghent, 2016 5/19

6.  Characteristic Scores and Scales (CSS) 
What are Characteristic Scores and Scales?
Characteristic scores are obtained from iteratively truncating samples at
their mean value and recalculating the mean of the truncated sample
until the procedure is stopped or no new scores are generated.
Visualisation of characteristic scores and scales for four classes
b1
b2
b3
Class 2
Class 1
Class 3
Class 4
Advantages:
• Self-adjusting (no arbitrary thresholds)
• No tie problems when assigning publications to classes
Glänzel et al., Citation Classes, Ghent, 2016 6/19

7. Characteristic Scores and Scales (CSS)
Now we define the following four classes.
• Class 1 contains ‘poorly cited’ papers,
• Class 2 contains ‘fairly cited’ papers,
• Class 3 contains ‘remarkably cited’ papers and
• Class 4 is the class of ‘outstandingly cited’ papers.
Classes 3 and 4 are used to identify highly cited papers.
Glänzel et al., Citation Classes, Ghent, 2016 7/19

8.  Profiling excellence in practice 
CSS proved extremely robust: Although classes are not directly linked to
percentiles, the baseline provides a distribution of papers over classes of
about
70% (Class 1), 21% (Class 2), 6%–7% (Class 3) and 2%–3% (Class 4),
independently of publication year, citation window and subject field.
If the unit or scientist under study is a true “mirror” of the world
standard, the distribution of publication output over classes is expected
to coincide with that of the world.
Instead of a simple “above” and “below” as usually obtained in research
evaluation, deviations from the expectation or other given profiles are
more complex.
☞ There are five paradigmatic types of deviation from this standard.
Glänzel et al., Citation Classes, Ghent, 2016 8/19

9. Profiling excellence in practice
Five different profiles according to CSS distributions
Class 1 Class 2 Class 3 Class 4
Type I
Type II
Type III
Type V
Type IV
Glänzel et al., Citation Classes, Ghent, 2016 9/19

10. Profiling excellence in practice
Deviation of national publication profiles from the reference standard in CSS classes in 2007
with 5-year citation window (left) and 2009 with 3-year citation window (right)
Class 1 (69.8%) Class 2 (21.5%) Class 3 (6.3%) Class 4 (2.4%)
AUT BEL DNK ISR NLD POL
Class 1 (69.7%) Class 2 (21.4%) Class 3 (6.4%) Class 4 (2.5%)
AUT BEL DNK ISR NLD POL
Legend: The blue horizontal line indicates the reference standard the respective value of which is normalised to 1.0 for comparison
Source: Thomson Reuters – Web of Knowledge
Glänzel et al., Citation Classes, Ghent, 2016 10/19

11. CSS in institutional research evaluation
CSS performance classes in institutional research assessment
• For the cross-institutional comparison of class profiles two
universities each from eleven European countries were selected.
• The universities’ profiles mostly mirror the national patterns but
distinctly more or less favourable situations than in the
corresponding national standard could be found as well.
• The high standard of the technical university (DK1) is an example
for the subject-independence of the method.
Glänzel et al., Citation Classes, Ghent, 2016 11/19

12. CSS in institutional research evaluation
Deviation of university profiles from the reference standard in CSS classes in 2007 with
5-year citation window
Class 1 (69.8%) Class 2 (21.5%) Class 3 (6.3%) Class 4 (2.4%)
DK1 DK2 DNK
FI1 FI2 FIN
NL1 NL2 NLD
Legend: The blue horizontal line indicates the reference standard the respective value of which is normalised to 1.0 for comparison
Source: Thomson Reuters – Web of Knowledge
Glänzel et al., Citation Classes, Ghent, 2016 12/19

13. CSS in research assessment of individuals
For this example all citable items indexed in the 2010 volume of the WoS
have been selected.
Then authors from eight selected countries with a unique ResearcherID
and at least 20 publications each were chosen (N = 4.271).
☞ The selection of these authors implies a built-in bias since authors
with an Researcher-ID are on an average more productive than
others.
Glänzel et al., Citation Classes, Ghent, 2016 13/19

14. CSS in research assessment of individuals
Distribution of papers over performance classes
Data Set Class 1 Class 2 Class 3 Class 4
All papers of authors with RID 44.3 32.9 14.5 8.4
All papers indexed in 2010 69.8 21.5 6.2 2.5
☞ The CSS values of the two sets illustrate that the total author
population obeys the 70%–21%–9% rule, while registered authors
form a biased sample (of Type IV profile).
Glänzel et al., Citation Classes, Ghent, 2016 14/19

15. CSS and tradition citation indicators
Rank correlation between performance classes and citation indicators for the
authors with ‘Researcher ID’
Class 1 Class 2 Class 3 Class 4 RCR NMCR
Class 2 -0.459
Class 3 -0.743 0.058∗
Class 4 -0.685 -0.055∗ 0.485
MOCR/MECR -0.630 -0.001∗ 0.547 0.734
MOCR/FECR -0.839 0.125 0.693 0.863 0.786
MECR/FECR -0.632 0.202 0.482 0.547 0.163 0.701
Legend: Values marked with ∗
do not statistically deviate from 0
☛ These observations substantiate that citation behaviour is not sufficiently be
represented by one single indicator alone. – A strong argument for the choice
of this method with four performance classes at this aggregation level too.
Glänzel et al., Citation Classes, Ghent, 2016 15/19

16. Author profiling
Sample of 20 selected authors
Author Class 1 Class 2 Class 3 Class 4 MOCR/MECR MOCR/FECR MECR/FECR Type
1 31.8% 27.3% 36.4% 4.5% 1.32 2.01 1.53 I
2 39.3% 60.7% 0.0% 0.0% 1.13 1.24 1.09 I
3 32.5% 45.0% 22.5% 0.0% 0.98 1.59 1.61 I
4 40.0% 24.4% 33.3% 2.2% 1.36 1.65 1.21 I
5 67.5% 28.6% 1.3% 2.6% 1.30 1.01 0.77 II
6 58.3% 32.5% 7.3% 2.0% 1.00 1.18 1.19 II
7 74.2% 16.1% 9.7% 0.0% 1.21 0.75 0.62 II
8 72.0% 20.0% 8.0% 0.0% 0.65 0.82 1.27 II
9 68.2% 27.3% 0.0% 4.5% 0.82 0.79 0.96 III
10 33.3% 52.4% 9.5% 4.8% 1.56 1.82 1.17 III
11 61.9% 38.1% 0.0% 0.0% 0.70 0.80 1.15 III
12 33.3% 28.9% 20.0% 17.8% 1.89 3.35 1.77 III
13 21.4% 21.4% 14.3% 42.9% 2.65 5.85 2.21 IV
14 9.4% 50.0% 21.9% 18.8% 1.68 2.72 1.62 IV
15 17.9% 14.3% 28.6% 39.3% 3.21 6.34 1.97 IV
16 30.0% 20.0% 40.0% 10.0% 1.80 2.29 1.27 IV
17 56.8% 22.4% 11.2% 9.6% 1.47 1.83 1.24 V
18 62.5% 16.1% 12.5% 8.9% 1.38 1.92 1.39 V
19 47.6% 9.5% 9.5% 33.3% 14.58 35.46 2.43 V
20 50.0% 10.7% 14.3% 25.0% 3.75 3.69 0.98 V
Glänzel et al., Citation Classes, Ghent, 2016 16/19

17.  Discussion and conclusions 
Main conclusions
• The analysis of the high end of scientific distributions is one of the most
challenging issues in research evaluation.
• In the past it was difficult to draw a borderline between “very good” and
“outstanding” and outstanding citation impact could not be explained by
the “standard” citation behaviour.
• Application of CSS proved independent of the unit’s or scientist’s research
profile.
• The method proved its general applicability to the level of individual
authors and to author profiling of candidates with scientific excellence.
• The four paradigmatic profile types are more pronounced at lower
aggregation levels, notably at the level of individual scientists.
• Will the CSS work for “altmetrics” as well?
☞ A pilot study has shown the same robustness and 70%–21%–9% rule in
Thomson Reuters ‘usage’ statistics (Glänzel & Chi, 2016), but further
research is needed.
Glänzel et al., Citation Classes, Ghent, 2016 17/19

18. Discussion and conclusions
Main conclusions (contd.)
• Scientometrics evolved from a sub-discipline of library and information
science to an instrument for evaluation and benchmarking in support of
science policy.
• This evolution requires the development of indicators that allow to map and
assess both institutional and individual profiles of scientific activity and
visibility.
• The CSS method proves to offer a robust and valid indicator set, thus
supporting the use of scientometric data for science policy purposes.
• The insights presented in this paper confirm the seamless integration of the
CSS method into the standard toolset of scientometric research evaluation.
The main idea being and remaining to step away from the traditional linear
thinking by depicting reality in a more differentiated way.
Glänzel et al., Citation Classes, Ghent, 2016 18/19

19. THANK YOU VERY MUCH FOR YOUR
ATTENTION!
Vielen Dank für Ihre Aufmerksamkeit!
Hartelijk dank voor uw aandacht!
¡Muchísimas gracias por su atención!
Köszönöm szépen a figyelmüket!