Bibliometrics and
scientometrics
Part I: the classical performance indicators
Pablo Achard - NCCR Affective Sciences
Tuesday, February 24, 2009

1.What are we talking about?
2.Basic indicators
3. Performance indicators at the
micro level
4.Performance indicators at the
meso and macro level
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

1.What are we talking about?
2.Basic indicators
3. Performance indicators at the
micro level
4.Performance indicators at the
meso and macro level
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Definitions
“Bibliometrics” is introduced by Pritchard in 1969:
“the application of mathematical and statistical
methods to books and other media of
communication”
“Scientometrics” is the science of measuring and
analyzing science. As such, it includes the bibliometrics of
scientific books and articles; but it also takes into
account funding, demography, geography, etc.
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

What is it used for
1. Science and Technology Studies (epistemology, science
sociology, science history,…) and the scientific study of
idea spread
2. Biblioeconomics
3. Science management
4. Rankings (used by journalists, students, scientists,…)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

What is it used for
P. Achard “Bibliometrics and scientometrics”
Lindberg, PhD Thesis
Tuesday, February 24, 2009

Why is it becoming so
important
1. Availability of large databases
2. Increased use of management tools in the research
administration
3. Globalization of the education market
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

What is this talk focusing on?
The managerial use of performance indicators at the
micro level (individuals, groups), and at the meso and
macro levels (from departments to continents)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

What is this talk focusing on?
The managerial use of performance indicators at the
micro level (individuals, groups), and at the meso and
macro levels (from departments to continents)
Therefore we will not deal with important topics like:
- The study of idea spread
- Demographics, education, collaborations
- Links between Research and Development, public
P. Achard “Bibliometrics and scientometrics”
understanding of science
Tuesday, February 24, 2009

P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

1.What are we talking about?
2.Basic indicators
3. Performance indicators at the
micro level
4.Performance indicators at the
meso and macro level
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Direct input indicators
Number of scientists
•
R&D budgets
•
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Direct output indicators
Number of publications / books / abstracts
•
Number of patents
•
Number of PhDs
•
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Influence indicators
Measure the “passive” effects
Number of citations
•
Number of downloads
•
Pagerank of a website
•
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Number of citations ≠ quality
The number of citations measures the reception
“if a paper receives 5 or 10 citations a year throughout
several years after its publication, it is very likely that its
content will become integrated into the body of knowledge
of the respective subject field; if, on the other hand, no
reference is made at all to the paper during 5 to 10 years
after publication, it is likely that the results involved do not
contribute essentially to the contemporary scientific
paradigm system of the subject field in question”
P. Achard “Bibliometrics and scientometrics”
(Braun et al. 1985)
Tuesday, February 24, 2009

Number of citations ≠ quality
Book evaluation vs citations:
“The J-shaped distribution of
citedness” (Nicolaisen 2002)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Number of citations: issues
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Number of citations: issues
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Normalized indicators
Number of citations / number of expected citations for
similar publications
The “Crown indicator” of Leiden University: Normalized
by publication type, year and field
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

The myth of delayed
recognition
Papers highly cited only after a period of 5 years = 60
P. Achard “Bibliometrics and scientometrics”
out of 450’000 published in 1980 or 0.013%
Tuesday, February 24, 2009

P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009
IF

IF
Statistics matter: the mean
value is a very rough
description of a distribution
P. Achard “Bibliometrics and scientometrics”
(W. Glänzel, 2003)
Tuesday, February 24, 2009

P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Methodological pitfalls
Field delineation
Aggregation level
Time frame
Counting scheme
Data quality
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

1.What are we talking about?
2.Basic indicators
3. Performance indicators at the
micro level
4.Performance indicators at the
meso and macro level
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Warning 1: Mating...
You wouldn’t mary someone based on his/her picture
on a mating website
But if you look for partners on a mating website, you
should care about the picture you upload
(using Photoshop is a matter of personal ethics)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Warning 1: Mating...
You wouldn’t mary someone based on his/her picture
on a mating website
But if you look for partners on a mating website, you
should care about the picture you upload
(using Photoshop is a matter of personal ethics)
Apply the same rules with evaluating someone’s
P. Achard “Bibliometrics and scientometrics”
scientific value and bibliometrics
Tuesday, February 24, 2009

Warning 2: Distortion...
1. Funding agencies and science policy makers want to
get the best science for their money. But there are
more scientists than one can know and more fields
than one can understand.
2. They fund / hire preferentially projects / scientists
with the best score on a metric that correlates with the
‘quality’ they are looking for.
3. Scientists adapt their publication behavior to
P. Achard “Bibliometrics and scientometrics”
increase their score on this metric.
Tuesday, February 24, 2009

Individual performance
indicators
Number of publications / books / abstracts
Number of patents
Problem: no indication of the influence of the work
Distortions: multiplication of small papers (Least
P. Achard “Bibliometrics and scientometrics”
Publishable Unit); “honorary” authors
Tuesday, February 24, 2009

Individual performance
indicators
Number of citations received
Problems: one article can make it all (highly skewed); too
long to build
Distortions: self-citations; citation exchange
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Individual performance
indicators
H-index:
\"A scientist has
index h if h of his or
her Np papers have
at least h citations
each and the other
(Np - h) papers have
fewer than h
citations each\"
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Individual performance
indicators
H-index:
Problems: very strong
\"A scientist has
correlation with the
index h if h of his or
number of publications;
her Np papers have
perfect to compare
at least h citations
scientists... at the end of
each and the other
their career!
(Np - h) papers have
fewer than h
Distortions: same as
citations each\"
P. Achard “Bibliometrics and scientometrics”
publication numbers
Tuesday, February 24, 2009

Individual performance
indicators
Many variants of the h-index
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Individual performance
indicators
Number of publications x IF of the journal in which they
where published
Problems: highly noisy; very field-dependent; IF doesn’t
determine future citation
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

The field dependency of IF
P. Achard “Bibliometrics and scientometrics”
(Leydesdorff, 2008)
Tuesday, February 24, 2009

IF-citation correlation?
Number of citations vs
IF of the journal in
which articles were
published for each and
every article of 4
researchers
P. Achard “Bibliometrics and scientometrics”
(Seglen, 1997)
Tuesday, February 24, 2009

Individual performance
indicators
Number of publications x IF of the journal in which they
where published
Problems: highly noisy; very field-dependent; IF doesn’t
determine future citation
Distortions: editors “hot topics”; worse at the
P. Achard “Bibliometrics and scientometrics”
macroscopic level
Tuesday, February 24, 2009

How to increase the IF of your
university?
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

How to increase the IF of your
university?
Just cut the departments with low citation rates!
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

How to increase the IF of your
university?
Just cut the departments with low citation rates!
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

How to increase the IF of your
university?
Just cut the departments with low citation rates!
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

How to increase the IF of your
university?
Just cut the departments with low citation rates!
Do you think I’m kidding?
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

How to increase the IF of your
university?
Just cut the departments with low citation rates!
Do you think I’m kidding?
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

How to increase the IF of your
university?
Just cut the departments with low citation rates!
Do you think I’m kidding?
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Individual performance
indicators
P. Achard “Bibliometrics and scientometrics”
www.phdcomics.com
Tuesday, February 24, 2009

1.What are we talking about?
2.Basic indicators
3. Performance indicators at the
micro level
4.Performance indicators at the
meso and macro level
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

University rankings
Shanghai = alumni awards score + staff awards score +
highly cited score + Nature and Science articles +
articles + a mixture of all this/number of faculty
Times Higher Education = peer-review score +
employer-review score + staff/student score + citation/
staff score + international student score
UNIGE ranking 2007 = 105 ranking 2008 = 68
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

University rankings
Shanghai = alumni awards score + staff awards score +
highly cited score + Nature and Science articles +
articles + a mixture of all this/number of faculty
Times Higher Education = peer-review score +
employer-review score + staff/student score + citation/
staff score + international student score
UNIGE ranking 2007 = 105 ranking 2008 = 68
P. Achard “Bibliometrics and scientometrics”
Let’s have a look at more serious indicators!
Tuesday, February 24, 2009

Publications per country
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Publications per country
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Publications per country
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Publications vs GDP
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Citations vs GDP (normalized)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

Field dependency (again)
Clinical medicine (MED)
Biomedical research (BRE)
Biology (BIO)
Chemistry (CHE)
Physics (PHY)
Mathematics (MAT)
Engineering (ENG)
Earth and space sciences (ESS)
P. Achard “Bibliometrics and scientometrics”
(Glänzel, 2003)
Tuesday, February 24, 2009

Citations vs GDP (normalized)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009

P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009
Conclusions?

“The use of a single index crashes the multidimensional
space of bibliometrics into one single dimension.”
(Wolfgang Glänzel)
“Not everything that can be counted counts,
and not everything that counts can be counted.”
(Albert Einstein)
P. Achard “Bibliometrics and scientometrics”
Tuesday, February 24, 2009