This document discusses university rankings and issues with various ranking systems. It begins by looking at the World Reputation Rankings 2014 where the University of Pavia obtained a good position, ranking fifth among the 15 Italian universities present and gaining 76 positions from 2013. It then discusses problems that have occurred with rankings, such as one scholar artificially inflating the ranking of their university by frequently self-publishing. Overall, the document examines the methodologies and shortcomings of different international university rankings.

1. Quell’oscuro
ogge,o
del
desiderio:
le
classifiche
internazionali
degli
atenei
Giuseppe
De
Nicolao
Università
di
Pavia

3. SOMMARIO
Prologo:
“mamma
li
Turchi!”
1. Il
grande
balzo
2. Dall’EgiBo
con
furore
3. Should
you
believe
in
the
Shanghai
ranking?
4. Numb3rs!
5. Il
potere
dei
numeri
6. Dove
sNamo
andando?

4. Prologo
Mamma
li
Turchi!

6. «Siamo agli ultimi posti nelle
classifiche mondiali. Per questo
motivo presenteremo a novembre la
riforma dell’Università, [...]
Mi auguro di non dover più vedere in
futuro - conclude - la prima
università italiana al 174mo posto»

7. Capitolo
1
Il
grande
balzo

8. 27
Aprile
2014

11. E’ stato pubblicato il World Reputation Rankings 2014,
la classifica delle università mondiali elaborata dalla
rivista inglese Times Higher Education.
L’Università di Pavia ottiene un ottimo
posizionamento, classificandosi quinta tra le 15
università italiane presenti, guadagnando 76 posizioni
rispetto al 2013.
La classifica è, come di consueto, dominata dalle
università anglosassoni che monopolizzano anche la top
ten con l’autorevole Harvard salda al primo posto forte di
una percentuale del 100% dei consensi espressi dagli
accademici interpellati.

12. ... con l’autorevole Harvard salda al primo posto forte
di una percentuale del 100% dei consensi espressi
dagli accademici interpellati.

13. ... con l’autorevole Harvard salda al primo posto forte
di una percentuale del 100% dei consensi espressi
dagli accademici interpellati.
Gli americani so’ forti…
Ammazza gli americani,
aoh! Non puoi mica
combattere contro gli
americani!

14. ... con l’autorevole Harvard salda al primo posto forte
di una percentuale del 100% dei consensi espressi
dagli accademici interpellati.

15. ... con l’autorevole Harvard salda al primo posto forte
di una percentuale del 100% dei consensi espressi
dagli accademici interpellati.
Gli americani so’ forti… ma non
fino al 100% dei consensi
(quelli sono i Bulgari)
The scores are based on the number of times an
institution is cited by respondents as being the
best in their field. The number one institution,
Harvard University, was selected most often.
The scores for all other institutions in the table
are expressed as a percentage of Harvard’s,
set at 100.

16. E’ stato pubblicato il World
Reputation Rankings 2014, la
classifica delle università
mondiali elaborata dalla rivista
inglese Times Higher Education.
L’Università di Pavia ottiene un
ottimo posizionamento,
classificandosi quinta tra le 15
università italiane presenti,
guadagnando 76 posizioni
rispetto al 2013.

17. March
6,
2014
100
università
nessuna
italiana

18. March
6,
2014
100
università
nessuna
italiana
?

19. ...
andiamo
a
guardare
il
video
su
YouTube
...

20. World
University
Rankings

21. sono
due
classifiche
diverse

22. 100
università
nessuna
italiana
400
università
15
italiane

23. Mistero
spiegato?
Sì,
ma
...

24. “Non
capisco
il
miscuglio
informa1vo
fa3o
dai
vostri
Top
Managers”

25. October
3,
2013
March
6,
2014
sono
due
classifiche
diverse
pubblicate
in
date
diverse

26. October
3,
2013
March
6,
2014
sono
due
classifiche
diverse
pubblicate
in
date
diverse

28. E allora consoliamoci con qualche dolcetto che ci riserva, comunque, questa
edizione. Abbiamo 15 Università nel “girone di serie B”, quello compreso fra il 200°
e il 400° posto; l’anno scorso erano 14. Ed inoltre la maggioranza di queste 15
guadagna “bande” migliori rispetto al 2012 (ricordiamo che in questo settore la
posizione è data per gruppi di 25 o di 50). Trento guida la pattuglia:

29. World
Univ.
Ranking
Home
page
Unipv
World
ReputaXon
Ranking

30. Capitolo
1
Dall’Egi,o
con
furore
Capitolo 2
Dall’Egitto
con furore

31. rewind
...
16
se,embre
2010

36. New
York
Times,
November
14,
2010
Alexandria’s
surprising
prominence
was
actually
due
to
“the
high
output
from
one
scholar
in
one
journal”
—
soon
idenNfied
on
various
blogs
as
Mohamed
El
Naschie,
an
EgypNan
academic
who
published
over
320
of
his
own
arXcles
in
a
scienXfic
journal
of
which
he
was
also
the
editor.

38. rewind
...
26
novembre
2008

40. • ...
of
the
400
papers
by
El
Naschie
indexed
in
Web
of
Science,
307
were
published
in
Chaos,
Solitons
and
Fractals
alone
while
he
was
editor-­‐
in-­‐chief.
• El
Naschie’s
papers
in
CSF
make
4992
citaXons,
about
2000
of
which
are
to
papers
published
in
CSF,
largely
his
own.

41. never
again?

42. THE
ranking
2012:
oops,
I
did
it
again!

43. • Only
a
few
of
the
more
than
100
co-­‐authors
of
2008
and
2010
reviews
were
from
MEPhI
• The
2008
parNcle
physics
review
received
nearly
300
Nmes
as
many
citaNons
in
the
year
a]er
publicaNon
as
the
mean
for
that
journal
• Cites
were
averaged
over
the
relaXvely
small
number
of
MEPhI’s
publicaNons
yielding
a
very
high
citaNon-­‐rate
• Further,
if
citaNons
are
generally
low
in
their
countries,
then
insNtuNons
get
some
more
value
added
(regional
modifica1on)
THE
ranking
2012:
oops,
I
did
it
again!

44. what
about
other
rankings?

45. the
top
10
most
spectacular
errors
of
…
reviewed
by
University
Ranking
Watch

46. QS
greatest
hits:
internaXonal
Students
and
Faculty
in
Malaysian
UniversiXes
• In
2004
UniversiN
Malaya
(UM)
in
Malaysia.
reached
89th
place
in
the
THES-­‐QS
world
rankings.
• In
2005
came
disaster.
UM
crashed
100
places
• PoliNcal
opposiNon:
shame
on
the
university
leadership!
• Real
explanaNon:
lot
of
Malaysian
ciNzens
of
Indian
and
Chinese
descent
erroneously
counted
as
“foreigners”.

47. QS
greatest
hits:
500
wrong
student
faculty
raXos
in
2007
QS
Guide
• Someone
slipped
three
rows
when
copying
and
pasNng
student
faculty
raNos:
Dublin
InsNtute
of
Technology
was
given
Duke’s
raNo,
Pretoria
got
Pune’s,
Aachen
RWT
got
Aberystwyth’s
(Wales).
And
so
on.
Altogether
over
500
errors.

48. Let’s
go
technical
...

49. Capitolo
3
Should
you
believe
in
the
Shanghai
ranking?

50. Shanghai:
criteri
e
importanza
%

52. The
“normalizaXon
trap”
1/2

53. The
“normalizaXon
trap”
2/2

54. Should
you
believe
in
the
Shanghai
ranking?
An
MCDM
view
J.-­‐C.
Billaut
D.
Bouyssou
P.
Vincke
• all
criteria
used
are
only
loosely
connected
with
what
they
intended
to
capture.
• several
arbitrary
parameters
and
many
micro-­‐decisions
that
are
not
documented.
• flawed
and
nonsensical
aggregaNon
method
• «the
Shanghai
ranking
is
a
poorly
conceived
quick
and
dirty
exercise»
«any
of
our
MCDM
student
that
would
have
proposed
such
a
methodology
in
her
Master’s
Thesis
would
have
surely
failed
according
to
our
own
standards»

55. HUNDREDS OF UNIVERSITIES
WITH SIMILAR SCORES
SCORE
SHANGHAI RANKING:
HOW MUCH RELIABLE?
RANK

57. THE RANKING
SHANGHAIRANKING

58. Twenty
Ways
to
Rise
in
the
Rankings
(1/3)
by
Richard
Holmes
h,p://rankingwatch.blogspot.it/2013/12/twenty-­‐ways-­‐to-­‐rise-­‐in-­‐rankings-­‐quickly.html
1.
Get
rid
of
students.
The
university
will
therefore
do
beBer
in
the
faculty
student
raNo
indicators.
2.
Kick
out
the
old
and
bring
in
the
young.
Get
rid
of
ageing
professors,
especially
if
unproducNve
and
expensive,
and
hire
lots
of
temporary
teachers
and
researchers.
5.
Get
a
medical
school.
Medical
research
produces
a
disproporNonate
number
of
papers
and
citaNons
which
is
good
for
the
QS
citaNons
per
faculty
indicator
and
the
ARWU
publicaNons
indicator.
Remember
this
strategy
may
not
help
with
THE
who
use
field
normalisaNon.

59. 7.
Amalgamate.
What
about
a
new
mega
university
formed
by
merging
LSE,
University
College
London
and
Imperial
College?
Or
a
tres
grande
ecole
from
all
those
liBle
grandes
ecoles
around
Paris?
9. The
wisdom
of
crowds.
Focus
on
research
projects
in
those
fields
that
have
huge
mulX
-­‐
“author”
publicaXons,
parNcle
physics,
astronomy
and
medicine
for
example.
Such
publicaNons
o]en
have
very
large
numbers
of
citaNons.
10.
Do
not
produce
too
much.
If
your
researchers
are
producing
five
thousand
papers
a
year,
then
those
five
hundred
citaNons
from
a
five
hundred
“author”
report
on
the
latest
discovery
in
parNcle
physics
will
not
have
much
impact.
Twenty
Ways
to
Rise
in
the
Rankings
(2/3)
by
Richard
Holmes
h,p://rankingwatch.blogspot.it/2013/12/twenty-­‐ways-­‐to-­‐rise-­‐in-­‐rankings-­‐quickly.html

60. 13.
The
importance
of
names.
Make
sure
that
your
researchers
know
which
university
they
are
affiliated
to
and
that
they
know
its
correct
name.
Keep
an
eye
on
Scopus
and
ISI
and
make
sure
they
know
what
you
are
called.
18.
Support
your
local
independence
movement.
Increasing
the
number
of
internaNonal
students
and
faculty
is
good
for
both
the
THE
and
QS
rankings.
If
it
is
difficult
to
move
students
across
borders
why
not
create
new
borders?
20.
Get
Thee
to
an
Island.
Leiden
Ranking
has
a
liBle
known
ranking
that
measures
the
distance
between
collaborators.
At
the
moment
the
first
place
goes
to
the
Australian
NaNonal
University.
Twenty
Ways
to
Rise
in
the
Rankings
(3/3)
by
Richard
Holmes
h,p://rankingwatch.blogspot.it/2013/12/twenty-­‐ways-­‐to-­‐rise-­‐in-­‐rankings-­‐quickly.html

61. The
Berlin
Principles
(1/2)
• In
2004,
the
InsNtute
for
Higher
EducaNon
Policy
in
Washington
and
the
UNESCO
European
Centre
for
Higher
EducaNon
(UNESCO-­‐CEPES)
in
Bucharest
founded
the
InternaXonal
Ranking
Expert
Group
(IREG)
in
2004.
• In
2006,
the
group
developed
a
set
of
principles
of
quality
and
good
pracNce
for
higher
educaNon
rankings
–
referred
to
as
the
Berlin
Principles
on
Ranking
of
Higher
EducaNon
InsNtuNons
(IREG,
2006).
• Guidelines
on
four
main
areas:
– purposes
and
goals
– methodologies
(design
and
weighNng
of
indicators),
– collecNon
and
processing
of
data,
and
– presentaNon
of
ranking
results.

62. The
Berlin
Principles
(2/2)
• Nel
2010,
su
una
scala
da
1
(nessuna
congruenza)
a
5
(congruenza
eccellente),
su
25
rankings
ben
13
non
raggiungevano
nemmeno
il
3
(congruenza
acceBabile),
tra
cui
THE-­‐QS,
allora
ancora
unificata,
che
o,eneva
2.25,
molto
vicino
al
2
(congruenza
scarsa).
STOLZ
HENDEL
HORN
(2010):
I.
Stolz,
D.
Hendel,
D.,
A.
Horn,
A,
Ranking
of
rankings:
benchmarking
twenty-­‐
five
higher
educaNon
ranking
systems
in
Europe,
Higher
EducaNon,
Online,
p.
22.
Springer,
Netherlands.
• Nel
dicembre
2011,
l’IREG
ha
reso
nota
la
possibilità
di
oBenere
una
cerNficazione
“IREG
approved”
per
le
classifiche
che
superassero
un
audit
basata
sui
Berlin
Principles.
• Fino
a
Maggio
2013,
nessun
“IREG
approved”.
• Un
anno
dopo,
solo
tre
rankings
sono
“IREG
approved”
,
tra
cui
QS
è
l’unico
internazionale.

63. Capitolo
3
Numb3rs!
Capitolo
4

64. Let’s
open
the
box
UNIVERSITY
RANKING
RAW
DATA

65. Fonte:
“Malata
e
Denigrata”
a
cura
di
M.
Regini,
Donzelli
2009
INTERNAZIONALIZZAZIONE

66. Let’s
open
the
box
1. “ScienNfic
excellence”
2. Funding

67. 1.
“ScienXfic
excellence”

68. Classifiche
degli
atenei:
valore
scien1fico
assai
dubbio.
Come
si
può
misurare
il
peso
di
una
nazione
nel
panorama
scien1fico
internazionale?
Contando
gli
arXcoli
scienXfici
che
produce
e
le
citazioni
che
quesX
o,engono

69. Italia:
8°
per
arXcoli
scienXfici
Fonte:
SCImago
su
daN
Scopus
1996-­‐2012

70. 0
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
1985
1990
1995
2000
2005
2010
Regno
Unito
Germania
Giappone
Francia
Canada
Italia
Spagna
Olanda
Svizzera
Svezia
PUBBLICAZIONI (WoS)

71. 0
1000000
2000000
3000000
4000000
5000000
6000000
PUBBLICAZIONI 2004-2010:
NUMERO DI CITAZIONI
Fonte: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 4.1)
(dati ISI Web of Knowledge, Thomson-Reuters)
http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

72. Efficienza:
Italia
ba,e
Germania,
Francia
e
Giappone

73. OCTOBER 2009
Ma come fanno questi italiani a produrre
così tanta ricerca con così poche risorse?

74. 2.
Funding

75. Quanto è elitaria la “top 500”?
OTHER 16,500
UNIVERSITIES TOP 500
PERFORMANCE
NUMBEROFUNIVERSITIES
... e cosa costa stare in cima?

76. OPERATING
EXPENSES
FONDO
FINANZIAMENTO
ORDINARIO
2012
MILIARDI
DI
EURO
LE
“OPERATING
EXPENSES”
DI
HARVARD
AMMONTANO
AL
44%
DEL
FONDO
DI
FINANZIAMENTO
DELL’INTERO
SISTEMA
UNIVERSITARIO
STATALE
ITALIANO

77. E.
Hazelkorn:
“EsFmated
yearly
budget
of
€1.5
billion
to
be
ranked
in
the
world’s
top
100”

78. Spesa per università (% PIL):
l’Italia è 30° su 33 (fonte: OCSE 2013)

79. e
ciò
nonostante
...

80. % di Atenei che entrano nei “top 500”
(Leiden: top 250)
Fonte
dei
daF:
“Malata
e
denigrata
:
l’universita
italiana
a
confronto
con
l’Europa”
(a
cura
di
M.
Regini,
Roma,
Donzelli
2009)
CLASSIFICA:

81. Capitolo
5
The
power
of
numbers

82. Rankings
• Fragile
scienNfic
grounds
• IncenNve
to
gaming
• Raw
data
are
obscured
• They
are
not
necessary
to
manage
funding
(see
RAE/REF)
Why,
then?

84. Rankings
are
based
on
composite
indicators
Science
or
pseudo-­‐science?

85. Aggregators
vs
non-­‐aggregators
• Aggregators:
value
in
combining
indicators:
extremely
useful
in
garnering
media
interest
and
hence
the
aVenFon
of
policy
makers
• Non-­‐aggregators:
key
objecFon
to
aggregaFon:
the
arbitrary
nature
of
the
weighFng
process
by
which
the
variables
are
combined

87. Germany
• “We
look
back
decades
and
people
came
to
German
universiFes;
today
they
go
to
US
universiFes.”
• The
ExzellenziniXaXve
(2005):
from
tradiNonal
emphasis
on
egalitarianism
towards
compeNNon
and
hierarchical
straXficaXon

88. France
• The
Shanghai
ranking
“generated
considerable
embarrassment
among
the
French
intelligentsia,
academia
and
government:
the
first
French
higher
educaFon
insFtuFon
in
the
ranking
came
only
in
65th
posi1on,
mostly
behind
American
universiFes
and
a
few
BriFsh
ones”

89. Australia
• The
SJT
and
QS:
at
least
two
Australian
universiNes
among
the
top
100.
• Opposing
strategic
opNons:
– fund
a
small
number
of
top-­‐Ner
compeNNve
universiNes
– “creaNon
of
a
diverse
set
of
high
performing,
globally-­‐focused
insNtuNons,
each
with
its
own
clear,
disXncXve
mission”.

90. Japan
• “The
government
wants
a
first
class
university
for
internaFonal
presFge
”
• “in
order
for
Japanese
HEIs
to
compete
globally,
the
government
will
close
down
some
regional
and
private
universiFes
and
direct
money
to
the
major
universi1es”
• some
insNtuNons
will
become
teaching
only.

91. Why obsessing about the “top 1%”?
OTHER 16,500
UNIVERSITIES TOP 1%
PERFORMANCE
NUMBEROFUNIVERSITIES

92. Answer:
trickle-­‐down
knowledge!

93. E.
Hazelkorn
on
rankings
• 90
or
95%
of
our
students
do
not
aBend
elite
insNtuNons.
Why
are
we
spending
so
much
on
what
people
aren’t
a,ending
as
opposed
to
what
they
are
aBending?
• EsNmated
yearly
budget
of
€1.5
billion
to
be
ranked
in
the
world’s
top
100.
May
detract
resources
from
pensions,
health,
housing,
....
• Are
“elite”
insNtuNons
really
driving
naNonal
or
regional
economic
and
social
development?

94. Does
trickle-­‐down
work?
“Governments and universities must stop obsessing about
global rankings and the top 1% of the world's 15,000
institutions. Instead of simply rewarding the achievements
of elites and flagship institutions, policy needs to focus on
the quality of the system-as-a-whole.”
There is little evidence
that trickle-down works.

95. Capitolo
6

96. Where
are
we?
• (Even)
Phil
Baty
(Times
Higher
EducaFon)
admits
that
there
are
aspects
of
academic
life
where
rankings
are
of
liBle
value
• Can
we/you
afford
the
‘reputaNon
race’?
• We
will
have
to
live
in
a
world
in
which
extremely
poor
rankings
are
regularly
published
and
used.
What
can
be
done
then?

97. “Stop
being
naive”
• There
is
no
such
thing
as
a
‘‘best
university’’
in
abstracto.
• Stop
talking
about
these
‘‘all
purpose
rankings’’.
They
are
meaningless.
• Lobby
in
our
own
insNtuNon
so
that
these
rankings
are
never
menXoned
in
insNtuNonal
communicaNon
• Produce
many
alternaXve
rankings
that
produce
vastly
different
results.

98. Grazie
per
l’a,enzione!