"Pride and Prejudices: Problems with national and international league tables" - Presentation delivered at AUA conference 19 April 2011

1. Pride and prejudices:
Problems with national
and international league
tables
Dr Paul Greatrix, Registrar,
University of Nottingham
1

2. League Tables
• Background
• Who wants to know?
• Who are the providers of this valuable
information?
• Mad, bad and dangerous
• National and international dimensions
• Measuring the unmeasurable?
2

3. Background
• They sell papers...
• ...and, some would suggest, create 3
perverse incentives

4. Who wants to know?
• The Government • Parents, teachers,
• The State advisors
• Employers – national and
• The Funding Councils
international
• Potential Students • Overseas sponsors
• Alumni • Jo Public
• Journalists • The Universities
• Dangerous obsessives
4

5. Who’s responsible for providing this
valuable data?
• The Times • Shanghai Jiao Tong
University
• Sunday Times
• QS
• The Guardian
• US News and World Report
• Complete University
Guide • HEFCE
• Others...
• Financial Times
• and, of course,
• THE
Government
5

6. Invaluable information
• For all stakeholders
• Intelligent decision-making
• Better than prejudice
• Reflecting the realities of
the market place
• We have a right to know
• Employers have right to know
• We aren’t stupid 6

7. League tables are a bad thing...
“The silly season that marks the publication of
University league tables is nonsensical and illogical.
As any New Scientist knows, letters into numbers;
quality into quantity won’t go. League tables are
simplistic, divisive and undermine the qualitative
nature of a University’s work”
7

8. But we’ll use them anyway!
“Having said that, I’m not
ashamed to report that we came a
very creditable 79th overall, with
my own department rating a
particularly good score for
research
- and as I remarked to the Dean,
you can’t get much better than
that.” 8

9. Dangerous...
• Criteria used do not reflect quality of education
• Historical data
• Variation over time
• Scores are institutional averages – mask strengths
• Distorting effect of weightings and scalings and
data manipulation
• Many of criteria used are inter-related
(ie not independent)
9

10. ... extremely dangerous...
• Apples and elephants and
paperclips
• Perverse incentives
• Hugely political
• Open to manipulation
• Delivered by journalists
• Spurious precision
• Serious consequences for
universities, departments,
staff and students 10

11. The Times 2010
1 Oxford 11 Edinburgh
2 Cambridge 12 Exeter
3 Imperial 13 Bath
4 St Andrews 14 Bristol
5 LSE 15 Leicester
6 Durham 16= Loughborough
7 UCL 16= King’s
8 Warwick 18 Sheffield
9 York 19 Southampton
10 Lancaster 20 Nottingham
11

12. Sunday Times 2010
1 Oxford 11 Bristol
2 Cambridge 12 Nottingham
3 Imperial 13 York
4 UCL 14 Edinburgh
5 LSE 15 Sheffield
6 Durham 16 Loughborough
7 St Andrews 17 Exeter
8 Warwick 18= Southampton
9 Bath 18= Birmingham
10 King’s 20 Newcastle
12

13. The Guardian 2010
1 Oxford 11 SOAS
2 Cambridge 12 Leicester
3 Warwick 13 Bath
4 St Andrews 14 Exeter
5 UCL 15= Sussex
6 Lancaster 15= Edinburgh
7 Imperial 17 Durham
8 LSE 18 Southampton
9= Loughborough 19 UEA
9= York 20 Surrey
13

14. Complete University Guide 2011
1 Cambridge 11 Bristol
2 Oxford 12 York
3 Imperial 13 Edinburgh
4 LSE 14 Southampton
5 Durham 15 Exeter
6 St Andrews 16 King’s
7 UCL 17 Nottingham
8 Warwick 18 SOAS
9 Lancaster 19= Loughborough
10 Bath 19= Sussex
Tuesday, April 19, 2011 14

15. And a completely different approach:
UEL: TQ in the South East 2005
Surplus of good practice over recommendations (in QAA audit reports)
1 East London 4
2 King’s College London 2
3 Brunel 2
4 Queen Mary, London 0
5 Kent 0
6 Hertfordshire 0
7 Royal Holloway -1
8 London South Bank -3
9 Greenwich -6
10 Essex -6
11 Anglia Polytechnic -9
15

16. International value-added
• HE is now a global business
• Global branding assisted by competitive
ranking
• International benchmarking increasingly
important, especially in research
• Student recruitment is increasingly
international...
• ...mobile students are increasingly
choosy
• It’s all good healthy fun 16

17. Just as dangerous...
• Again, the criteria used do not reflect
quality of education
• Archaic and irrelevant data
• Major biases to large, English-
speaking, research- and science-
intensive universities
• Institutional scores are extraordinarily
broad brush
• Distorting effect of weightings,
scalings and data manipulation
• Far from comprehensive surveys of
peers and employers
17

18. Times Higher Education World Rankings 2010
6= Cambridge 79= Sussex
6= Oxford 81= York
9 Imperial College 85 Durham
22 University College 86 LSE
London 87 Manchester
40 Edinburgh 88 Royal Holloway
68= Bristol 90= Southampton
77 King’s
18

19. QS World Rankings 2010
1 Cambridge (2) 69 Sheffield (82)
4 UCL (4) 73 Nottingham (91)
6 Oxford (5=) 77 Glasgow (79)
7 Imperial (5=) 80 LSE (67=)
21 King’s (23) 81 Southampton (95=)
22 Edinburgh (20=) 85 Leeds (99)
27 Bristol (34) 88 York (70=)
30 Manchester (26) 92 Durham (103=)
53 Warwick (58) 95 St Andrews (87=)
59 Birmingham (66)
19

20. Shanghai Jiao Tong 2010
5 Cambridge (4) 63 King’s (65)
10 Oxford (10) 66 Bristol (61)
21 UCL (21) 84 Nottingham (79)
26 Imperial (26) 88 Sheffield (81)
44 Manchester (41) 99 Birmingham (94)
54 Edinburgh (53)
20

21. ‘High Impact Universities’ 2010
1 Harvard University 11 Columbia University
2 Stanford University 12 University of Minnesota
3 MIT 13 University of Cambridge
4 University of California, 14 University of Toronto
Los Angeles
15 University of Chicago
5 University of California,
Berkeley 16 Cornell University
6 University of Michigan 17 University of Oxford
7 University of Washington 18 University of Wisconsin,
8 University of Pennsylvania Madison
9 Johns Hopkins University 19 Yale University
10 University of California, 20 Pennsylvania State
San Diego
21

22. Economist FT MBA Rankings 2010
15 Cranfield 67 Ashridge
17 Henley 68 Birmingham
19 London Business School 71 Oxford – Said
29 Bath 74 Aston
30 Cambridge - Judge 75 Edinburgh
34 Warwick 77 Strathclyde
55 Durham 89 Lancaster
61 Manchester 97 Leeds
22

23. Another approach:
Ecole des Mines de Paris Rankings
1 Harvard 60= Cambridge
10 Ecole des Mines de Paris 89= Heriot-Watt
11 Oxford 89= Bath
26 Manchester 89= Huddersfield
89= Sheffield
35= Queen’s
89= Uni of Wales:
35= Glasgow
Swansea
(and over 100 others…)
23

24. Which league table?
1 University of California, Berkeley 11 Bangor University
2 University of Nottingham 12 University of Sussex
3 York University 13 Escuela Superior Politécnica del
Litoral
4 Northeastern University
14 University of Kent
5 Cornell University
15 Universitas Indonesia
6 Universiti Putra Malaysia
16 Universidad de Alcalá
7 Washington University In St. Louis 17 King Mongkut´s University of
8 Georgia Institute of Technology Technology Thonburi
9 University of Wisconsin-Madison 18 Institut Teknologi Sepuluh November
10 University of North Carolina, 19 Hokkaido University
Chapel Hill 20 National University of Singapore
24

25. Which league table?
1 Greenwich
2 Sheffield Hallam
3 Kingston
4 Westminster
5 East London
6 Central Lancashire
7 Leeds Met
8 Wolverhampton
9 Coventry
10 Middlesex
25

26. Which league table?
1 Loughborough
2 Sheffield
3 UEA
4 Cambridge
5 Dundee
6 Oxford
7 Glasgow
8 Leeds
9 Aberystwyth
10 Southampton
26

27. Which league table?
1 Webb Institute 70.9
2 Carleton College 61.3
3 Princeton University 60.3
4 Middlebury 60.1
5 Amherst 59.5
6 Williams College 57.6
7 Centre College 56.7
8 Indiana Institute of Technology 55.1
9 Davidson College 54.9
10 Thomas Aquinas College 52.5
27

28. Which league table?
1 Massachusetts Institute of Technology
2 Stanford University
3 Harvard University
4 University of California Berkeley
5 Cornell University
6 University of Michigan
7 California Institute of Technology
8 University of Minnesota
9 University of Illinois Urbana Champaign
10 University of Texas Austin
28

29. Which league table?
1 University of Cambridge, World Rank: 28
2 Swiss Federal Institute of Technology WR: 40
3 University of Oxford WR: 42
4 University of Helsinki WR: 58
5 University of Oslo WR: 60
6 University of Edinburgh WR: 65
7 University College London WR: 68
8 Utrecht University WR: 71
9 University of Manchester WR: 82
10 Royal Institute of Technology (Sweden) WR: 84
29

30. International approaches 1
Shanghai Jiao Tong: Ranking of World Universities
– 6 indicators covering: quality of education;
quality of faculty; research output;
performance relative to size
– Includes: Nobel and Fields winners among
alumni and staff; highly cited researchers;
articles in Nature and Science; articles in
citation indices.
30

31. International approaches 2
31

32. Other tables
• ALOHA, PISA, OECD
• China – number of billionaires
• Just research performance: ranking of
scientific papers
• Most conservative, best parties, most interns
• European initiative – U-Multirank
• Islamic universities
32

33. An indicator too far
(not covered in the UK tables - yet)
• Alumni giving
• Academic staff pay
• Research income
• Citations
• Brand impact
But international tables will increasingly influence
methodologies of UK tables
33

34. What is to be done
• Publish prolifically and get top RAE grades
• Be Highly Cited
• Get articles in Nature and Science
• Win a Nobel Prize or a Fields Medal
• Make sure your students all get top jobs with big multinationals (and
then win Nobels)
• Plug the institution relentlessly and cultivate peers, Headmasters/
mistresses and employers
• Improve NSS results (and SSRs) - every little helps
• Spend more on Library and IT and everything else to do with teaching
and learning
• Recruit more international students and staff (all with Nobel potential)
34

35. Conclusions
• They aren’t going to go away
• The international dimension will become increasingly
significant
• Methodologies - for both national and international tables
- are all dubious, at best
• They can and will be used by many different groups – but
can be dangerous in the wrong hands
• Handle with great care!
35

36. Finally
For a copy of the presentation and further league table
commentary and observations, visit:
http://registrarism.wordpress.com/
Also on Twitter: @registrarism
36