Successfully reported this slideshow.

Aua 2008 A


Published on

aua 08

Published in: Education, Sports
  • Be the first to comment

  • Be the first to like this

Aua 2008 A

  1. 1. International League Tables: a new kind of game or more of the same? Dr Paul Greatrix, Registrar, The University of Nottingham Dr Tony Rich, Registrar and Secretary, University of Essex
  2. 2. League Tables <ul><li>Background </li></ul><ul><li>Who wants to know? </li></ul><ul><li>Who are the providers of this valuable information? </li></ul><ul><li>Mad, bad and dangerous </li></ul><ul><li>National and international dimensions </li></ul><ul><li>Doing it yourself </li></ul><ul><li>Measuring the unmeasurable? </li></ul>
  3. 3. Background <ul><li>The US experience </li></ul><ul><li>Regulatory interest </li></ul><ul><li>In the UK: The Times (1992) </li></ul><ul><li>International dimensions </li></ul><ul><li>They sell papers... </li></ul><ul><li>...and, some would suggest, create perverse incentives </li></ul>
  4. 4. Who wants to know? <ul><li>The Government </li></ul><ul><li>The State </li></ul><ul><li>The Funding Councils </li></ul><ul><li>Potential Students </li></ul><ul><li>Alumni </li></ul><ul><li>Journalists </li></ul><ul><li>Parents, teachers, advisors </li></ul><ul><li>Employers - national and international </li></ul><ul><li>Overseas sponsors </li></ul><ul><li>Jo Public </li></ul><ul><li>The Universities </li></ul><ul><li>Dangerous obsessives </li></ul>
  5. 5. Who’s responsible for providing this valuable data? <ul><li>The Times </li></ul><ul><li>Sunday Times </li></ul><ul><li>The Guardian </li></ul><ul><li>GUG </li></ul><ul><li>Financial Times </li></ul><ul><li>(Daily Telegraph) </li></ul><ul><li>Shanghai Jiao Tong University </li></ul><ul><li>Times Higher Ed (THE) </li></ul><ul><li>Newsweek </li></ul><ul><li>HEFCE </li></ul><ul><li>others... </li></ul><ul><li>and, of course, Government </li></ul>
  6. 6. Invaluable information <ul><li>For all stakeholders </li></ul><ul><li>Intelligent decision-making </li></ul><ul><li>Better than prejudice </li></ul><ul><li>Reflecting the realities of the market place </li></ul><ul><li>We have a right to know </li></ul><ul><li>Employers have a right to know </li></ul><ul><li>We aren’t stupid </li></ul>
  7. 7. League tables are a bad thing... “ The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”
  8. 8. But we’ll use them anyway! “ Having said that, I’m not ashamed to report that we came a very creditable 79 th overall, with my own department rating a particularly good score for research - and as I remarked to the Dean, you can’t get much better than that.”
  9. 9. Dangerous... <ul><li>Criteria used do not reflect quality of education </li></ul><ul><li>Historical data </li></ul><ul><li>Variation over time </li></ul><ul><li>Scores are institutional averages – mask strengths </li></ul><ul><li>Distorting effect of weightings and scalings and data manipulation </li></ul><ul><li>Many of criteria used are inter-related (ie not independent) </li></ul>
  10. 10. ... extremely dangerous... <ul><li>Apples and elephants and paperclips </li></ul><ul><li>Perverse incentives </li></ul><ul><li>Hugely political – Government interest </li></ul><ul><li>Open to manipulation </li></ul><ul><li>Delivered by journalists </li></ul><ul><li>Spurious precision – there’s no such thing as a good league table </li></ul><ul><li>Serious consequences for universities, departments, staff and students </li></ul>
  11. 11. The Times 2007 <ul><li>1 Oxford </li></ul><ul><li>2 Cambridge </li></ul><ul><li>3 Imperial </li></ul><ul><li>4 LSE </li></ul><ul><li>5 St Andrews </li></ul><ul><li>6 UCL </li></ul><ul><li>7 Warwick </li></ul><ul><li>8 Bristol </li></ul><ul><li>9 Durham </li></ul><ul><li>10 King’s </li></ul>11 Bath 12 Loughborough 13 Edinburgh 14 Southampton 15 Aston 16 York 17 Exeter 18 SOAS 19 Nottingham 20 UEA
  12. 12. Sunday Times 2007 <ul><li>1 Cambridge </li></ul><ul><li>2 Oxford </li></ul><ul><li>3 LSE </li></ul><ul><li>4 Imperial </li></ul><ul><li>5 UCL </li></ul><ul><li>6 St Andrews </li></ul><ul><li>7 Warwick </li></ul><ul><li>8 York </li></ul><ul><li>9 Bath </li></ul><ul><li>10 Bristol </li></ul>11 Durham 12 King’s 13 Loughborough 14 Edinburgh 15 Nottingham 16 Southampton 17 Exeter 18 Lancaster 19= Manchester 19= Leicester
  13. 13. The Guardian 2007 <ul><li>1 Oxford </li></ul><ul><li>2 Cambridge </li></ul><ul><li>3 Imperial </li></ul><ul><li>4 St Andrews </li></ul><ul><li>5 UCL </li></ul><ul><li>6 LSE </li></ul><ul><li>7 Edinburgh </li></ul><ul><li>8 Warwick </li></ul><ul><li>9 Loughborough </li></ul><ul><li>10 Bath </li></ul>11 SOAS 12 King’s 13 Southampton 14 Bristol 15 York 16 Manchester 17 Durham 18 Birmingham 19 Nottingham 20 Leeds
  14. 14. Good University Guide 2007 <ul><li>1 Cambridge </li></ul><ul><li>2 Oxford </li></ul><ul><li>3 Imperial </li></ul><ul><li>4 LSE </li></ul><ul><li>5 St Andrews </li></ul><ul><li>6 UCL </li></ul><ul><li>7 Bristol </li></ul><ul><li>8 Warwick </li></ul><ul><li>9 Bath </li></ul><ul><li>10 Durham </li></ul><ul><li>11 Loughborough </li></ul>12 Aston 13 Royal Holloway 14= Nottingham 14= York 16 Edinburgh 17= King’s 17= Exeter 19 Lancaster 20= UEA 20= Leicester 20= Southampton
  15. 15. And a completely different approach: UEL: TQ in the South East 2005 Surplus of good practice over recommendations (in QAA audit reports) 1 East London 4 2 King's College London 2 3 Brunel 2 4 Queen Mary, London 0 5 Kent 0 6 Hertfordshire 0 7 Royal Holloway -1 8 London South Bank -3 9 Greenwich -6 10 Essex -6 11 Anglia Polytechnic -9
  16. 16. International value-added <ul><li>Higher Education is now a global business </li></ul><ul><li>Global branding assisted by competitive ranking </li></ul><ul><li>International benchmarking increasingly important, especially in research </li></ul><ul><li>Student recruitment is increasingly international… </li></ul><ul><li>… mobile students are increasingly choosy </li></ul><ul><li>It’s all good healthy fun </li></ul>
  17. 17. Just as dangerous... <ul><li>Again, the criteria used do not reflect quality of education </li></ul><ul><li>Archaic and irrelevant data </li></ul><ul><li>Major biases towards large, English-speaking, research-intensive and science-focused universities </li></ul><ul><li>Institutional scores are extraordinarily broad brush </li></ul><ul><li>Distorting effect of weightings, scalings and data manipulation </li></ul><ul><li>Far from comprehensive surveys of peers and employers </li></ul>
  18. 18. Times Higher World Rankings 2007 <ul><li>2= Cambridge (2) </li></ul><ul><li>3= Oxford (3) </li></ul><ul><li>5 Imperial College (13) </li></ul><ul><li>9 University College London (25) </li></ul><ul><li>23 Edinburgh (33) </li></ul><ul><li>24 King’s London (46) </li></ul><ul><li>30 Manchester (40) </li></ul><ul><li>37 Bristol (64) </li></ul><ul><li>57 Warwick (73) </li></ul><ul><li>59 London School of Economics (17) </li></ul>65 Birmingham (90) 68 Sheffield (102) 70 Nottingham (85) 74 York (124) 76 St Andrews (109) 80= Leeds (121) 80= Southampton (141) 83 Glasgow (81) 99 Cardiff (141)
  19. 19. Shanghai Jiao Tong 2007 <ul><li>4 Cambridge (2) </li></ul><ul><li>10 Oxford (10) </li></ul><ul><li>23 Imperial (23) </li></ul><ul><li>25 UCL (26) </li></ul><ul><li>48 Manchester (50) </li></ul><ul><li>53 Edinburgh (52) </li></ul>62 Bristol (62) 72 Sheffield (69) 81 Nottingham (79) 83 King’s (83) 92 Birmingham (90)
  20. 20. Shanghai Jiao Tong 2007: Regional Analysis 5 2 1 Africas 100 64 42 24 9 1 Asia/ Pacific 208 172 123 80 34 2 Europe 197 164 138 98 58 17 North/ Latin America Top 500 Top 400 Top 300 Top 200 Top 100 Top 20 Region
  21. 21. Economist 2007 Global MBA Rankings <ul><li>7 Cambridge: Judge </li></ul><ul><li>10 Henley </li></ul><ul><li>11 Cranfield </li></ul><ul><li>15 London Business School </li></ul><ul><li>19 Ashridge </li></ul><ul><li>27 Warwick </li></ul><ul><li>31 Oxford: Said </li></ul><ul><li>38 City: Cass </li></ul><ul><li>50 Lancaster </li></ul><ul><li>52 Leeds </li></ul><ul><li>54 Aston </li></ul><ul><li>Edinburgh </li></ul><ul><li>Manchester </li></ul><ul><li>Imperial: Tanaka </li></ul><ul><li>59 Durham </li></ul><ul><li>62 Nottingham </li></ul><ul><li>63 Birmingham </li></ul><ul><li>64 Strathclyde </li></ul><ul><li>65 Bath </li></ul><ul><li>75 Glasgow </li></ul><ul><li>88 Sheffield </li></ul><ul><li>93 Bradford </li></ul><ul><li>98 Newcastle </li></ul>
  22. 22. Financial Times 2008 Global MBA Rankings <ul><li>2 London Business School </li></ul><ul><li>10 Cambridge: Judge </li></ul><ul><li>19 Oxford: Said </li></ul><ul><li>22= Lancaster </li></ul><ul><li> Manchester </li></ul><ul><li>29 Warwick </li></ul><ul><li>30= Cranfield </li></ul><ul><li>Strathclyde </li></ul>35 Imperial: Tanaka 41 City: Cass 44 Edinburgh 48= Leeds 53 Bradford/TiasNimbas 69 Bath 76 Nottingham
  23. 23. Another approach: Ecole des Mines de Paris Rankings <ul><li>1 Harvard </li></ul><ul><li>10 Ecole des Mines de Paris </li></ul><ul><li>11 Oxford </li></ul><ul><li>26 Manchester </li></ul><ul><li>35= Queen’s </li></ul><ul><li>35= Glasgow </li></ul>60= Cambridge 89= Heriot-Watt 89= Bath 89= Huddersfield 89= Sheffield 89= Uni of Wales: Swansea (and over 100 others…)
  24. 24. Do it yourself Your chance to show that you have a better understanding than the press of what really matters in HE
  25. 25. Do it yourself All you have to do is use your skill and judgement to determine the most sensible weighting which should be applied to the criteria you select
  26. 26. UK approaches <ul><li>The Times </li></ul><ul><ul><li>8 indicators, z-scoring used </li></ul></ul><ul><ul><li>Teaching weighted @ 1.5 (now using only NSS) </li></ul></ul><ul><ul><li>Research weighted @ 1.5 </li></ul></ul><ul><li>The Sunday Times </li></ul><ul><ul><li>9 indicators - student satisfaction weighted x1.5 (NSS); teaching excellence x1.0 </li></ul></ul><ul><ul><li>A level scores weighted x2.5 and research weighted x2 </li></ul></ul><ul><ul><li>Also – Head Teachers’ assessment and drop-out rate (variable) </li></ul></ul>
  27. 27. UK approaches 2 <ul><li>The Guardian </li></ul><ul><ul><li>7 indicators in subject tables (research ratings do not feature) </li></ul></ul><ul><ul><li>Teaching quality (10%) and Student Feedback (5%) in each subject table - both derived from NSS </li></ul></ul><ul><ul><li>Entry qualifications, SSR, Spend per student, job prospects, Value-added @ 17% </li></ul></ul><ul><ul><li>Overall table in 2007 is an average of subject tables </li></ul></ul>
  28. 28. International approaches 1 <ul><li>Shanghai Jiao Tong: Ranking of World Universities </li></ul><ul><ul><li>6 indicators covering: quality of education; quality of faculty; research output; performance relative to size </li></ul></ul><ul><ul><li>Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science ; articles in citation indices. </li></ul></ul>
  29. 29. International approaches 2 <ul><li>Times Higher World University Rankings </li></ul><ul><ul><li>Peer review: 40% </li></ul></ul><ul><ul><li>Citations per Faculty member and SSR: each 20% </li></ul></ul><ul><ul><li>Recruiter review: 10% </li></ul></ul><ul><ul><li>Proportions of international Faculty and students: each 5% </li></ul></ul>
  30. 30. The critical factors (by frequency of appearance in the tables) <ul><li>Teaching evaluation: NSS </li></ul><ul><li>A level scores </li></ul><ul><li>Staff:Student Ratios </li></ul><ul><li>Graduate employment rates </li></ul><ul><li>RAE results </li></ul><ul><li>Spending on library and computing </li></ul><ul><li>Degree classifications </li></ul><ul><li>Completion rates </li></ul>
  31. 31. An indicator too far (not covered in the UK tables - yet) <ul><li>Alumni giving </li></ul><ul><li>Academic staff pay </li></ul><ul><li>Research income </li></ul><ul><li>Citations </li></ul><ul><li>Brand impact </li></ul>But international tables will increasingly influence methodologies of UK tables
  32. 32. What is to be done <ul><li>Publish prolifically and get top RAE grades </li></ul><ul><li>Be Highly Cited </li></ul><ul><li>Get articles in Nature and Science </li></ul><ul><li>Win a Nobel Prize or a Fields Medal </li></ul><ul><li>Make sure your students all get top jobs with big multinationals (and then win Nobels) </li></ul><ul><li>Plug the institution relentlessly and cultivate peers, Headmasters/mistresses and employers </li></ul><ul><li>Improve NSS results (and SSRs) - every little helps </li></ul><ul><li>Spend more on Library and IT and everything else to do with teaching and learning </li></ul><ul><li>Recruit more international students and staff (all with Nobel potential) </li></ul>
  33. 33. Conclusions <ul><li>They aren’t going to go away </li></ul><ul><li>The international dimension will become increasingly significant </li></ul><ul><li>Methodologies - for both national and international tables - are all dubious, at best </li></ul><ul><li>They can and will be used by many different groups – but can be dangerous in the wrong hands </li></ul><ul><li>Handle with great care! </li></ul>
  34. 34. Finally For a copy of the presentation and further league table commentary and observations, visit: