AUA 2007 League Tables


Published on

"League Tables: valuable market information or dangerous nonsense" - presentation by Paul Greatrix and Tony Rich at AUA conference 2007 held at the University of Nottingham

Published in: Business, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • AUA 2007 League Tables

    1. 1. League Tables: valuable market information or dangerous nonsense? Dr Paul Greatrix, Registrar, The University of Nottingham Dr Tony Rich, Registrar and Secretary, University of Essex
    2. 2. League Tables <ul><li>Background </li></ul><ul><li>Who wants to know? </li></ul><ul><li>Who are the providers of this valuable information? </li></ul><ul><li>Mad, bad and dangerous </li></ul><ul><li>Doing it yourself </li></ul><ul><li>Measuring the unmeasurable? </li></ul>
    3. 3. Background <ul><li>The US experience </li></ul><ul><li>Regulatory interest </li></ul><ul><li>The Times (1992) </li></ul><ul><li>They sell papers... </li></ul><ul><li>...and, some would suggest, create perverse incentives </li></ul>
    4. 4. Who wants to know? <ul><li>The Government </li></ul><ul><li>The State </li></ul><ul><li>The Funding Councils </li></ul><ul><li>Potential Students </li></ul><ul><li>Alumni </li></ul><ul><li>Parents, teachers, advisors </li></ul><ul><li>Employers </li></ul><ul><li>Overseas sponsors </li></ul><ul><li>Jo Public </li></ul><ul><li>The Universities </li></ul>
    5. 5. Who’s responsible for providing this valuable data? <ul><li>The Times </li></ul><ul><li>Sunday Times </li></ul><ul><li>Financial Times </li></ul><ul><li>The Higher </li></ul><ul><li>Daily Telegraph </li></ul><ul><li>The Guardian </li></ul><ul><li>Shanghai Jiao Tong University </li></ul><ul><li>HEFCE </li></ul><ul><li>BBC Online </li></ul><ul><li>others... </li></ul><ul><li>and, of course, Government </li></ul>
    6. 6. Invaluable information <ul><li>For all stakeholders </li></ul><ul><li>Intelligent decision-making </li></ul><ul><li>Better than prejudice </li></ul><ul><li>Reflecting the realities of the market place </li></ul><ul><li>We have a right to know </li></ul><ul><li>We aren’t stupid </li></ul>
    7. 7. League tables are a bad thing... “ The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”
    8. 8. But we’ll use them anyway! “ Having said that, I’m not ashamed to report that we came a very creditable 79 th overall, with my own department rating a particularly good score for research - and as I remarked to the Dean, you can’t get much better than that.”
    9. 9. Dangerous... <ul><li>Criteria used do not reflect quality of education </li></ul><ul><li>Historical data </li></ul><ul><li>Variation over time </li></ul><ul><li>Scores are institutional averages – mask strengths </li></ul><ul><li>Distorting effect of weightings and scalings and data manipulation </li></ul><ul><li>Many of criteria used are inter-related (ie not independent) </li></ul>
    10. 10. ... extremely dangerous... <ul><li>Apples and elephants and paperclips </li></ul><ul><li>Perverse incentives </li></ul><ul><li>Hugely political – Government interest </li></ul><ul><li>Open to manipulation </li></ul><ul><li>Spurious precision – there’s no such thing as a good league table </li></ul><ul><li>Serious consequences for universities, departments, staff and students </li></ul>
    11. 11. The Times 2006 <ul><li>1 Oxford </li></ul><ul><li>2 Cambridge </li></ul><ul><li>3 Imperial </li></ul><ul><li>4 LSE </li></ul><ul><li>5 UCL </li></ul><ul><li>6 Loughborough </li></ul><ul><li>7 Bristol </li></ul><ul><li>8 Warwick </li></ul><ul><li>9 Bath </li></ul><ul><li>10 Durham </li></ul>11 Edinburgh 12 Royal Holloway 13 Aston 14 Nottingham 15 York 16 Cardiff 17= UEA 17= King’s 19= SOAS, Leicester, St Andrews
    12. 12. Sunday Times 2006 <ul><li>1 Cambridge </li></ul><ul><li>2 Oxford </li></ul><ul><li>3 LSE </li></ul><ul><li>4 Imperial </li></ul><ul><li>5 UCL </li></ul><ul><li>6 Warwick </li></ul><ul><li>7 York </li></ul><ul><li>8 Durham </li></ul><ul><li>9 Bristol </li></ul><ul><li>10 St Andrews </li></ul>11 Bath 12 Nottingham 13 King’s 14 Edinburgh 15 Manchester 16 Southampton 17 Loughborough 18 Exeter 19 UEA 20 Sheffield
    13. 13. The Guardian 2006 <ul><li>1 Cambridge </li></ul><ul><li>2 Oxford </li></ul><ul><li>3 LSE </li></ul><ul><li>4 UCL </li></ul><ul><li>5 Imperial </li></ul><ul><li>6 SOAS </li></ul><ul><li>7 King’s </li></ul><ul><li>8 Warwick </li></ul><ul><li>9 Bath </li></ul><ul><li>10 Edinburgh </li></ul>11 Nottingham 12 Surrey 13 Bristol 14 Manchester 15 York 16 Birmingham 17 Leeds 18 Goldsmiths 19 Aston 20 Royal Holloway
    14. 14. Financial Times 2003 <ul><li>1 Cambridge </li></ul><ul><li>2 Oxford </li></ul><ul><li>3 Imperial </li></ul><ul><li>4 LSE </li></ul><ul><li>5 UCL </li></ul><ul><li>6 Warwick </li></ul><ul><li>7 Bristol </li></ul><ul><li>8 York </li></ul><ul><li>9 Nottingham </li></ul><ul><li>10 King’s </li></ul>11 UMIST 12 Bath 13 Loughborough 14 Edinburgh 15 Manchester 16 Birmingham 17 SOAS 18 Leicester 19 St Andrews 20 Southampton
    15. 15. ‘ Poll of polls’ (Daily Telegraph 2003) <ul><li>1 Cambridge </li></ul><ul><li>2 Oxford </li></ul><ul><li>3 Imperial </li></ul><ul><li>4 LSE </li></ul><ul><li>5 Warwick </li></ul><ul><li>6= Nottingham </li></ul><ul><li>6= UCL </li></ul><ul><li>8 York </li></ul><ul><li>9 Bristol </li></ul><ul><li>10 Manchester </li></ul>11 SOAS 12 Bath 13 Birmingham 14 Loughborough 15 King’s 16 = Durham 16= St Andrews 18= Southampton 18= Edinburgh 20 Sheffield
    16. 16. Times Higher World Rankings 2006 <ul><li>2 Cambridge (3) </li></ul><ul><li>3 Oxford (4) </li></ul><ul><li>Imperial (13) </li></ul><ul><li>17 LSE (11=) </li></ul><ul><li>25 UCL (28) </li></ul><ul><li>33= Edinburgh (30) </li></ul><ul><li>Manchester (35) </li></ul><ul><li>46= King’s (73=) </li></ul><ul><li>64= Bristol (49) </li></ul><ul><li>70= SOAS (103=) </li></ul><ul><li>Warwick (77=) </li></ul><ul><li>81 Glasgow (101=) </li></ul><ul><li>90= Birmingham (143) </li></ul><ul><li>85 Nottingham (97) </li></ul><ul><li>99= Queen Mary (112) </li></ul>
    17. 17. Shanghai Jiao Tong 2006 <ul><li>2 Cambridge (2) </li></ul><ul><li>10 Oxford (10) </li></ul><ul><li>23 Imperial (23) </li></ul><ul><li>26 UCL (26) </li></ul><ul><li>Manchester (53) </li></ul><ul><li>52 Edinburgh (47) </li></ul><ul><li>62 Bristol (64) </li></ul><ul><li>69 Sheffield (65) </li></ul><ul><li>Nottingham (83) </li></ul><ul><li>83 King’s (80) </li></ul><ul><li>90 Birmingham (98) </li></ul>
    18. 18. And a completely different approach: UEL: TQ in the South East 2005 Surplus of good practice over recommendations (in QAA audit reports) 1 East London 4 2 King's College London 2 3 Brunel 2 4 Queen Mary, London 0 5 Kent 0 6 Hertfordshire 0 7 Royal Holloway -1 8 London South Bank -3 9 Greenwich -6 10 Essex -6 11 Anglia Polytechnic -9
    19. 19. Different approaches 1 <ul><li>The Times </li></ul><ul><ul><li>9 indicators, z-scoring used </li></ul></ul><ul><ul><li>Teaching weighted @ 1.5 (now using only NSS) </li></ul></ul><ul><ul><li>Research weighted @ 1.5 </li></ul></ul><ul><li>The Financial Times (if they ever do again) </li></ul><ul><ul><li>15 indicators, z-scoring used </li></ul></ul><ul><ul><li>Weighting of T and R, both 15% </li></ul></ul><ul><li>The Sunday Times </li></ul><ul><ul><li>9 indicators - student satisfaction weighted x1.5 (NSS); teaching excellence x1.0 </li></ul></ul><ul><ul><li>A level scores weighted x2.5 and research weighted x2 </li></ul></ul><ul><ul><li>Also – Heads’ assessment x1 (up from 0.5 in 2005) and drop-out rate (variable) </li></ul></ul>
    20. 20. Different approaches 2 <ul><li>The Guardian </li></ul><ul><ul><li>7 indicators in subject tables (research ratings do not feature, no longer using TQA but not adopted NSS) </li></ul></ul><ul><ul><li>T staff qualifications weighted @ 15% in each subject table </li></ul></ul><ul><ul><li>Entry qualifications and SSR @ 20% </li></ul></ul><ul><ul><li>Spend per student and Value-added @ 10% </li></ul></ul><ul><ul><li>Destinations @ 17% </li></ul></ul><ul><ul><li>‘ inclusiveness’ @ 8% </li></ul></ul><ul><ul><li>Overall table in 2006 is an average of subject tables </li></ul></ul><ul><li>The Daily Telegraph </li></ul><ul><ul><li>just the one indicator (keeps things nice and simple) </li></ul></ul>
    21. 21. International approaches 1 <ul><li>Shanghai Jiao Tong: Ranking of World Universities </li></ul><ul><ul><li>6 indicators covering: quality of education; quality of faculty; research output; performance relative to size </li></ul></ul><ul><ul><li>Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science ; articles in citation indices. </li></ul></ul>
    22. 22. International approaches 2 <ul><li>Times Higher World University Rankings </li></ul><ul><ul><li>Peer review: 40% </li></ul></ul><ul><ul><li>Citations per Faculty member and SSR: each 20% </li></ul></ul><ul><ul><li>Recruiter review: 10% </li></ul></ul><ul><ul><li>Proportions of international Faculty and students: each 5% </li></ul></ul>
    23. 23. The critical factors (by frequency of appearance in the tables) <ul><li>Subject review/teaching quality results/NSS </li></ul><ul><li>A level scores </li></ul><ul><li>Staff:Student Ratios </li></ul><ul><li>Graduate employment rates </li></ul><ul><li>RAE results </li></ul><ul><li>Spending on library and computing </li></ul><ul><li>Degree classifications </li></ul><ul><li>Completion rates </li></ul>
    24. 24. One-offs (only appeared in one UK table) <ul><li>Spending on facilities </li></ul><ul><li>Access </li></ul><ul><li>Value-added </li></ul><ul><li>%age of graduates entering professional training </li></ul><ul><li>%age of graduates entering higher degree </li></ul><ul><li>%age of postgraduate research students </li></ul><ul><li>%age of taught postgraduate students </li></ul><ul><li>Income from industry </li></ul><ul><li>Income from research </li></ul><ul><li>Applications to places </li></ul><ul><li>Inclusiveness </li></ul>
    25. 25. An indicator too far (not covered in the UK tables - yet) <ul><li>Alumni giving </li></ul><ul><li>Academic staff pay </li></ul><ul><li>Percentage of full-time academic staff </li></ul><ul><li>Citations </li></ul><ul><li>Brand impact </li></ul>But international tables will increasingly influence methodologies of UK tables
    26. 26. Conclusions <ul><li>They aren’t going to go away </li></ul><ul><li>The international dimension will become increasingly significant </li></ul><ul><li>They can and will be used by many different groups – but can be dangerous in the wrong hands </li></ul><ul><li>Handle with great care! </li></ul>