Ranking ≠ Comparabilidad, a cargo de Tony Stevenson, vicerrector adjunto de Planificación y Recursos, Universidad de Newcastle. La conferencia se presentó en el 1er Seminario Internacional sobre Rankings en Educación Superior y E-learning organizado por la UOC.

1. Ranking ≠ ComparabilityBarcelona –September 21st2011
Professor Tony Stevenson
Pro Vice Chancellor –Planning and Resources
University of Newcastle

2. Introduction
•
Most ranking systems aim to rank top 50 or top 250 worldwide......
•
But there are at least 15,000 Higher Education Institutions worldwide
•
National Ranking systems –some 50 : e.g. CHE; Times
•
International rankings –some 10: e.g. Jiao-Tong; QS; THES etc; Webometricsetc

3. Thought experiment!
•
After extensive consultation we invented a rating system for restaurants:
•
4* = World-class
•
3* = Of European renown
•
2* = Of national renown
•
1* = Acceptable. Popular in the city
•
0* = Poor

4. Restaurant qualitypercentage of restaurants submitted in each category
0% 10% 20% 30% 40% 50% 4*3*2*1*0* Someburgh0% 10% 20% 30% 40% 50% 4*3*2*1*0* Otherville0% 10% 20% 30% 40% 50% 4*3*2*1*0* Thistown0% 10% 20% 30% 40% 50% 4*3*2*1*0* Elseport

5. Which City is best?
•
Weighted Grade Point Average (GPA)
–
4:3:2:1 as advocated by Gastronomy Times or 16:9:4:1 as advocated by Restaurant Fortnight
•
Medals tables -only 4*; 4*+3*; all medals as advocated by Association for Haute Cuisine
Rank order
Gastronomy Times:ESTO
Restaurant Fortnight:TEOS
AHC Gold medal table:TOSE
AHC Gold and Silver medal table:T(EO)S
AHC all medal table:ET (OS)

6. Which City is best?
Power –Tourist Board decided that its the number of restaurants submitted that is important
–
Someburghsubmitted 40
–
Othervillesubmitted 30
–
Thistownsubmitted 20
–
Elseportsubmitted 20
–
Impact of Power on the rankings –Multiply the score by the number of restaurants submitted for each city
Power Rank order
Gastronomy Times:SOET
Restaurant Fortnight:SOTE
AHC Gold medal table:(OT)SE
AHC Gold and Silver medal table:(OS)TE
AHC all medal table:SOET

7. Which City is best?
BUT: the Chamber of Commerce point out that cities may have played a game and only submitted what they thought were good restaurants
They discover that their database reveals that:
•
Someburghsubmitted 40 out of 64 restaurants in total
•
Othervillesubmitted 30 out of 33
•
Thistownsubmitted 20 out of 40
•
Elseportsubmitted 20 out of 30
So you can adjust game playing by altering (downweighting) the power scores by the proportion submitted
Ratio Rank order
Gastronomy Times:OEST
Restaurant Fortnight:OETS
AHC Gold medal table:OTSE
AHC Gold and Silver medal table:O(ET)S
AHC all medal table:OEST

8. Which is right?
Rank orderPower rankRatio Rank
Gastronomy Times:ESTOSOETOEST
Restaurant Fortnight:TEOSSOTEOETS
AHC Gold medal table:TOSE(OT)SE OTSE
AHC Gold and Silver medal table:T(EO)S(OS)TE O(ET)S
AHC all medal table:ET(OS)SOETOEST

9. Comparability
05101520254*3*2*1*0*NSSomeburgh05101520254*3*2*1*0*NSOtherville05101520254*3*2*1*0*NSThistown05101520254*3*2*1*0*NSElseport

10. •
Any resemblance of this to the UK’s Research assessment exercise is entirely coincidental

14. Conclusions
•
Difference does not necessarily reflect ordering
•
The interests of the ranker affect the results as much as any intrinsic qualities of those being ranked
•
Ranking systems create reality –constrain University Missions
•
Is a University Excellent –how measured; universities are differentiated in their missions

15. Conclusions ctd
•
Diversity replaced by Hierarchy –networks of power
•
Is a University Good or Good at what it does? -How Measured
•
Simple graphical representations can be richer than a linear scale to compare quality
•
Rankings are here to stay –but lets have lots of them