The document discusses changes to methodologies used by major global university rankings over time. It summarizes changes made by QS (Quacquarelli Symonds) in 2005, 2007, 2011, and 2015 related to data sources, calculation of indicator scores, and measures to reduce volatility. These included adding employer surveys, changing FTE counts and z-scores, and excluding papers with many authors from citation counts. The document also summarizes changes made by Shanghai ARWU in 2004, 2013, and 2014-15 related to research indicators and highly cited researchers. It notes that rankings changes have generally helped universities with social science strengths. The author poses questions about current QS and THE methodologies and whether further changes are planned.

1. GLOBAL UNIVERSITY
RANKINGS: CHANGING
METHODOLOGIES
Richard Holmes
University Ranking Watch
Ekaterinburg, Russian Federation

2. INTRODUCTION
 All major global rankings, to varying degrees, have
introduced changes in their methodology
Changes include
 Source of data
 Calculation of indicator scores
 Measures to reduce fluctuation
 Measures to eliminate outliers and anomalies

3. REVIEW OF CHANGES: QS
 2004 Times Higher Education Supplement (THES) –
Quacquarelli Symonds World University Rankings
 2005 added employer survey with a 10% weighting
 A cluster of changes in 2007 – Full time equivalent
instead of head counts, z-scores for each indicator, no
voting for own university in the survey, change to
data from Scopus.

4. QS CONTINUED
 Continued development of academic and employer
survey and procedures against manipulation

5. QS CONTINUED
Second cluster of changes in 2015
 normalisation – five subject groups treated as
contributing equally to citations indicator
 Roll-over period for survey responses increased from
3 to 5 years
 Papers with “freakish” numbers of authors not
counted for citations

6. EFFECTS OF QS 2015 CHANGES
 Surveys more stable
 Universities strong in physics and medicine suffered
while those strong in social sciences and engineering
benefitted

7. THE CHANGES 2011
 International research collaboration added
 International students and international faculty given
equal weighting
 Public research income as % of research income
dropped
 Reduced weighting for citations, counted for 6 years
 Field normalization applied to a further 3 research
indicators

8. CONTINUED
 The main result of these changes was to help
universities with strengths in the social sciences
 Also reduced the anomaly of Alexandria University as
4th best university in the world for research impact

9. THE CHANGES 2015
 Switch from Thomson Reuters to Scopus as a source
of data
 Expansion of academic survey
 Regional modification applied to half of citations
indicator
 Excluding papers with many authors from citation
counts

10. CONTINUED
 Universities that participated in multi-contributor
physics projects and benefitted from the regional
modification now suffered
 Including universities in France, Korea, Japan and
Turkey
 Oxford and Cambridge overtook Harvard
 Small and specialized institutions did well

11. OTHER MAJOR RANKINGS:
SHANGHAI ARWU
 2004 helped social science institutions by not
counting Nature and Science publications, and giving
a double score to social science publications
 2013 highly cited researchers with secondary
affiliations based on estimates from researchers
 2014-15 used old and new lists – secondary
affiliations not counted

12. US NEWS BEST GLOBAL
UNIVERSITIES
 2015 introduced new indicators – books and
indicators

13. QUESTIONS FOR QS
 Do you think that not counting multi-contributor
papers for citations is a good idea? Have you
considered fractional counting?
 Are you satisfied with the moderate degree of field
normalization introduced in 2015?
 Do you consider a 40% weighting for the academic
survey acceptable?
 And … do you plan on any changes?

14. QUESTIONS FOR THE
 Do you think that not counting multi-contributor
papers for citations is a good idea? Have you
considered fractional counting?
 Do you consider 30% an acceptable weighting for the
citations indicator?
 Can you explain how Oxford and Cambridge have
surpassed Harvard and MIT?
 And … do you plan on any more changes?