Paper presented in Moscow April 2015

1. Richard Holmes
Moscow 10th April 2015
THE QS AND TIMES HIGHER EDUCATION
[THE] RANKINGS BY SUBJECT COMPARED

2. BRIEF HISTORY OF THE QS AND TIMES HIGHER
EDUCATION WORLD RANKINGS
• 2004 Times Higher Education Supplement teamed up
with QS Quacquarelli Symonds Ltd to produce World
University Rankings
• 2005 and 2007 methodological changes
• 2009 THE announced they were changing partners and
THE and QS both produced world rankings

3. QS WORLD RANKINGS
• Structure
• 40% academic survey
• 10% employer survey
• 20% citations per faculty
• 20% faculty student ratio
• 10% international faculty and students

4. TIMES HIGHER EDUCATION WORLD RANKINGS
• First edition 2010
• Methodological changes in 2011
• 2014 announced that it was changing partners
again and would use bibliometric data from
Scopus and collect institutional data itself

5. THE WORLD RANKINGS
Structure
• 30% teaching – 5 indicators including reputation
survey
• 30% research (18%) -- 3 indicators including
reputation survey
• 7.5% international orientation – 3 indicators
• 2.5% industry income
• 30% citations

6. COMPARISON OF WORLD RANKINGS
Academic reputation survey
• QS 40% weighting – derived from 5 sources –
subject to restrictions and verification
• Open to influence by universities – submission
of names, notification about sign up facility
• Bias to UK, Asia and Latin America

7. CONTINUED
• THE – 33% -- divided into 2 questions –
respondents from TR lists – this year from
Scopus
• Fewer restrictions than QS
• Limited scope for influence by universities
• Results more plausible than QS

8. GRADUATE EMPLOYABILITY: EMPLOYER SURVEY
• QS 10% but not THE
• Proxy for graduate employability?
• Six channels for respondents
• High scores for Asian and Latin American
Universities
• Open to influence by universities

9. INTERNATIONAL INDICATORS
• QS – 10% international students, international
faculty
• THE -- 7.5% international students, international
faculty, international research collaboration

10. TEACHING INDICATORS
• QS – 20% faculty student ratio [includes
research only staff]
• THE – 30% -- 5 indicators – reputation, income,
staff student ratio, ratio doctoral to bachelor
degrees, doctoral degrees per academic staff

11. RESEARCH INDICATORS
• QS – no measurement other than reputation
survey and citations
• THE – 30% -- Research includes publication,
research income, research reputation

12. RELATIONS WITH INDUSTRY
• QS – not measured
• THE – 2.5% -- income from industry and
commerce per academic staff

13. CITATIONS
• QS – 20% citations per faculty, data from
Scopus
• THE – 30% citations per paper, normalised for
field and year, regional modifcation, until now
data from TR

14. COMPARISON OF WORLD UNIVERSITY
RANKINGS
• Both QS and THE use reputation survey data,
institutional data and bibliometric data
• Both measure faculty study ratio,
internationalisation and citations
• QS measures graduate employability through an
employer survey – THE does not
• THE measures income, publications and
research collaboration – QS does not

15. DISTINCTIVE FEATURES
• QS – academic survey – justified as including
arts and humanities – “a methodological black
hole” – can be influenced by institutions
• -- implausible results – high scores for some
Asian and Latin American universities
• Quite volatile

16. DISTINCTIVE FEATURES
• THE – citations: research impact
• Year and field normalised
• Replaces bias towards medicine with a bias
towards physics
• Regional modification
• Produces fantastic results

17. SUBJECT RANKINGS: QS
• QS provides subject group rankings (5) and
subjects (30)
• For both, QS use 2 indicators from the world
ranking --, academic and employer survey – and
2 additional ones – h-index and citations per
paper in varying combinations

18. CONTINUED
• QS does not use institutional data in its subject
group or subject rankings
• (but there has been talk of field normalisation)
• an obvious reform would be to ask for data
about staff and student numbers

19. CONTINUED
• This means the QS subject group and subject
rankings are more research and research impact
orientated
• Weighting for academic survey ranges from 40
to 60%
• Employer survey from 10 to 30%
• Citations from 10 to 25%

20. CONTINUED
• H-index from 10 to 15%
• Major criticism is that QS subject rankings rely
on a few data points – handful of nominations in
the surveys, a small number of citations
• At the subject level or national level can be
inaccurate or unreliable

21. RUSSIAN UNIVERSITIES IN QS SUBJECT GROUP
RANKINGS 2014 TOP 400
• Arts and humanities – 2
• Engineering and technology – 2
• Life sciences and medicine – 1
• Natural sciences – 5
• Social sciences and management -- 1

22. QS SUBJECT RANKINGS
• 30 subjects – 4 indicators in varying proportions
• http://www.iu.qs.com/university-rankings/subject-
tables/subject-weighting/

23. PERFORMANCE OF RUSSIAN UNIVERSITIES IN
SUBJECT RANKINGS TOP 400
• Modern languages – 2
• Philosophy -- 1
• CSIS -- 1
• EE engineering -- 1
• MAM engineering -- 1
• Biology -- 1

24. CONTINUED
• Chemistry -- 2
• Earth and marines sciences -- 1
• Environmental sciences -- 1
• Materials science -- 1
• Mathematics -- 3
• Physics and astronomy -- 5

25. CONTINUED
• Statistics and operational research -- 1
• Note QS also supply lists of top 5 universities in
every subject for each country

26. TIMES HIGHER EDUCATION SUBJECT GROUP
RANKINGS
• 6 groups – clinical, pre-clinical and health in its
own group
• Use all the world rankings indicators but with
different weightings
• 2014 – top 100 in each group ranked

27. CONTINUED
• Teaching ranges from 37.5% (arts and
humanities) to 27.5% (clinical, pre-clinical and
health, life sciences, physical sciences)
• Research ranges from 37.5% (arts and
humanities) to 27.5% (clinical, pre-clinical and
health, life sciences, physical sciences)
• International outlook is the same – 7.5%

28. CONTINUED
• Industry income: innovation ranges from 5%
(engineering and technology) to 2.5% (all other
subjects)
• Citations: research impact ranges from 15%
(arts and humanities) to 35% (clinical, pre-
clinical and health, life sciences, physical
sciences)

29. PERFORMANCE OF RUSSIAN UNIVERSITIES 2014
• Engineering and Technology
• 66th Lomonosov Moscow State University
• Physical sciences
• 56th Lomonosov Moscow State University
• 85th Novosibirsk State University
• 95th National Research Nuclear University MEPhI

30. COMPARISON OF QS AND THE
SUBJECT/SUBJECT GROUP RANKINGS
Warnings
• QS has delayed publication of subject rankings –
why?
• THE might expand its repertoire of subject
rankings

31. CONTINUED
• In the subject rankings QS uses two subjective
indicators from the world rankings plus 2 citation
based indicators
• Various combinations of 4 (2 in the case of
English)
• THE uses all 13 indicators (or 5 indicator
groups) in various combinations

32. CONTINUED
• QS subject rankings continue to emphasise the
survey indicators – 100% for English
• THE indicators emphasise citations – as high as
35% for 3 subject groups (in reality even higher
because of the regional modification)

33. CONTINUED
• The structure of the academic survey in the QS
subject rankings favors large “national flagship”
universities
• Lack of normalisation in the citation indicators
gives universities with a medical school an
advantage in the Medicine and life sciences
subject group

34. CONTINUED
• And also in the science subject rankings where
there may be “leakages” into biology and
chemistry

35. CONTINUED
• The THE rankings are heavily biased to large
multi-”author”, hyper-cited publications, mainly in
physics but also in genetics, astronomy,
medicine and computer science
• Tend to favour small specialised institutions –
several indicators scaled

36. CONTINUED
• Both QS and THE rely on institutional data for
their general world rankings – QS drops them
for the subject group and subject rankings but
they may be added in near future

37. CONTINUED
• Ambiguities in instructions and variations in
degree of engagement mean that there is
opportunity to influence the rankings
• QS – submission of names for survey, exploiting
the sign up facility

38. CONTINUED
• THE -- recruitment of researchers from citation
rich projects or fields where citations are low

39. CONTINUED
• For both sets of rankings, it is important to read
the fine print of the data submission forms
carefully -- definitions of faculty, students,
international etc
• Not relevant to QS subject rankings at present
but should consider subjects of survey nominees

40. SPECULATION
• Rankings are constantly changing
• Should be prepare for new rankings and
changes in methodology of the existing ones.
• Specifically for subject rankings – it is likely that
QS will add to their range of indicators at least
for the subject group rankings -- add institutional
data and THE will add to the number of ranked
universities

41. CONTINUED
• THE may try to reform its citation indicator, may
add to the number of universities in its subject
rankings
• May be some pruning of redundant indicators
such as industry income
• Whatever happens, the ranking scene will be
every different in another five years