This document discusses the use of metrics and rankings to measure higher education institutions' work and performance, particularly regarding sustainability. It provides reasons why metrics are increasingly used, such as higher education's more central role in society. Examples of sustainability-focused rankings are described. Both benefits and limitations of rankings are outlined. The document argues that while rankings provide useful data, they cannot fully capture universities' complexity, and institutions should not prioritize rankings over their core missions. Overall metrics and rankings are an established part of quality assessment but must be interpreted carefully.

1. 11
Measuring Work and Performance of the University: ReasonsMeasuring Work and Performance of the University: Reasons
and Practices [in the context of sustainability and otherand Practices [in the context of sustainability and other
challenges]challenges]
Dr Jan SadlakDr Jan Sadlak,,
President, IREG Observatory on Academic Ranking andPresident, IREG Observatory on Academic Ranking and
ExcellenceExcellence
Panel Session:Panel Session:
Metrics: Measuring Sustainability Outcomes for Global ImpactMetrics: Measuring Sustainability Outcomes for Global Impact
2014 ISCN Conference,2014 ISCN Conference,
Harvard University/MIT, Cambridge, MA, 3 June, 2014Harvard University/MIT, Cambridge, MA, 3 June, 2014

2. 22
Reasons for use of "metrics" in higher educationReasons for use of "metrics" in higher education
 Higher education has moved from a guild-like status onHigher education has moved from a guild-like status on
the periphery of society to more central place as thethe periphery of society to more central place as the
wealth and good functioning of nations now depends onwealth and good functioning of nations now depends on
the performance of higher education as never before.the performance of higher education as never before.
 This change is reflected by such trends as;This change is reflected by such trends as;
massificationmassification,, diversificationdiversification [institutional and study[institutional and study
programs], pressure onprograms], pressure on public fundingpublic funding,, competitioncompetition atat
national and international levels.national and international levels.
 Higher education is under increasing pressure toHigher education is under increasing pressure to
demonstrate its inner working and performance.demonstrate its inner working and performance. In otherIn other
words, it is a call for “words, it is a call for “evidenced-based accountabilityevidenced-based accountability””
while striving forwhile striving for ““excellenceexcellence”” has to be demonstrated inhas to be demonstrated in
a manner going beyond self-declaration.a manner going beyond self-declaration.

3. 33
Higher education andHigher education and ““sustainable developmentsustainable development””
 Sustainable development is recognized as a definingSustainable development is recognized as a defining
concept for responding to Earth’s problems withconcept for responding to Earth’s problems with
imperative of economic and social development as wellimperative of economic and social development as well
as being a subject to good governance.as being a subject to good governance.
 Coping with those challenges requires an unprecedentedCoping with those challenges requires an unprecedented
integration of insights across various academicintegration of insights across various academic
disciplinesdisciplines. Only HE institutions bring together this. Only HE institutions bring together this
range of knowledge.range of knowledge.
 HE institutions see their own functioning from theHE institutions see their own functioning from the
perspective of “perspective of “sustainabilitysustainability”” especially in the contextespecially in the context
of its relationship with society [of its relationship with society [““Third MissionThird Mission””].].

4. 44
Forms, strength and weakness of principal metrics ofForms, strength and weakness of principal metrics of
quality and excellencequality and excellence
The number and type of instruments for analysis of “quality” and “excellence”The number and type of instruments for analysis of “quality” and “excellence”
is quite restrained. It includes:is quite restrained. It includes:
• Accreditation;Accreditation;
• Peer-review;Peer-review;
• Benchmarking;Benchmarking;
• Awards;Awards;
• Ranking [league tables].Ranking [league tables].
All the above instruments rely onAll the above instruments rely on quantitative measurementsquantitative measurements. This is. This is
specially the case of “rankings” which collects information from:specially the case of “rankings” which collects information from:
• Public domain data and informationPublic domain data and information;;
• Data and information collected by institutions of higher education;Data and information collected by institutions of higher education;
• [International data[International data bases -bases - ISI Web of Knowledge, Scopus,ISI Web of Knowledge, Scopus, andand
Google’s ScholarGoogle’s Scholar; and; and
• Surveys.Surveys.

5. 55
University rankingsUniversity rankings
 rankings give evidence of performance related torankings give evidence of performance related to researchresearch
performanceperformance andand approximation of teaching qualityapproximation of teaching quality;;
 provide information on the standing of higher educationprovide information on the standing of higher education
systems of higher education, institutions, academicsystems of higher education, institutions, academic
discipline and study programmes;discipline and study programmes;
 foster clarification of the mission among higher educationfoster clarification of the mission among higher education
institutions and influence pattern and forms of internationalinstitutions and influence pattern and forms of international
collaboration;collaboration;
 provide benchmarking against highly-performing systems,provide benchmarking against highly-performing systems,
institutions and study programmes;institutions and study programmes;
 provide additional rationale for allocation of funds;provide additional rationale for allocation of funds;
 represent a key differentiator between legitimaterepresent a key differentiator between legitimate
institutions and programmes and degree mills;institutions and programmes and degree mills;
 reveal certain taboos about inner workings of academicsreveal certain taboos about inner workings of academics
and institutions.and institutions.

6. 66
University rankings and sustainability [Third Mission]University rankings and sustainability [Third Mission]
Majority of rankings are covering research and teaching activities. However,Majority of rankings are covering research and teaching activities. However,
measuring work and performance of HE institutions in relation tomeasuring work and performance of HE institutions in relation to
“sustainability” is gaining growing attention. In this regard a pioneering“sustainability” is gaining growing attention. In this regard a pioneering
role has been carried out by such international initiatives as:role has been carried out by such international initiatives as:
 UI Green Metric World University RankingUI Green Metric World University Ranking, which since 2010 is, which since 2010 is
produced by the University of Indonesia [in collaboration with the Higherproduced by the University of Indonesia [in collaboration with the Higher
Education Evaluation & Accreditation Council of Taiwan and EducationalEducation Evaluation & Accreditation Council of Taiwan and Educational
Policy Canada];Policy Canada];

 E3M: European Indicators and Ranking Methodology for UniversityE3M: European Indicators and Ranking Methodology for University
Third MissionThird Mission, has been a three years research project (2008-2011), has been a three years research project (2008-2011)
which was carried out by consortium of 8 universities and higher educationwhich was carried out by consortium of 8 universities and higher education
institutions [coordinated by Technical University of Valencia, Spain] withinstitutions [coordinated by Technical University of Valencia, Spain] with
financial support of the Europe Commission;financial support of the Europe Commission;
 U-MultirankU-Multirank is an independent ranking [implemented by consortium] withis an independent ranking [implemented by consortium] with
financial support in its initial years from the European Union. Its firstfinancial support in its initial years from the European Union. Its first
edition was presented on May 13, 2014.edition was presented on May 13, 2014.

7. 77
Use of rankings for enhancement of performance,Use of rankings for enhancement of performance,
excellence and innovation [1]excellence and innovation [1]
 even the critics admit thateven the critics admit that rankings provide thought-provoking datarankings provide thought-provoking data
for the whole range of activities including those directly and indirectlyfor the whole range of activities including those directly and indirectly
relevant to sustainability;relevant to sustainability;
 aaassessment can produce tensionsassessment can produce tensions butbut metrics provide a transparentmetrics provide a transparent
and reliable lens that allows us to filter out emotionalism and ideology. Notand reliable lens that allows us to filter out emotionalism and ideology. Not
to mention verification of validity of research findings;to mention verification of validity of research findings;
 complexity of “sustainability” agenda should not discouragecomplexity of “sustainability” agenda should not discourage searching ofsearching of
its metricsits metrics..
““No matter how difficult your measurement problem seems, the first stepNo matter how difficult your measurement problem seems, the first step
in measurement is to change and assumption you may have been makingin measurement is to change and assumption you may have been making
for a very long time. Instead of assuming that it is a fundamentallyfor a very long time. Instead of assuming that it is a fundamentally
immeasurable intangible, assume that it is measurable. Then the onlyimmeasurable intangible, assume that it is measurable. Then the only
question becomes whether you are clever enough to figure out how”.question becomes whether you are clever enough to figure out how”.
[[SourceSource: Douglas Hubbard. “Everything is Measurable” downloaded from: Douglas Hubbard. “Everything is Measurable” downloaded from
http://www.cio.com/article/print/112101 on 16 April 2010http://www.cio.com/article/print/112101 on 16 April 2010]]

8. 88
Use of rankings for enhancement of performance,Use of rankings for enhancement of performance,
excellence and innovation [2]excellence and innovation [2]
 rankings are convenient tool forrankings are convenient tool for marketingmarketing,, policypolicy
debatesdebates, and, and setting up the benchmarks forsetting up the benchmarks for
excellenceexcellence. Metrics provided by rankings can facilitate. Metrics provided by rankings can facilitate
taking adequate account of the complex relation betweentaking adequate account of the complex relation between
potential and display of pursuing excellence in higherpotential and display of pursuing excellence in higher
education.education.
 rankings are data driven therefore HE institutions arerankings are data driven therefore HE institutions are
confronted with such issues as data interpretation whichconfronted with such issues as data interpretation which
can be constructive or paralyzing when outcomes ofcan be constructive or paralyzing when outcomes of
rankings are being interpreted with excessive zeal.rankings are being interpreted with excessive zeal.
 we are moving from the “rankings are here to stay andwe are moving from the “rankings are here to stay and
there is nothing you can do about it” tothere is nothing you can do about it” to a more nuanceda more nuanced
and sophisticated dialogueand sophisticated dialogue about assessment,about assessment,
evaluation and improvement of learning and instruction.evaluation and improvement of learning and instruction.

9. 99
Academic excellence and sustainability agendaAcademic excellence and sustainability agenda
Academic excellence is an intersection of a number of factors types ofAcademic excellence is an intersection of a number of factors types of
institution, mission, history, location, funding and institutionalinstitution, mission, history, location, funding and institutional
leadershipleadership. It is therefore difficult to formulate “one fits all” set of metrics. It is therefore difficult to formulate “one fits all” set of metrics
for enhancement of performance, collaboration and innovation, whichfor enhancement of performance, collaboration and innovation, which
represent a framework for sustainability agenda.represent a framework for sustainability agenda.
Principles of ExcellencePrinciples of Excellence [formulated for “teaching and learning”] but its[formulated for “teaching and learning”] but its
basic features can be expanded to other works of the university, includingbasic features can be expanded to other works of the university, including
those under agenda of “sustainability”:those under agenda of “sustainability”:
 Aim High – and Make Excellence Inclusive;Aim High – and Make Excellence Inclusive;
 Give Students a Compass;Give Students a Compass;
 Teach the Arts of Inquiry and Innovation;Teach the Arts of Inquiry and Innovation;
 Engage the Big Questions;Engage the Big Questions;
 Connect Knowledge with Choices and Action;Connect Knowledge with Choices and Action;
 Foster Civic, Intercultural, and Ethical Learning;Foster Civic, Intercultural, and Ethical Learning;
 Assess Students’ Ability to Apply Learning to Complex Problems.Assess Students’ Ability to Apply Learning to Complex Problems.
[[SourceSource: Association of American Colleges and Universities (AACU), 2007].: Association of American Colleges and Universities (AACU), 2007].

10. 1010
Concluding remarksConcluding remarks
It needs to be repeatedly said that ranking can only be aIt needs to be repeatedly said that ranking can only be a
proxy reflection of the complex work of higher educationproxy reflection of the complex work of higher education
enterprise. Therefore:enterprise. Therefore:
 Do not subject the mission of the university to “tyranny ofDo not subject the mission of the university to “tyranny of
seeking status” it is therefore wrong policy andseeking status” it is therefore wrong policy and
management to take any personal or institutional solelymanagement to take any personal or institutional solely
based on rankings;based on rankings;
 HE institutions need to come to terms with new landscapeHE institutions need to come to terms with new landscape
of HE in which competition is more evident than before butof HE in which competition is more evident than before but
if they are to compete they should be inspired but do notif they are to compete they should be inspired but do not
be mesmerized by ranking;be mesmerized by ranking;
 Academic rankings are an entrenched phenomenon aroundAcademic rankings are an entrenched phenomenon around
the world and it is recognized as one of widely-usedthe world and it is recognized as one of widely-used
methods of “quality assessment”.methods of “quality assessment”.