Asegurar la calidad y los rankings, a cargo de Josep Anton Ferré, director de la Agencia para la Calidad del Sistema Universitario de Cataluña (AQU). La conferencia se presentó en el 1er Seminario Internacional sobre Rankings en Educación Superior y E-learning organizado por la UOC.

1. www.aqu.cat
La qualitat, garantia de millora.
www.aqu.cat
La qualitat, garantia de millora
Quality AssuranceandRankings
Josep A. Ferré
Director
September 2011

2. www.aqu.cat
La qualitat, garantia de millora.
Quality assurance and rankings?

3. www.aqu.cat
La qualitat, garantia de millora.
Some HEI in Spain have read this, but not many seem to have understood it !
The QA problem
OUR MESSAGE:
1.
Our objective is that you (the HEI) have such a good Internal Quality Assurance System (IQAS) that we (the QA agency) may develop external quality assurance processes that are of very low intensity.
2.
Meanwhile … if you do not have an IQAS we’ll help you to conceive, design and implementone, the one that you choose for you (AUDIT program).
3.
In addition, we’ll help you to apply the IQAS to your programs through the life cycle defined in the Framework for Accreditation of programmesof AQU.
4.
After, we’ll come back to evaluate how this IQAS is operating. However we’ll not analyze directly the IQAS, rather the focus will be on the outputs of your programmes. If they are “within control limits” we’ll conclude that …
•
your choice of this IQAS was correct and it is working
•
your processes are under control
5.
If you have a good IQAS and good results then you can become more transparent and step into higher accountability, and then ….
•
your stakeholders will grant more autonomy to your institution
•
more resources will be committed to you (… perhaps!)
•
you will strengthen your social leadership and recognition

4. www.aqu.cat
La qualitat, garantia de millora.
Framework for the Accreditation of programmes
AccreditationPlanDevelopMonitorImprovePlanDevelopMonitorImprovePlanDevelopMonitorImprovePlanDevelopMonitorImprove
Monitoring
Monitoring report
Monitoring report
External evaluation
Monitoring report
Ex-ante accreditation

5. www.aqu.cat
La qualitat, garantia de millora.
Monitoring report
Annual plan
Quantitative indicators
Assessment of the indicators
Improvement actions
ModificationPlanDevelopMonitorImprove
How would you improve it?
What do you want to do? Why?
How will you do it?
Why that way?
How do you know it’s OK?
How will you measure it?
How have you done it?
Would you repeat it?

6. www.aqu.cat
La qualitat, garantia de millora.
OUTPUT
Quantitative indicators
Mission
Vision
Values
OUTPUT
PROCESS
INPUT
HR
MR
ORG
GOVERNANCE
€
GOV
UNIV
EST
SOC
Specifications
Quantitative indicators:
•
Characteristics of the input
•
Contributions of the university to the teaching-learning process
•
Characteristics of the output
Which quantitative indicators should we monitor?

7. www.aqu.cat
La qualitat, garantia de millora.
Quantitative indicators
1.
Previous qualifications and degrees of the incoming students
2.
Personal, family and social characteristics of the students
3.
Qualifications and competence of the teaching staff
4.
Hours of lectures, seminars, labs, etc.. and average group size
5.
Classrooms, labs, libraries and other material resources used
6.
Use of the virtual campus (volume of text rich documents, downloads, etc.)
7.
Availability, updatability and usability of the web information
8.
Evidence of the assessment methods used (exams, reports, etc.)
9.
Student internship, mobility and career advice
10.
Student satisfaction, teachers satisfaction
11.
Academic results (performance, residence time, abandon rate, etc)
12.
Personal results (personal and inter-personal skills)
13.
Employability (occupation and profession)

8. www.aqu.cat
La qualitat, garantia de millora.
Quality assurance and rankings?
Rankings are not an ingredient of the QA recipe … but …
…what would happen if you add a ranking to your QA recipe?
To add a ranking to your QA recipe, first you should check that both are “compatible”. A very nice approach to this compatibility test are the …
You will find them and many more in this recent document of the EUA …..
I recommend reading it with great care without skipping over a single word !!

9. www.aqu.cat
La qualitat, garantia de millora.
An example of a bibliometricranking

10. www.aqu.cat
La qualitat, garantia de millora.
If I was to make a ranking (1) …
2–How HEI’s stakeholders (society, public authorities, students, HEI’s employees) understand these three missions are best achieved? How they evaluate the quality of any/all the three missions?
Fitness for purpose?Value for money?
Excellence?Perfection?Transformation?
1 -Which are the relevant missions of the institutions to be ranked?
•
Teaching or “only teaching”?
•
Research or “also research”?
•
3rd mission? (… what’s that?)
4 -How homogeneous/comparable is the data?
to which domain can it be extended without lacking context?
(regional, national, international) ×(field)
3 –How the success in each of the assigned missions is measured?
•
which data we already have? (do we use it for decision making?)
•
which data we do not have?
•
do we have plans to collect the missing data? (do we plan to use it?)

11. www.aqu.cat
La qualitat, garantia de millora.
3rdmission
research
teaching
society
public authorities
students
HEI employees
fitness for purpose
value for money
excellence
perfection
transformation
5 ×4 ×3 = 60 !
Which data we have on this?
Which data we do not have?
Multidimensional reality perception / conceptual domain

12. www.aqu.cat
La qualitat, garantia de millora.
Who is interested in having rakings that span this range of domains?
….
Our stakeholders are interested in all the physical domains?
…
Or, are we thinking about future customers rather than current stakeholders?
Which data we have on this?
Which data we do not have yet?
Which data we’ll never have?
3rdmission
research
teaching
society
public authorities
students
HEI employees
all the world
international
national
regional
field
Multidimensional physical domain

13. www.aqu.cat
La qualitat, garantia de millora.
If I was to make a ranking (2) …
5 –Raw data of pre-processed data?
Pre-processing options:
•
how do we make it “size invariant”? … OUTPUT/SIZE ?
•
how do we align the zero level? …. (OUTPUT -INPUT) ?
•
how do we relate to cost? …. (OUTPUT-INPUT)/€?
•
only €matters? … (OUT-IN) / (€+LEGISLATION+GOVERNANCE) ?
7–Moving average, current value or rate of change data?
6–Keep it multi-dimensional or cook into a single number?
•
will users be allowed to select the weights?
•
poor teaching can be replaced by good research?
•
no-research can be replaced by good teaching?
•
can we skip the 3rd mission from the list? …

14. www.aqu.cat
La qualitat, garantia de millora.
And now we have it !
8 –What’s for? … How we’ll use it?
-it’s a drive for continuous improving?
-connects with the strategic management of the institution?
-is related to some data driven decision making process?
… If quality in higher education is a core success factor for institutional success, the quality assurance processes should not be considered as something “extra” in addition to the other core processes in higher education, but should be integral. Guaranteeing a certain quality or enhancing the quality of a programme becomes an integral part of the regular management of a HEI. There is a trend of moving from quality assurance towards quality oriented higher education management, which means that former quality assurance processes get directly linked to core management processes.

15. www.aqu.cat
La qualitat, garantia de millora.
-or is just part of the marketing (reputation management) of the HEI?
And now we have it !
8 –What’s for? … How we’ll use it?
-it’s a drive for continuous improving?
-connects with the strategic management of the institution?
-is related to some data driven decision making process?
-It’s a part of the public policy of the government?
-It’s the “placebo” of the public policy of the government?
-It’s the substitute of the policy that public authorities …
~ do not have? (…not yet developed)
~ do not want to have? (…neo-liberal approach)
~ do not have time to elaborate? (…too fast!)
~ are happy to have externalized (…no comment!)
Are there some connections here?
Do HEI prefer rankings to an IQAS or to transparency accountability?
Have QA agencies failed not putting transparency accountability on top of the list?

16. www.aqu.cat
La qualitat, garantia de millora.
Surveys
MEDIA
… quality assurance
… transparency
… enhancement of quality
… accountability
HEI
DATA
QA
(ESG 1.6 and 1.7)
DATA
Data drivendecision making and strategic management
Transparency… “to show list of all the things HEI do”
Accountability… “to check the list of all the things HEI do .vs. stakeholder’s expectations”
Accountability+… “to check with the stakeholders the list of (output-input)/resources”
Communication… “to publish the list of all the things HEI do”
QA, rankings and transparency tools
RANKING
GOVERMENT
SOCIETY
ESTUD.
Surveys
Should QA have pushed up ESG 1.6 and 1.7 to the top of the list?

17. www.aqu.cat
La qualitat, garantia de millora.
Why transparency and accountability?
Intended mission(s)
Resources and autonomy
Transparency and accountability
What activities stakeholders expect from universities to develop?
How universities demonstrate that they develop their mission(s) with effectiveness and efficiency?
What resources (€and rules) are allocated to universities?
The resources allocated and the transparency and accountability methods are always an evidenceof the true intended missions (even if these have not been formulated explicitly).
Therefore it is easy to detect inconsistencywhen nice intended missions are associated to insufficientresources or poortransparency and accountability tools, for example, rankings!
What universities do?
What society expects from universities?
How universities serve the social needs?

18. www.aqu.cat
La qualitat, garantia de millora.
QA and rankings …. revisited
When you read carefully the ranking recipe is difficult to find there any reference to … •
“fitness for purpose”… the mission of the HEI ... the effectiveness •
“value for money” … the resources allocated to the HEI … efficiency
???
The QA recipe and the rankings recipe share some common ingredients… •
“a continuous improving strategy” … the internal QA effort •
“a self-evaluation report” … transparency accountability •
but are cooked in a very different way•
are served with different dressings•
point to opposite goals•
“experts committee” … the external QA review

19. www.aqu.cat
La qualitat, garantia de millora.
Rankings are like …
When we say “food” we do not figure up the same things!

20. www.aqu.cat
La qualitat, garantia de millora.
Thank you very much!
20