The presentation by Mary Catharine Lennon and Richard Wiggers discusses the evaluation of higher education learning outcomes by the Higher Education Quality Council of Ontario, emphasizing the importance of transparent quality measures and the impact of explicit outcomes on student learning. It highlights various assessment initiatives aimed at improving educational quality, including the Tuning project, the Assessment of Higher Education Learning Outcomes (AHELO), and the Collegiate Learning Assessment (CLA). Additionally, it addresses the challenges and benefits of large-scale assessments in higher education, including faculty resistance and the potential for benchmarking.

1. MARY CATHARINE LENNON
SENIOR RESEARCH ANALYST
AND
RICHARD WIGGERS
EXECUTIVE DIRECTOR, RESEARCH AND PROGRAMS
PRESENTATION TO:
CANADIAN EVALUATION SOCIETY ANNUAL CONFERENCE
TORONTO, ONTARIO
JUNE 10, 2013
EVALUATING HIGHER
EDUCATION LEARNING
OUTCOMES
Informing the Future of Higher Education

2. WHAT IS THE HIGHER EDUCATION
QUALITY COUNCIL OF ONTARIO?
• An independent agency of provincial government
• Mandate: To conduct research and offer policy
advice to government on access, quality and
accountability of the postsecondary education
system
• Influence based on evidence and best practices

3. TRADITIONAL MEASURES OF PSE
‘QUALITY’
• Inputs: funding, library holdings, students
receiving merit scholarships, etc.
• Outputs: retention rate, graduation rate, credit
transfer agreements, etc.
• Proxies: student evaluations, student
satisfaction, labour market outcomes, etc.
3
Informing the Future of Higher Education

4. WHAT CAN LEARNING OUTCOMES
DO?
• Promote quality in three primary ways:
– create transparency for the public, employers and
students,
– support institutional improvement through better
coordination and improved teaching and learning,
and
– enhance system level quality assurance and
accountability.
4
Informing the Future of Higher Education

5. WHAT IS THE VALUE OF IDENTIFYING
LEARNING OUTCOMES?
Hattie, J. (2009). The Black Box of Tertiary Assessment: An Impending Revolution. In L. H. Meyer, S. Davidson, H. Anderson, R. Fletcher, P.M. Johnston, & M.
Rees (Eds.), Tertiary Assessment & Higher Education Student Outcomes: Policy, Practice & Research (pp.259-275). Wellington, New Zealand: Ako Aotearoa
A study synthesizing:
800 meta-analyses
50,000+ studies
200+ million students
found that explicit outcomes and
assessment has one of the largest effects
on learning…

6. 0 0.2 0.4 0.6 0.8 1 1.2 1.4
Student self-assessment
Formative evalution to instructor
Explicit objectives and assessment
Reciprocal teaching
Feedback
Spaced vs. mass practice
Metacognitive strategies
Creativity programs
Self-questioning
Professional development
Problem solving teaching
…
Teaching quality
Time on task
Computer assisted instruction
Effect size (performance gain in σ)

7. HEQCO LEARNING OUTCOMES
PROJECTS
• ‘Measuring the Value of a Postsecondary Education’
conference and book
• HEQCO initiated a number of projects to examining
generic, sector-level, and discipline-specific learning
outcomes
– ‘Tuning’: Identifying and Measuring Sector-Based
Learning Outcomes in Postsecondary Education
– Assessment of Higher Education Learning Outcomes
(AHELO): Civil Engineering Strand
– Collegiate Learning Assessment (CLA)/College
Collegiate Learning Assessment (CCLA)
• Learning Outcomes Assessment Consortium
Informing the Future of Higher Education

8. MEASURING THE VALUE OF A
POSTSECONDARY EDUCATION
Conference and book
• What do we mean by ‘quality’
education?
• What do courses and programs
promise to deliver and do they
succeed?
• What do we know about
improving learning outcomes,
and is reform possible?
8
Informing the Future of Higher Education

9. TUNING:
• Establish clear statements of what students
should know and be able to do
• Intended to promote mobility, credit transfer
and degree recognition, transparency
• Life and Health, Physical Science, Social Sciences
• 2 year diploma, 3 year diploma, 4 year
Bachelors, Research Masters
9 Informing the Future of Higher Education

10. 10 Informing the Future of Higher Education

11. COLLEGIATE LEARNING ASSESSMENT
• Written assessment of ‘generic skills’
• Examines critical thinking, analytic reasoning,
problem solving, and written communication
skills gained from first to final year.
• Provides information on the ‘value-add’ of the
institution
11
Informing the Future of Higher Education

12. ASSESSMENT OF HIGHER EDUCATION
LEARNING OUTCOMES (AHELO)
AHELO seeks to determine if there are internationally
appropriate standard tests which measure what final-
year undergraduate students know and can do.
Primary questions
• Is it possible to have agreement on expected learning
outcomes?
• Is it possible to implement a test across cultures and
languages?
• Are the assessments valid and reliable?
12
Informing the Future of Higher Education

13. AHELO
• Three strands:
– Generic: Assessment of analytical skills, written
communication, leadership ability, etc.
– Discipline specific: Assessment of context knowledge
and students’ capacity to use or apply the
knowledge
• Economics and Civil Engineering
– Context: Surveys of student, faculty and institution
characteristics
13
Informing the Future of Higher Education

14. CIVIL ENGINEERING STRAND
PARTICIPANTS
Australia
Ontario
Colombia
Egypt
Japan
Mexico
Russia
Slovakia
Abu Dhabi
14
Informing the Future of Higher Education

15. INTERESTING PRELIMINARY FINDINGS
15
Informing the Future of Higher Education
500
520
540
560
580
600
620
Among the
top
students
Above
average
About
average
Below
average
Among the
bottom
students
Mean scores by student assessment of
own performance
500
520
540
560
580
600
620
Excellent Good Fair Poor
Mean scores by level of satisfaction with
entire undergraduate experience

16. PRELIMINARY FINDINGS FROM AHELO
• In Ontario
– There is considerable interest from programs in jurisdictional and
international comparison
– Institutions have been prompted to reflect on their curriculum design and
delivery, and on the assessment techniques they employ
– Faculty members have been prompted to reflect on their own methods
and re-evaluate what they require of students
• At the OECD
– Theoretically and practically possible to assess common student learning
outcomes internationally
– Conversations continue on the ultimate goals, purpose and possibilities of
such information
– Determining the value of the data is key to assessing the success of the
feasibility study
16
Informing the Future of Higher Education

17. BENEFITS OF LARGE SCALE
ASSESSMENTS
• Provides ‘unbiased’ indication of teaching and
learning quality
• Useful in benchmarking programs, institutions,
and systems
• Supports transferability and mobility
17
Informing the Future of Higher Education

18. CHALLENGES OF LARGE SCALE
ASSESSMENTS
• Faculty Resistance
• Institutional concern with student level data
collection
• Student recruitment
– Cumbersome, expensive
• Validity of results
– Appropriate sample? Student effort on low-stakes
test?
18
Informing the Future of Higher Education

19. LEARNING OUTCOMES CONSORTIUM
• Durham College - Student success e-portfolio: Essential
employability skills project
• Humber College - Develop rubrics to assess skills
• George Brown College - Building capacity to measure
essential employability skills
• University of Toronto - Development of analytic rubrics
for learning outcomes assessment
• University of Guelph - A pilot project for the
development of an online learning outcome assessment
strategy
• Queen’s University - Learning outcomes assessment and
program improvement
19
Informing the Future of Higher Education

20. For more information please visit
www.heqco.ca
20
Informing the Future of Higher Education