1) The document outlines a study that assessed the reliability and validity of students' self-reported learning outcomes against stated course learning outcomes. 2) Preliminary results found the outcome-specific questionnaires to have high reliability. Most courses saw significant increases in student self-ratings from beginning to end. 3) Further analysis is still needed to determine if end-of-course self-ratings correlate with assessments of student task performance against learning outcomes.

1. TCI/MOC Asia
Conference
2021
Outcome-Specific Feedback on Courses
Preliminary Psychometric Results of
Scholarly Research
Associate Prof. Monirith Ly
CamEd Business School
Phnom Penh, Cambodia
02 July 2021

2. Rationale
• External quality assurance and accreditation agencies across the world are
increasingly requiring outcome-based approaches to education.
• As part of the fitness for purpose quality model, an outcome-based curriculum
must be evaluated to measure its effectiveness in meeting its stated
outcomes.
• However, few course evaluation instruments ask students to assess their own
learning outcomes.

3. Significance
• Curriculum dev.: In an outcome-based education system if course
coordinators and faculty members use feedback to improve CLOs, all other
course components will need to be aligned accordingly. This will affect the
whole course curriculum to improve student learning and, therefore, better
meet the needs of students, faculty and the program.
• IQA: sustainable, systematic inputs for outcome-based curriculum reviews in
continuous quality improvement cycles
• EQA: documentation for accreditation & accountability

4. Research Questions
This study assesses the reliability and validity of students’ self-measures of
learning outcomes in relation to course intended learning outcomes (CLOs):
1. Do students’ perceptions of their beginning abilities against CLOs change
after exposure to the course? If so, do their self-ratings significantly
increase or decrease?
2. Do students significantly improve, during the course period, their self-
reported outcomes against CLOs? If so, to what extent?
3. Are post-survey self-ratings correlated to the assessment of students’ task
performance against CLOs? If so, how correlated are they?

5. Research Design
(RQ3)
(RQ2)
(RQ2)
(RQ1)
calibrated
paired t-tests
Outcome-Specific
Assessment (OSA)
Post-Ability
correlation tests
Retro-Ability
Pre-Ability

6. Data Collection
• Course sampling criteria: large enrollment per course & assessable course-
specific LOs (knowledge & cognitive skills)
• Survey data have been collected—mostly online—from 1,209 undergraduate
students of 8 courses at 3 prominent higher education institutions in
Cambodia.
• Assessment data have been provided by 13 lecturers of 6 courses.

7. Sampled Courses
1. Khmer & Regional History
2. English I
3. English II
4. Introduction to Environmental Science
5. Cultural Anthropology
6. Health Education & Fitness
7. Introduction to Computers
8. English 1P

8. Instruments
Due to the lack of existing outcome-specific instruments, the researcher
developed all of the tools:
• Outcome-Specific Questionnaires (OSQ) for students
• Outcome-Specific Assessment Tools (OSAT) for lecturers to match CLOs to
assessment items or tasks

9. Sample Pre-OSQ: Khmer & Regional History Course
Please honestly rate your abilities at the start of this semester.
weak moderate good strong
1. Grasp the meanings of history and its importance for
themselves and the nation, and understanding the origins and
evolutions of people and languages on Southeast Asia.
⃝ ⃝ ⃝ ⃝
2. Analyze how early states in the region were formed through
various historical sources.
⃝ ⃝ ⃝ ⃝
3. Investigate changes when Southeast Asia states were
introduced to Indianization, shifts in Chinese trade policies and
European colonialism.
⃝ ⃝ ⃝ ⃝
4. Compare and contrast nationalism movements in the
region.
⃝ ⃝ ⃝ ⃝
5. Evaluate Cambodia nation-building efforts in the post-
independence period.
⃝ ⃝ ⃝ ⃝

10. Sample Post-OSQ: Khmer & Regional History Course
Please honestly rate your abilities at the semester start and now.
weak moderate good strong
1a. Grasp the meanings of history and its importance for themselves and the nation,
and understanding the origins and evolutions of people and languages on Southeast
Asia. (term start)
⃝ ⃝ ⃝ ⃝
1b. (now) ⃝ ⃝ ⃝ ⃝
2a. Analyze how early states in the region were formed through various historical
sources. (term start)
⃝ ⃝ ⃝ ⃝
2b. (now) ⃝ ⃝ ⃝ ⃝
3a. Investigate changes when Southeast Asia states were introduced to Indianization,
shifts in Chinese trade policies and European colonialism. (term start)
⃝ ⃝ ⃝ ⃝
3b. (now) ⃝ ⃝ ⃝ ⃝
4a. Compare and contrast nationalism movements in the region. (term start) ⃝ ⃝ ⃝ ⃝
4b. (now) ⃝ ⃝ ⃝ ⃝
5a. Evaluate Cambodia nation-building efforts in the post-independence period.
(term start)
⃝ ⃝ ⃝ ⃝
5b. (now) ⃝ ⃝ ⃝ ⃝

11. Data Analysis Procedures
1. Conducting principal component analyses (PCAs) on SPSS to determine
items for each principal component to meet the unidimensionality
assumption for calibration.
2. Conducting scale reliability analyses on SPSS.
3. Calibrating the items of each principal component with a graded response
model (GRM) of item response theory (IRT) on RStudio.
4. Conducting calibrated paired t-tests on Excel to answer RQ1 & RQ2.
5. Conducting Pearson or Spearman correlation analyses on SPSS to answer
RQ3.

12. Reliability Results: Cronbach’s Alpha
Cronbach’s alpha results range
from .72 to .96, which indicate
high levels of reliability of the
Outcome-Specific
Questionnaires.
Course items Pre Retro Post
Khmer & Regional History 5 .81 .89 .79
English I 18 .93 .95 .93
English II 11 .89 .88 .89
Introduction to
Environmental Science
10 .88 .93 .90
Cultural Anthropology 6 .90 .93 .91
Health Education & Fitness 4 .86 .83 .72
Introduction to Computers 11 .94 .96 .95
English 1P 10 N/A .93 .89

13. Results for Research Question 1
• RQ1: Do students’ perceptions of their beginning abilities against CLOs change
after exposure to the course? If so, do their self-ratings significantly increase
or decrease?
• Results:
- In 4/7 courses, non-significant difference between Pre- & Retro-Abilities
- In 3/7 courses, significant difference (decrease) between Pre- & Retro-
Abilities with 2 medium & 1 small effect sizes

14. Results for Research Question 2
• RQ2: Do students significantly improve, during the course period, their self-
reported outcomes against CLOs? If so, to what extent?
• Results:
- In 6/7 courses, significant increase from Pre- to Post-Abilities with 3
medium, 2 small & 1 large effect sizes
- In 1/7 courses, non-significant increase from Pre- to Post-Abilities
- In 8/8 courses, significant increase from Retro- to Post-Abilities with 4
medium, 3 large & 1 small effect sizes

15. Results for Research Question 3
• RQ3: Are post-survey self-ratings correlated to the assessment of students’
task performance against CLOs? If so, how correlated are they?
• Results: not yet completed

16. Implications
• All OSQs used are reliable.
• The Post-OSQs can be used without the Pre-OSQs.
• Each set of course-specific learning outcomes belongs to one principal
component.
• Survey self-ratings, as ordinal data, must be calibrated before appropriate
statistical summaries and inferences can be drawn because the gap between
ratings of 1 and 2 may not be the same as that between 2 and 3.