Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3)

ROJoson OCIL on Medical Education
OCIL = Online Collaborative and Interactive
Learning
Tools in Measuring Program and Learning Outcomes
of Medical Students
Tools in Measuring Program and Learning Outcomes of Medical
Students – A Guide in Scoring and Grading (Part 3)
Subtitle: Tools in Measuring Knowledge, Skill and Attitude of Medical
Students - A Guide in Scoring and Grading (Part 3)

Learning
Tools in Measuring Program and Learning Outcomes of
Medical Students
Students – A Guide in Designing a Master Blueprint (Part 1)
Students – A Guide in Designing the Tools (Part 2)
Recommended Prior Study:
• Assessment and Evaluation of Medical Students – ROJoson TPORs
• Tools in Measuring Program and Learning Outcomes of Medical Students – A
Guide in Designing a Master Blueprint (Part 1); A Guide in Designing the Tools
(Part 2)

Learning
Learning objectives:
At the end of the session, the student must be able to:
• Define scoring.
• Define grading.
• Define points.
• Define raw scores.
• Give examples of translated scores from raw scores.

Learning
• Differentiate norm-referenced and criterion-referenced system of
grading.
• Discuss the different methods on how to determine the pass or cut
score.
• Discuss the limitations of scoring and grading systems in a medical
school.

Learning
• Explain why medical schools should go for a criterion-referenced
systems of scoring and grading.
• Discuss the limitations of scoring and grading systems in medical
schools.
• Discuss the usefulness of item analysis.

Learning
Students – A Guide in Designing a Master Blueprint (Part 1)
ROJoson recommended framework
for the design and development of tools to measure the global
accomplishment of the student learning and performance
Elements to
Measure
Tools to Use
to Measure
Performance
Indicators
Assessment
and
Evaluation
Program Educational
Objectives
Program Outcomes
Course-level Learning
Outcomes
Session-level Learning
Objectives

Learning
Students – A Guide in Designing the Tools (Part 2)
GENERAL FORMAT OF A BLUEPRINT FOR EDUCATIONAL MEASURING TOOLS WITH
SEVERAL LEARNING OBJECTIVES AND OUTCOMES
Contents for
Assessment
Plans
Time
allotment
(overall)
Weight
(relative
importance
of contents)
Number of
questions
Types of
questions
Instructions
and notes
LO1
LO2
LO3
LO4
LO5
*LO – Learning Objectives / Outcomes

Learning
Assumptions on activities that have been undertaken:
• The tools have been formulated. These consist of 3 categories:
TEST-BASED TOOL, OBSERVATION-BASED TOOL, and PROJECT-
BASED TOOL (per ROJoson recommendation for medical schools).
• What will be done now are: PLANS ON HOW TO SCORE and HOW TO
GRADE.

Learning
SCORING AND GRADING MANAGEMENT SYSTEMS
2 distinct but interrelated and should be integrated systems:
Scoring Management System
Grading Management System
TEST-BASED
TOOLS
(e.g.
Written/Oral)
OBSERVATION-
BASED TOOLS
(e.g. OSCE)
PROJECT-
BASED TOOLS
(e.g. Case
Study)
Scoring Mgt
System
Grading Mgt
System

Learning
SCORING AND GRADING MANAGEMENT SYSTEMS
2 distinct but interrelated and should be integrated systems:
Scoring Management System
Grading Management System
TEST-BASED
TOOLS
(e.g.
Written/Oral)
OBSERVATION-
BASED TOOLS
(e.g. OSCE)
PROJECT-
BASED TOOLS
(e.g. Case
Study)
Scoring Mgt
System
Grading Mgt
System
Scoring and grading systems will be dependent
on the type of educational measuring tools to be
used. They may differ.

Learning
SCORING vs GRADING MANAGEMENT SYSTEM
Scoring Management System refers to a set of policies, processes and
procedures on scoring of educational measuring tools.
Scoring refers to the process of assigning numerical units to represent
points that students will get and/or have gotten in educational
measuring tools such as test-based, observation-based and project-
based tools.

Learning
Grading Management System refers to a set of policies, processes and
procedures on grading of educational measuring tools.
Grading refers to the process of applying standardized measurements
for varying levels of achievements in a learning activity or course of
study using the scores the students will get and/or have gotten in
educational measuring tools such as test-based, observation-based
and project-based tools.

Learning
In terms of implementation of actual processes, there is a proper
sequence:
• Scoring precedes grading.
• Assign scores and get scores before giving grades.
However, in the formulation of the plan for scoring and grading, there
is already integration of the two processes – how to score and how to
grade are already placed in the plan.

Learning
POINTS, MARKS, SCORES, RATINGS, GRADES etc.
Some words of clarifications on use of terms which can be used
interchangeably and which oftentimes cause confusion
POINTS = MARKS
SCORES MAY BE THE SAME AS POINTS BUT NOT ALL THE TIME.
TOTAL POINTS = SCORES
RATINGS USED IN BROADER SENSE THAN GRADES.
GRADES UTILIZE THE SCORES FOR DECISION-MAKING.
SCORES ARE CONVERTED TO GRADES.

Learning
POINTS
Points are numeric units which represent relative value or weight of
importance determined by a teacher on specific items in educational
measuring tools such as tests, observation technique and project-
based technique.
The total point gotten by the students in each educational measuring
tool is the sum of all points assigned to each item of assessment.
The total point gotten by the students is the sum of all points on
correct answers and sum of all numeric values in a rating scale.

Learning
POINTS on a test with a numeric value on each question or item
Question
or item
Points assigned Student 1
Points gotten
(correct answers)
Student 2
Points gotten
(correct answers)
1 3 3 3
2 2 2 0
3 2 2 0
4 3 3 3
5 5 5 0
Total Points = 15 Total = 15 (100%) Total = 6 (40%)
Total points = Score (which may be the GRADE or not, if score will be standardized)

Learning
POINTS on an assessment tool with rating scale with assigned values
on each item (given an example of a student performance)
LP1 LP2 LP3 LP4 LP5
Items /
Criteria
1 2 3 4 5
1 ❸
2 ❹
3 ❺
Total points gotten by student = 12
Key:
1 – Very poor
2 – Poor
3 – Average
4 – Good
5 – Excellent
There is an
accompanying
descriptor of
criteria of level of
performance.
LP – Level of Performance with Descriptor
Assigned points

Learning
SCORES
SCORES are numeric units which represent the total number of points
that students will get and/or have gotten in educational measuring
tools such as tests, observation technique and project-based
technique.
SCORES = TOTAL POINTS
POINTS in specific or groups of items or questions – “scores” not
usually used, but “points.”

Learning
POINTS on a test with a numeric value on each question or item
Question
or item
Points assigned Student 1
Points gotten
(correct answers)
Student 2
Points gotten
(correct answers)
1 3 3 3
2 2 2 0
3 2 2 0
4 3 3 3
5 5 5 0
Total = 15 Total = 15 (100%) Total = 6 (40%)
If above is the whole test, the total point of 15 of Student 1 is also
the score; total points of 6 of Student 2 is the score.
NOT SCORE
but
individual
points for
each item

Learning
SCORES
RAW SCORES – scores without any sort of adjustment or
transformation such as simple number of questions answered
correctly
TRANSLATED SCORES – the result of some transformation(s) applied to
the raw scores – adjusted and converted to a standardized grading
system
The score at which a student passes or fails is called
the pass or cut score.

Learning
Translated Scores – Example
Raw Scores
(examples)
Range of Scores Grade Point
System
Letter Grade
System
96 95-100 1 A
92 90 -94 1.5 B
86 85-89 2 C
83 80-84 2.5 D
76 75-79 3 E
70 70-74 4 R
60 Below 70 5 F

Learning
Raw Scores
(examples)
Norm-
referenced
scores
Range of
Scores
Grade Point
System
Letter Grade
System
96 Raw scores
adjusted to
the mean –
0.5 to 1 SD
as the pass
score based
on norm-
reference
system
95-100 1 A
92 90 -94 1.5 B
86 85-89 2 C
83 80-84 2.5 D
76 75-79 3 E
70 70-74 4 R
60 Below 70 5 F

Learning
Raw Scores
(examples)
Criterion-
reference
scores
Range of
Scores
Grade Point
System
Letter Grade
System
96 Raw
scores
adjusted
to
criterion-
referenced
system
95-100 1 A
92 90 -94 1.5 B
86 85-89 2 C
83 80-84 2.5 D
76 75-79 3 E
70 70-74 4 R
60 Below 70 5 F

Learning
GRADES
Grades can be assigned as letters (usually A through F), as a range (for
example, 1 to 5), as a percentage, or as a number out of a possible
total (often out of 100).
Grades in medical schools are usually in numeric units.
May be using raw scores or translated scores.
May be using norm-referenced or criterion-referenced systems for the
passing grades.

Learning
GRADES
GRADES CAN BE BASED ON RAW SCORES.
For example, if 100 points is the total score, and a student got 100
points or total score, the student gets a grade of 100%. Another
student got 90 points or total score, then this student gets a grade of
90%.

Learning
GRADES
GRADES can be based on TRANSLATED SCORES.
If scores within
95 -100 points or score = Grade of 1
90 – 94 points or score = Grade of 1.5
85 – 89 points or score = Grade of 2
For example, a student got 93 total points or score, the student gets a
grade of 1.5. Another student got 89 points or total score, then this
student gets a grade of 2.

Learning
GRADES
GRADES CAN BE BASED ON:
Norm-referenced vs Criterion-referenced Systems
Norm-referenced means referencing how the score of a student
compares to other students. (grading on a curve using mean and
standard deviations)
Criterion-referenced means referencing how the score compares to a
criterion such as a cut score or a body of knowledge. (grading using a
set of defined standards)

Learning
Determining the Pass or Cut Scores
After constructing the educational measuring tools (test-based tools,
observation-based tools, and project-based tools),
the next step is to determine how to grade and most importantly, how
to determine the pass or cut scores.
Let us now FOCUS on how to determine the pass or cut scores.
The score at which a student passes or fails is called the cut
score or pass score or passing score.
Minimum Pass Level (MPL) is another common term.

Learning
ROJoson’s Recommended Ways:
Some Preliminaries
• ROJoson’s categorization of educational measuring tools for medical
students consist of TEST-BASED, OBSERVATION-BASED, AND
PROJECT-BASED TOOLS.
• TEST-BASED Tools consist of SELECTIVE OBJECTIVE TESTS and ORAL /
ESSAY TESTS.

Learning
Some Preliminaries
• Scoring and grading systems will be dependent on the types of
educational measuring tools.
• The SELECTIVE OBJECTIVE TESTS will be managed differently from
the ORAL / ESSAY TESTS, OBSERVATION-BASED and PROJECT-BASED
Tools.
• The ORAL / ESSAY Tools, OBSERVATION-BASED and PROJECT-BASED
Tools will be managed similarly.

Learning
Some Preliminaries
Policies and Procedures on Scoring and Grading System
• The medical school should come out with policies and procedures
on its scoring and grading systems.
• The students should be informed of these policies and procedures
so that there will be NO misunderstanding.

Learning
• The passing score for an educational measuring tool should be set in
accordance with the purpose of the exam (formative, summative,
knowledge, skills, attitude, etc.)
• The passing score set for an educational measuring tool must be
defensible, acceptable, and credible (not arbitrary).

Learning
• The process of determining an effective and valid cut score for an
educational measuring tool is referred to as standard setting.
• This requires a properly selected panel of expert judges and a
suitable standard setting method, which best fits the purpose of the
educational measuring tool.

Learning
There are a number of methods for determining the pass or cut scores,
some of which are mentioned below:
• Using norm-referenced system (or grading on a curve): the teacher
computes for the mean and standard deviations and then decides
whether the pass or cut score (or minimum pass level) is 0.5 or 1
deviations below the mean. Some would just use the mean as the
pass or cut score. Critique: Designed to only pass the “best of the
best”; relative standard as students of medicine need to be judged
on their mastery of their professional content.

Learning
There are a number of methods for determining the pass or cut scores,
some of which are mentioned below:
• Standard setting using criterion-referenced system: process of
evaluating (and grading) the learning of students against a set of
pre-specified qualities or criteria, without reference to the
achievement of others. The pre-specified qualities or criteria are
what students have to do during assessment in order to
demonstrate that they have achieved the learning outcomes.

Learning
There are a number of methods for standard setting using criterion-
referenced system.
• Institution’s policy on absolute pass score: commonly, 70% or 75%
(just make sure that the tools are properly constructed and valid with
clear criteria)

Learning
There are a number of methods for standard setting, some of which
are mentioned below:
• Institution’s policy on absolute pass score: commonly, 70% or 75%
• If flexibility is allowed, the teacher or panel of teachers can decide
on the pass or cut score based on importance of content, level of
expectations on the students’ learning and difficulty or ease of
exam. Thus, some tests may have a pass or cut score of 65%; some,
even 80%. This arrangement on flexibility will be defensible,
acceptable, and credible.

Learning
There are a number of methods for standard setting, some of which
are mentioned below:
FOR OBJECTIVE WRITTEN TESTS (MULTIPLE CHOICE QUESTIONS):
• Nedelsky Method
• Angoff Method
• Ebel Method
*Among the three, Angoff Method is most commonly used (and will
be described below).

Learning
The Angoff method relies on subject-matter experts (SMEs) who
examine the content of each test question (item) and then predict how
many minimally-qualified candidates would answer the item correctly.
The average of the judges’ predictions for a test question becomes
its predicted difficulty. The sum of the predicted difficulty values for
each item averaged across the judges and items on a test is the
recommended Angoff cut score (or minimum pass level – MPL).
MPL = Sum of MPL of all items /no. of items x 100

Learning
The Angoff method
The experts assess each question and determine the percentage of 100
minimally competent students who will be able to answer the item
correctly.
The MPL of the test is computed as follows:
MPL = Sum of MPL of all items /no. of items x 100

Learning
The Angoff method
Questions Points (one point
each)
MPL by experts (% will answer
correctly)
1 1 0.6
2 1 0.7
3 1 0.9
↓ 1 1.0
100 1 0.8
100 Points Sum (example = 65.5)
MPL for whole test = (65.5 / 100) x 100 = 65.5

Learning
Criterion-referenced System - Rubrics for ORAL / ESSAY TESTS
Oral /
Essay
Questions
No correct
answers
whatsoever (0
point)
With correct
answers but
less than 50%
complete (1
point)
With correct
answers more than
50% but less than
100% complete (2
points)
With 100%
complete
correct
answers (3
points)
Q1
Q2
Q3
Q4
Q5

Learning
Items /
Criteria
Level of
performance
with
DESCRIPTOR
and
Points or Score
Assigned
Level of
performance
with
DESCRIPTOR
and
Points or Score
Assigned
Level of
performance
with
DESCRIPTOR
and
Points or Score
Assigned
Level of
performance
with
DESCRIPTOR
and
Points or Score
Assigned
Level of
performance
with
DESCRIPTOR
and
Points or Score
Assigned
1
2
3
4
5
Criterion-referenced System – Rubrics for OBSERVATION and
PROJECT-BASED TOOLS

Learning
Examples of Scoring and Grading Sheets
being used by different institutions

Learning
a
Grade Point
Equivalence
Letter Grade
Equivalence
% Equivalence
4.0 A 92–100%
3.5 B+ 87–91%
3.0 B 83–86%
2.5 C+ 79–82%
2.0 C 75–78%
1.0 D 70–74%
0.0 F below 70%
Letter Grade System

Learning
Grade Point Equivalence Equivalence Description
1.00 96–100% Excellent
1.25 94–95% Very Good
1.50 92–93% Very Good
1.75 89–91% Good
2.00 87–88% Good
2.25 84–86% Good
2.50 82–83% Fair
2.75 79–81% Fair
3.00 75–78% Pass
5.00 Below 75% Failure
INC Incomplete
Grade Point System

Learning
Latin Honors
Grade Point
Equivalence Range
Equivalence
Summa Cum Laude 1.00–1.50 92% and above
Magna Cum Laude 1.51–1.85 88.5% to 91.9%
Cum Laude 1.86–2.10 86% to 88.4%
Latin Honors
Grade Point Equivalence
Range
Summa Cum Laude 3.87–4.00
Magna Cum Laude 3.70–3.86
Cum Laude 3.50–3.69

Learning
Subjects Ratings
1 Biochemistry 84
2 Anatomy and Histology 86
3 Microbiology 85
4 Physiology 89
5 Legal Medicine, Ethics, and Medical Jurisprudence 88
6 Pathology 81
7 Pharmacology and Therapeutics 82
8 Surgery and Ophthalmology, Otorhinolaryngology and Rhinology 85
9 Medicine 82
10 Obstetrics and Gynecology 82
11 Pediatrics and Nutrition 93
12 Preventive Medicine and Public Health 81
GENERAL AVERAGE RATING 84.83 (Ʃ/12)
REMARKS PASSED
Philippine Physician Licensure Examination – Sample Complete Examination Report
To pass the examination, an examinee must obtain a general average rating of seventy five
percent (75%) without a grade lower than fifty percent (50%) in any subject.

Learning
Subjects Ratings
1 Biochemistry 84
2 Anatomy and Histology 86
3 Microbiology 85
4 Physiology 89
5 Legal Medicine, Ethics, and Medical Jurisprudence 88
6 Pathology 81
7 Pharmacology and Therapeutics 82
8 Surgery and Ophthalmology, Otorhinolaryngology and Rhinology 85
9 Medicine 82
10 Obstetrics and Gynecology 82
11 Pediatrics and Nutrition 93
12 Preventive Medicine and Public Health 81
GENERAL AVERAGE RATING 84.83 (Ʃ/12)
REMARKS PASSED
Philippine Physician Licensure Examination – Sample Complete Examination Report
To pass the examination, an examinee must obtain a general average rating of seventy five
percent (75%) without a grade lower than fifty percent (50%) in any subject.
Note the use of the word
“rating,“ same as “grade”.
General average rating = general
average grade

Learning
Educational Measuring Tools for Medical Students’ Learning and
Performance - Scoring and Grading
ROJoson Thoughts, Perceptions, Opinions and Recommendations
There is a need to measure the learning and performance of medical
students after the course of study and thereafter, to decide whether or
not to certify them as graduates who are qualified to practice
medicine in the Philippines (that is, after passing the Philippines
Physician Licensure Exam).
Certifying the students to graduate means they have acquired the
needed competencies to practice as basic-generalist physicians in the
Philippines.

Learning
Certifying the students to have acquired the needed competencies to
practice as basic-generalist physicians in the Philippines means they
have mastered the professional contents of the medical curriculum.
Professional contents cover the cognitive knowledge, skills and
attitude domains required of basic-generalist physicians, not just
passing written tests.

Learning
The educational measuring tools for medical students’ learning and
performance are not perfect and probably will never be perfect,
whatever they are.
Despite this imperfection, they should still be administered because
there is a need to measure, to evaluate and to determine who are
eligible to be basic-generalist physicians.

Learning
However, the tools should be CREDIBLE and VALID.
They should not ARBITRARY and not using the norm-referenced
scoring and grading (using mean or average as a passing score, so
called grade on a curve).
Criterion-referenced scoring and grading must be used.

Learning
Even the criterion-referenced scoring and grading are not perfect. But
at least, students are assessed and evaluated against a set of criteria
or standards on mastery of the professional contents of the medical
curriculum.

Learning
The standard setting processes utilized for such purposes generate
absolute standards, in contrast to relative standards where students
are judged against each other as is being done in norm-referenced
system.
These standards are considered absolute, because they are expressed
in terms of how much content the students need to know.

Learning
However, to repeat, there is no “gold standard” for setting passing
scores, in the standard setting in the criterion-referenced system.
Usually the choice of the standard setting method is based on the
available resources and the practical realities of the educational
environment. As such, it is critical to document all procedures used in
establishing the passing standard, especially in a language the school
is willing to make public.

Learning
Every effort should be exerted to use one of the standard setting
approaches described in the literature so as to make the passing
scores defensible and meaningful.
Also important is the continual improvement of the scoring and
grading systems. There should be evaluation and review of the
systems at planned intervals.

Learning
One specific activity recommended in the continual evaluation and
review of the scoring and grading systems is the so-called ITEM
ANALYSIS.
Item analysis is a process which examines student responses to
individual test items (questions) in order to assess the quality of those
items and of the test as a whole.

Learning
Item analysis typically focuses on four major pieces of information:
item difficulty, item discrimination, item distractors and response
frequency.

Learning
Item analysis is especially valuable in improving items which will be
used again in later tests, but it can also be used to eliminate
ambiguous or misleading items in a single test administration.
In addition, item analysis is valuable for increasing instructors’ skills in
test construction and identifying specific areas of course content
which need greater emphasis or clarity.

Learning
Last words:
An exam standard is ultimately just a recommendation.
The final evaluation of a student’s performance lies with educators
and educational institutions.
No exam is perfect: an educator will always know more about a
student than any exam ever could.
When considering a student’s performance on a summative exam,
educators should take into account the body of evidence that student
has built up during the duration of the academic career.

Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3)

Recommended

Recommended

More Related Content

Similar to Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3)

Similar to Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3) (20)

More from Reynaldo Joson

More from Reynaldo Joson (20)

Recently uploaded

Recently uploaded (20)

Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3)

Editor's Notes