The document discusses assessment methods and principles in medical education. It begins by defining assessment as determining the extent to which students achieve instructional objectives. It then outlines Miller's pyramid of assessment as a framework, with levels including "knows," "knows how," "shows how," and "does." Different assessment tools are suited to different levels of the pyramid. The document also covers principles of assessment like reliability, validity, and feasibility. It describes factors that influence reliability and different types of validity. Overall, the summary emphasizes the importance of selecting appropriate assessment methods based on learning objectives and ensuring assessments are reliable, valid, and feasible.

1. 25/01/2023
1
Assessment:
methodsandits
principles
1
Professor Dr Shabana Ali
Head of Anatomy Department
Assistant Dean Medical Education
Riphah International University
Shabana.ali@riphah.edu.pk
Outline of Session
Discuss assessment
methods in relation
to Miller’s pyramid
Explain various
Principles of
assessment
1/25/2023 2
1 2

2. 25/01/2023
2
What does
assessment
mean?
• It is a process of
determining the extent to
which students have
achieved the instructional
objectives.
1/25/2023 3 1/25/2023 4
3 4

3. 25/01/2023
3
What to assess?
1/25/2023 5
• Knowledge
• Relevant Knowledge
• Application of Knowledge & problem
solving
• Skills
• Clinical: History , Patient examination,
Procedural skills
• Communication skills
• Critical Reasoning Skills: Data
interpretation→ decision- making
• Attitudes and behaviors
1/25/2023 6
We teach
5 6

4. 25/01/2023
4
Miller stated that
• No single assessment
method can provide all the
data required for judgment
of anything so complex as
the delivery of professional
services by a successful
physician.
Miller’s Pyramid of
Assessment
• Miller’s Pyramid of Assessment
provides a framework for
assessing clinical competence
in medical education and can
assist clinical teachers in
matching learning outcomes
with expectations of what the
learner should be able to do at
any stage
1/25/2023 8
7 8

5. 25/01/2023
5
Miller proposed a pyramidal structure with four
levels, each of which required specific methods
of assessment.
Does
Shows how
Knows how
Knows
Behaviour
Cognition
Professional
authenticity
1/25/2023
Miller’s pyramid of clinical competence
1/25/2023 10
9 10

6. 25/01/2023
6
DOES : Mini-CEX, DOPS, Checklist, 360
degree evaluation, Logbooks, Portfolios
SHOWS HOW: Long / Short Case,
OSCE, OSPE, SP- based test
KNOWS HOW; One best MCQ, MEQs,
Oral exam
Knows : True and false , MCQ,
essay, oral
1/25/2023 11
How to assess?
1/25/2023 12
11 12

7. 25/01/2023
7
We use various assessment tools
during
Written
Exams
Practical
Exams
Clinical
Exams
Viva Voce
13
1/25/2023
Medical Teachers/supervisors must
be skilled to choose, develop
assessment items as well as in scoring
and interpreting results along with
future planning
13 14

8. 25/01/2023
8
Principles which forms general framework on
which assessment may be based…
1/25/2023 15
The purposes of assessment must be identified before selecting
the assessment method
The priority…. state the learning outcomes first ……is to determine
what is to be assessed ……the foundation of assessment
Select appropriate tool for assessment according to purpose and
examiner should be aware of strengths and limitations of tool.
A variety of techniques is required for comprehensive
assessment in single exam.
What helps you while deciding and
designing about assessment?
Selection of appropriate grading and reporting system and
incorporation of feedback
Selection of criterion for judgment of performance
Adequate test size selection required to assess the student
performance
Clarity about learning outcomes and variety of available
assessment tools
15 16

9. 25/01/2023
9
Principles
of
assessment
1/25/2023 17
Task
1/25/2023
17 18

10. 25/01/2023
10
Characteristic of a good method of
assessment
Validity Reliability Feasibility
Educational
impact
Acceptability
Cost
effective
Objectivity
1/25/2023 19
Cees Van Der Vleuten
&
Utility Index
validity x reliability x feasibility x cost
x Educational impact x acceptability
1/25/2023 20
According to GMC optimizing any assessment tool/
program is about balancing these six components of
the utility index.
19 20

11. 25/01/2023
11
Reliability
1/25/2023 21
Reliability
▪Consistency/repeatibility of test results
▪Students will get same marks
▪on repeating the test
▪On changing the examiner
▪The consistency of results indicate that test items are
relatively free of errors and we can rely on these.
21 22

12. 25/01/2023
12
Factors that affect reliability of the test
Test factor
Teacher and
student factor
Environment
factor
Test
administration
factor
Marking factor
Test Factor
•In general, longer tests produce higher
reliabilities. Due to the chances of
guessing, the scores will be more accurate if
the duration of the test is longer.
•A valid test is said to be reliable but a
reliable test need not be valid. A consistent
score does not necessary measure what is
intended to measure.
23 24

13. 25/01/2023
13
The teacher and student Factor
•In most tests, it is normal for teachers
to construct and administer tests for
students.
• The factors that contribute to positive
effects to the reliability of a test include
teacher’s encouragement, positive
mental and physical condition,
familiarity to the test formats, and
motivation
Environmental factor
•Any favourable environment with comfortable
chairs and desks, good ventilation, sufficient
light and space will improve the reliability of the
test.
•On the contrary, a non-conducive environment
will affect test-takers’ performance and test
reliability
25 26

14. 25/01/2023
14
Administrative Factor
•Test administrators should strive to provide
clear and accurate instructions, sufficient
time and careful monitoring of tests to
improve the reliability of their tests. A test-
retest technique can be used to determine
test reliability.
Marking Factor
•Human judges have many opportunities to
introduce error in scoring essays (Linn & Gronlund, 2000).
•Different teachers award different marks for the
same answer.
•A marker’s assessment may vary from time to
time and with different situations. Conversely, it
does not happen to the objective type of tests
since the responses are fixed. Thus, objectivity is a
condition for reliability.
27 28

15. 25/01/2023
15
Methods to assess reliability of a test
Test
retest
Inter rater Intra rater
Test
retest
The same test is re-administered to the
same people. It is expected the
correlation between the two scores of
the two tests would be high. However,
the effect of practice as well as
memory may influence the correlation
value. Market, industry, equipment
1/25/2023 30
29 30

16. 25/01/2023
16
Inter
rater
Two or more judges or raters are involved
in grading. A score is more reliable and
accurate measure if two or more raters
agree on it.(Examiner + tool)
Intra
rater
The consistency of grading by a single
rater at the same time. When a rater
grades tests at different time, he/she may
become inconsistent in grading for various
reasons. (Examiner + tool)
1/25/2023 31
RELIABILITY
• Is concerned with how consistent test scores
are from one test to another.
• Refers to results obtained with an assessment
instrument.
• Is usually measured by a computer. The name of
the method is called Kuder- Richardson or
Cronbach’s Alpha Reliability.
1/25/2023 33
31 33

17. 25/01/2023
17
1/25/2023 34
The value of
Crobach’s alpha
varies from 0 to 1.
An internationally
minimum acceptable
value is 0.7
• An alpha value of 0.8 is
considered to be good.
Reliability
improves
with:
1/25/2023 35
• Increasing testing time
• Computer marking
• Good item design
• Item-banking and
• Examiner training
34 35

18. 25/01/2023
18
Reliability of various assessment tools
Assessment tool Reliability
MCQ 0.7-0.9
OSCE 0.6-0.8
Essay Question 0.4-0.6
1/25/2023 36
Validity
36 37

19. 25/01/2023
19
For validity of results
Constructing a representative sample of test items
Deciding the format and tools for assessment aligned with
objectives
Clearly defining the objectives for teaching anatomy of upper
limb
Validity refers to whether the test is actually
measuring what it claims to measure
(Arshad, 2004).
Validity is “the extent to which
inferences made from assessment
results are appropriate, meaningful,
and useful in terms of the purpose of
the assessment” (Gronlund, 1998).
38 39

20. 25/01/2023
20
Types of validity
1/25/2023 40
• Test looks like a test even at first impression.
Face validity
• The test matches with the topics and the
learning objectives”…. Table of specification
Content validity
• The test is measuring what it is supposed to
measure
Construct validity
Predictive validity
Face Validity
•The degree to which a test looks
right and appears to measure the
knowledge or abilities it claims to
measure, based on the
subjective judgement of the
examinees who take it. Mousavi
(2009) .
• Upper limb test must display all major
components of upper limb
40 41

21. 25/01/2023
21
Content
Validity
•whether or not the content of the
test is sufficiently representative
measure of what it is supposed to
measure” (Henning, 1987).
• all content domains are presented in
the test.
• use of Table of Test Specification
(C1,C2,C3 LEVEL Los)
• that can give detailed information
knowledge, skills and attitude from
whole content.
Construct
Validity
•The most obvious reflection of
whether a test measures what it is
supposed to measure
•In other words, construct validity
refers to whether the underlying
theoretical constructs that the test
measures are themselves valid
•Personality trait inventory must
cover all theoretical components of
personality traits
•Module assessment
42 43

22. 25/01/2023
22
VALIDITY
1/25/2023
44
Validity is essentially
concerned with the:
• The content of the assessment
(are the right things being tested?)
• The effect of learning (are the
students learning what we want
them to learn?)
• The assessment methods (are we
using the methods for the
purpose?)
1/25/2023
45
In other words it is the
degree to which the test
‘truly’ measures what it is
intended to measure.
It also includes an
evaluation of the adequacy
and appropriateness of the
uses that are made of the
assessment result
44 45

23. 25/01/2023
23
validity
improves
with:
1/25/2023 46
• Defining the curriculum /syllabus
• Mapping exam content against the curriculum
• Emphasis on essential and important curriculum content
• Selecting appropriate methods
FEASIBILITY
• An exam should be cost beneficial and easy to score. It
should not be a burden on financial and human
resources of the organization.
• Feasibility affects the choice of assessment instruments
and aspirations for high validity and reliability, e.g.
Resources to deliver the tests and acceptability to the
trainees.
• OMR machine
1/25/2023 47
46 47

24. 25/01/2023
24
The
Authenticity
•AN AUTHENTIC TEST…
•contains language that is as
natural as possible.
• has items that are
contextualised rather than
isolated.
•includes meaningful,
relevant, interesting topics.
•Simulation of real-world
tasks
The Objectivity
• An exam is objective if answers to the questions are
predetermined before exam.
•The resulting scores are not influenced by the
examiner’s skills and emotions.
•Oral exam, SEQS, MCQS
•Most standardised tests of aptitude and achievement
are high in objectivity.
48 49

25. 25/01/2023
25
The Objectivity
•In fact, such tests are usually constructed so that
they can be accurately graded by trained
examiners and scoring machines.
•The test is said to have high objectivity when the
examiner is able to give the same score to the
similar answers guided by the mark scheme.
Objectivity
•An objective test has the highest level of objectivity due
to the scoring that is not influenced by the examiner’s
skills and emotions.
•The subjective test has the lowest objectivity. Based on
various researches, different examiners tend to award
different scores to an essay test. The same examiner
would give different scores to the same essay if s/he is
to re-check at different time.
50 51

26. 25/01/2023
26
Cost effective
•A practical test stays within budgetary limits.
•appropriately utilises available human resources.
•does not exceed available material resources.
•considers the time and effort involved for both
design and scoring.
1/25/2023 53
52 53

27. 25/01/2023
27
Good assessment policy
1/25/2023 54
A good assessment method/ tool/
technique should be
▪ Reliable: Same result on test retest
▪ Valid: Tests what should be measured
▪ Practical/ Feasible
▪ Cost effective :
▪ Objective: Decreases Bias
1/25/2023 55
54 55

28. 25/01/2023
28
Characteristics of Different Tests
56
Reliable Valid Standar
dized
Objective Practical
1/25/2023 1/25/2023 57
56 57

29. 25/01/2023
29
58
1/25/2023
Steps in developing
assessment plan
• Outline the course content and learning
outcomes.
• Identify teaching/learning activities
• Develop a grade plan based on the
guidelines.(CRT/NRT)(Formative/summative)
• Determine assessment tools (miller’s
pyramid) that are appropriate to measure
the achievement of course content and
objectives.
1/25/2023 59
58 59

30. 25/01/2023
30
Steps in developing
assessment plan
• Determine what content and course objectives to
include on each assessment. (modules /
regions/blocks)
• Develop blueprint to address course content and
objectives.
• Schedule exams and due dates for other assignments..
• Include the above information in the course syllabus.
• Discuss all of the above information with the students.
1/25/2023 60
Summary ?
1/25/2023 61
60 61

31. 25/01/2023
31
Thank You
1/25/2023 62
62