Good assessments should demonstrate validity, reliability, and equivalence. The appropriate characteristics depend on the assessment's purpose (formative or summative) and stakeholders (learners, teachers, patients, regulators). Formative assessments aim to enhance learning and should strongly exhibit catalytic and educational effects, while summative assessments evaluate learning and require high validity, reliability, and equivalence. Stakeholder priorities also vary, such as learners prioritizing educational effect and regulators prioritizing validity.

1. Criteria for Good Assessment
John Norcini, Ph.D.

2. Overview
 Some background
What is a „test‟ or assessment?
How does assessment affect learners?
How are assessments changing?
Who cares about assessment?
 Characteristics of good assessment
How do they relate to purpose?
How do they relate to the stakeholders?

3. What is a test or assessment?
 Two definitions of „test‟
„To discover the worth of
something by trial‟
 Summative, evaluation
„To improve the quality of
something by trial‟
 Formative, assessment
 For today, a „test‟ or
„assessment‟ is any
measurement process
Formative or summative
 These are central to
Understanding the importance
of assessment
Understanding the application
of assessment
Identifying criteria for good
assessment

4. How does assessment affect
learners?
 Assessment is a powerful tool in education
Motivates students to learn
Directs learning by signaling what it important
 Unintended effects when there is a mismatch between
curriculum and assessment
Ensures that learning has occurred
Creates learning

5. How are assessments changing?
Past
“That part of eternity with some small fraction of
which we have a slight and regrettable
acquaintance.”
A Bierce
 Little realism
 Little person/patient involvement
 Lack of structure during workplace
training
 Episodic and infrequent
continuing education
Future
“That period of time in which our affairs
prosper, our friends are true, and our happiness
is assured.”
A Bierce
 Authenticity
 Active person/patient involvement
 More structured workplace
training
 Life-long learning and quality
improvement

6. Who cares about assessment?
 Variety of stakeholders
Students
 Key to learning and demonstration of competence
Teachers and individual/collective educational institutions
 Accountability and self-assessment
Person/patients and healthcare systems
 Accountability and efficiency
Regulatory bodies
 Accountability and maintenance of competence

7. Characteristics of good assessment
 No single set of criteria apply equally to all
assessment situations
Different purposes may lead to different values for the
criteria
Different stakeholders and different mixes of
stakeholders may value the criteria differently
 Ottawa conference produced a set of criteria and
how they are modified by purpose and
stakeholders

8. Characteristics
 Validity or coherence
Is there a body of evidence
that „hangs together‟ and
supports the use of
assessment for a particular
purpose
 Is a property of the inferences
drawn from a test, not the test
itself
 Is a matter of degree
 Requires the ongoing collection
of data
“In seeking absolute truth we
aim at the unattainable,
and must be content with
finding broken portions.”
William Osler, 1889

9. Characteristics
 Reliability or reproducibility
Will scores be the same if
examinees are reassessed?
History of reliability
 Test-retest reliability
 Alternate form reliability
 Split-half reliability
 Reliability index
“Arithmetic…is a bad
master, but it is a good
servant.”
Anonymous
Lancet, 1921

10. Characteristics
 Equivalence
Different versions of an
assessment yield equivalent
scores or decisions
A challenge for assessment
in the workplace
 Catalytic effect
The assessment provides
results and feedback in a
fashion that enhances
learning
A requirement for formative
assessment

11. Characteristics
 Educational effect
The assessment motivates
those who take it to prepare
in a fashion that has
educational benefit
 How do students prepare for
the test?
 Acceptability
Stakeholders find the
assessment process and
results to be credible
 Feasibility
The test is practical, realistic,
and sensible, given the
circumstances and context

12. Characteristics and Purpose
Formative Summative
Validity-coherence ++ +++
Reproducibility-consistency + +++
Equivalence + +++
Educational effect ++ ++
Catalytic effect +++ +
Feasibility ++ ++
Acceptability ++ ++

13. Characteristics and Stakeholders
Examinees-Formative Examinees-Summative
Validity-coherence ++ +++
Reproducibility-consistency + +++
Equivalence + +++
Educational effect +++ +
Catalytic effect +++ +
Feasibility ++ ++
Acceptability +++ ++

14. Characteristics and Stakeholders
Teacher-Institution
Formative
Teacher-Institution
Summative
Validity-coherence ++ +++
Reproducibility-consistency + +++
Equivalence + +++
Educational effect ++ ++
Catalytic effect +++ +
Feasibility ++ ++
Acceptability ++ ++

15. Characteristics and Stakeholders
Persons/patients Healthcare System
Validity-coherence +++ +++
Reproducibility-consistency +++ +++
Equivalence +++ +++
Educational effect + +
Catalytic effect + ++
Feasibility + ++
Acceptability + +

16. Characteristics and Stakeholders
Regulators
Validity-coherence +++
Reproducibility-consistency +++
Equivalence +++
Educational effect +
Catalytic effect ++
Feasibility ++
Acceptability ++

17. Summary
 Characteristics of good assessment
Validity, reliability, equivalence, catalytic effect,
educational effect, acceptability, feasibility
 Influenced by the purpose
Formative and summative
 Influenced by the stakeholders
Learners, teachers-institutions, patients-healthcare
system, regulators