The document provides guidance on designing and using short-answer and essay test items to assess students' knowledge and skills. It offers tips for writing effective test items, guidelines for scoring student responses, strategies for fairly grading essays, and advice for giving feedback to students on their exam performance. The key recommendations include saving essays for higher-order thinking, giving students advice on test-taking, considering response layout and grading criteria when writing items, and promptly returning and reviewing exams with students.
2. 2
Objectives
• Define these test items
• List advantages and disadvantages
• Discuss strategies for improving short-
answer/essay items
3. 3
Description
• Short-answer items ask students to define a
term or concept, or state the importance of
some idea or event
• For example: list three reasons why …
• Essay items let students display their
overall understanding of a topic and
demonstrate their ability to think critically,
organize their thoughts, and be creative and
original
4. 4
• Advantages
– They are easier to design than multiple
choice items
– Essay items are the best measures of
students’ skills in higher-order thinking and
written expression
• Disadvantages
– More difficult and time consuming to score
– Suffer from unreliable grading (grades on the
same response may vary from reader to
reader and from time to time by the same
reader)
5. 5
Strategies for improvement when
short-answer/essay items are used
• Save essays items for testing higher levels
of thought (application, synthesis, and
evaluation) and use appropriate tasks
including:
– Comparing: e. g. Identify similarities and
differences between …
– Relating cause and effect: e. g. What are the
major causes of …
6. 6
– Justifying: e. g. Explain why you agree or
disagree …
– Generalizing: e.g. State a set of principles that
can explain the following events.
– Inferring: e. g. How would character X react to
the following?
– Creating: what would happen if …?
7. 7
– Applying: e. g. Describe a situation that
illustrates the principle of …
– Analyzing: e. g. Find and correct the reasoning
errors in the following clinical case
– Synthesizing: e. g. Describe a plan for proving
that …
– Evaluating: e.g. Assess the strengths and
weaknesses of …
8. 8
• Give students advice on how to approach
an essay or short-answer item
• The following are some suggested examples of
advice from the medical education literature:
– Survey the entire test quickly
– Divide the time available among the questions
9. 9
– Analyze each question and its parts
– Outline each answer before you begin to write
– If the question is too hard for you, jot down
anything you can think of that could possibly
be relevant
– Write a thesis statement that expresses your
main point or conclusion
10. 10
– Follow your outline as you write
– Support your point of view with specific
examples and relevant evidence
– Reread your exam before you turn it in
– If you run out of time, outline your main points
and examples and write “ran out of time”
11. 11
• Don’t give students a choice of questions to
answer
– Some students will waste time trying to decide
which questions to answer
– You will not know whether all students are
equally knowledgeable about all the topics
covered on the test
– Some questions are likely to be harder than
others and the test could be unfair
• Include more than one items in a test with
only essay items (wider sampling of test
items more reliable and valid)
12. 12
• Write effective Test Items
– State the question clearly and precisely
– Avoid vague questions that could lead students
to different interpretations
– Avoid so general questions that no two
students will answer alike (serious difficulties in
equitably grading the responses)
– The use of the word how or why in an essay is
associated with students being better able to
develop a clear thesis (Tollefson, 1998)
13. 13
Some examples of short-answer and essay
items
Poor:
Why does an internal combustion engine work?
Better:
Explain the functions of fuel, carburetor, distributor,
and the operation of the cylinder’s components in
making an internal combustion engine run
14. 14
More examples …
Poor:
Did the patient in the above clinical case require
digoxin or an ACE inhibitor? Why do you think so?
Better:
Decide whether the patient in the above clinical
case required digoxin or an ACE inhibitor. Support
your position by identifying and explaining specific
pharmacokinetic and pharmacodynamic properties
of a drug suitable for treating heart failure
15. 15
• Consider the layout of the item
• Write out the correct answer yourself
• This will help you to:
– Use your version to revise the item as needed
– Estimate how much time students will need to
complete the item
– Use estimates in determining the number of
items to include on the exam
– Give students advice on how much time to
spend on each item
16. 16
• Decide on guidelines for full and partial
credit
– Decide which specific facts or ideas a student
must mention to earn full credit and how you
will award partial credit
17. 17
An example scoring guide
Full credit – six points
The essay clearly states a position, provides
support for the position, and raises a counter
argument or objection and refutes it. Evidence is
both persuasive and original. Counterargument is
significant. The essay contains no extraneous
information
18. 18
Five points:
The essay states a position, supports it, and
raises a counterargument and refutes it. The
essay contains one or more of the following
rugged edges: evidence is not uniformly
persuasive, counterargument is not a serious
threat to the position, some idea seem out of
place
...
One point:
The essay does not state the students position
on the issue. Instead, it restates the position
presented in the question and summarizes
evidence discussed in class or in the reading
19. 19
• Grade and evaluate items fairly
– Read the exams without looking at students’
names (not to bias your grading)
– Skim all the exams quickly, without assigning
any grades (get an overview of the general
level of performance and the range of students’
responses
– Choose examples of exams to serve as
anchors or standards (excellent, good,
adequate and poor) – refresh your memory of
standards ensuring fairness over the grading
period
20. 20
– Grade each exam question by question rather
than grading all questions for a single student
(distribute fatigue factor randomly)
– Avoid judging exams on extraneous factors
(handwriting, use of a pen or pencil)
– Write comments on student exams, indicate
what they have done well and where need to
improve (feedback)
21. 21
– Read only a modest number of exams at a
time (take up short breaks to keep up
concentration, try to set limits on how long to
spend on each paper)
– If you can, read some of the papers twice (wait
two days or so, rereading helps increase
reliability of grader)
– Place the grade on the last page of the exam
(protect privacy when returning exams)
– If postgraduate student instructors assist in
grading, set up standardized procedure
22. 22
• Returning Essay Exams
– Return exam promptly (reinforce learning,
capitalizes on students’ interest in results,
within a week or so)
– Review the exam in class (what a good answer
included, the commonest errors the students
made, distribution of scores
23. 23
– Use groups to discuss test questions (small
groups, unresolved questions brought up to
class as a whole)
– Get feedback from the class about the test
(pass along to next year’s class tips on specific
skills and strategies this class found effective)
– Keep a file of essay questions (include copy
with annotations on ways to improve, mistakes
made, student comments, copies of good and
poor exams
24. SUMMARY
• Returning Essay Exams
• Grade and evaluate items fairly
• Decide on guidelines for full and partial credit
• Consider the layout of the item
• Save essays items for testing higher levels of thought
(application, synthesis, and evaluation) and use
appropriate tasks including:
• Give students advice on how to approach an essay or
short-answer item
• Write effective Test Items
• Grade and evaluate items fairly
24
25. 25
Conclusions
• A well-developed knowledge base provides an
important foundation upon which clinical
competence and medical expertise are built
• Written examinations assess medical knowledge
and its application in a reliable and efficient
manner
• They are appropriately applied to assessing the
proficiency of individual trainees and the overall
success of educational programs in attaining
important knowledge objectives
26. 26
• Thus, they remain an important tool even
in CBE (Competence based examination)
• However, inappropriate emphasis on
written examinations as assessment tool
may have negative influence on students’
attention to other aspects of clinical
competence (e.g. communication skills,
teamwork, practice-based learning and
improvement)
27. 27
• As we introduce complementary assessment tools
(e.g. OSCE (Objective session clinical Examination),
OSPE(Objective session Practical Examination) , Trainee
observations and feedback), when we implement the
new curriculum, we will continue using written
examinations to assess knowledge application
• But we need to improve their reliability and validity
– Test blue printing
– Effective one-best-answer MCQs (“application”)
– Reserve short-answer/essay items for knowledge
application which cannot be assessed by MCQs
