Presentation from an assessment workshop at the Institute of Veterinary, Animal and Biomedical Sciences, Massey University, NZ

1. Writing short and long answer
exam questions
Liz Norman
Massey University

2. “Effective item writers are trained,
not born” (Downing, 2006)
Steven M Downing (2006) Twelve steps for effective test development, In: Downing SM and Haladyna TM (Eds), Handbook of Test Development, Lawrence Erlbaum
Associates, Mahwah, NJ, USA, p.11

3. Breadth and level of the questions
• Are the questions focused on middle to higher
level tasks such as creation, evaluation,
analysis and application?

4. Recall-higher order classification
Fact recall Questions capable of being answered by
reference to one paragraph in a text or notes
(or several paragraphs for questions requiring
recall of several facts)
Applied
(higher order)
Questions that require the use of facts or
concepts, the solution of a diagnostic or
physiologic problem, the perception of a
relationship, or other process beyond recalling
discrete fact
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical
students. Academic Medicine, 65(9), S59-60.

5. Knowledge-understanding
classification
Knowledge knowing about; a body of coherent facts; can be
thought of as right or wrong.
Understanding knowing how and why; the meaning of facts; the
theory that links facts and provides meaning; how
sense is made of facts to enable them to be applied
to analysis, synthesis, evaluation; to be able to
explain why particular facts or skills are applicable
to a particular situation; to know which fact to
apply when; to be able to create new knowledge or
modify or adapt an idea to a new situation.
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.

6. SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.

7. Prestructural Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash, et al. (2010). Adv Physiol Educ, 34(3), 145-149.

8. Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
Quantitative change
Qualitative change

9. Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
surface
deep

10. Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
recall
application

11. Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
knowledge
understanding

12. Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple
instructions, calculate, reproduce, arrange, recognise, find, note,
seek, sketch, pick
Multistructural Combine, classify, structure, describe, enumerate, list, do
algorithm, apply method, account for execute, formulate, solve,
conduct, prove, complete, illustrate, express, characterise
Relational Analyse, compare, contrast, integrate, relate, explain causes,
apply theory (to its own domain), argue, implement, plan,
summarize, construct, design, interpret (some senses), structure,
conclude, substantiate, exemplify, derive, adapt
Extended
abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory
(to new domain), assess, evaluate, interpret (some senses),
critically reflect, predict, criticise, reason
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.

13. Webb’s depth of knowledge
• Level 1 – recall
– Recall information or perform a simple step, identify, measure,
describe, explain simple ideas
• Level 2 – skill/concept
– Requires more than one step, comparing, interpreting,
estimating, making observations, explaining, organising and
displaying data
• Level 3 – strategic thinking
– Requires planning and using evidence, explaining reasoning,
make conjectures, draw conclusions, solving problems
• Level 4 – extended thinking
– Complex reasoning, planning developing and thinking over an
extended period of time, making multiple connections,
synthesis of ideas into new concepts
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25

14. Breadth and level of the questions
• Are the questions focused on middle to higher
level tasks such as creation, evaluation,
analysis and application?
• Are the questions at the right level of
difficulty?

15. Are the questions at the right level of
difficulty?
• There needs to be a match between the depth
of knowledge and degree of skill (both
practical and cognitive) required to answer
the question, and those specified for the topic
in the learning outcomes.
• Learning outcomes specify what students are
able to do when they have completed the
course—so you need to think about the level
of study of the student
• Day one competencies

16. Some factors that affect demand and
difficulty
• Type of operation
• Degree of novelty
• Number of components or ideas involved
• Whether resources are provided or need to be
generated by the candidate
• The question wording and any images, diagrams,
or tables provided
• Degree of abstraction
• Response strategy – simple, stepwise, integrated

17. For EACH ECG, make a recommendation for
management of the horse and explain your
reasoning.

18. Question 1:
Explain the physiological actions of insulin.
Question 2:
Explain the physiological actions of ghrelin.

19. Question 1:
Compare and contrast the clinical signs of
hypoadrenocorticism with those of
hyperadrenocorticism in dogs.
Question 2:
Compare and contrast the clinical signs of
diabetes mellitus with those of
hyperadrenocorticism in dogs.

20. Question 3:
Compare and contrast the clinical signs of
diabetes mellitus with those of
hyperadrenocorticism in dogs and cats.
Question 4:
Compare and contrast the clinical signs of
diabetes mellitus with those of
hypoadrenocorticism in dogs and cats.

22. Describe the skull. (30 marks)

23. Question 1
Discuss the use of insulin for the treatment of diabetes
mellitus in cats (25 marks)
Question 2
a) Describe the advantages and disadvantages of insulin
therapy for diabetes mellitus in cats (10 marks)
b) Indicate the dose and frequency of administration of
insulin you would prescribe to a newly diagnosed cat with
diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the
frequency and timing of feeding in relation to insulin
dosing in cats with diabetes mellitus (10 marks).

24. You have been contacted by a farmer producing Pacific
oysters (Crassostrea gigas) intertidally, in a bay containing
a number of oyster farms. The farmer is concerned with
the amount of dead shell they are seeing during the
current grading. Explain how you would approach this
scenario. (20 marks)
Include in your answer how the information you could
gather might influence your assessment, what differential
diagnoses you consider and detail how you might further
investigate potential causes and what advice you would
provide.

25. Breadth and level of the questions
• Are the questions focused on middle to higher
level tasks such as creation, evaluation,
analysis and application?
• Are the questions at the right level of
difficulty?
• Do the questions provide the right breadth of
coverage and relative weighting of the
learning outcomes?

26. Example weighting system
Scores for impact and frequency are multiplied
to give a total weighting for each curriculum
area.
McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547.
Impact Weight Frequency Weight
Less important 1 Rarely seen 1
Essential 2 Relatively common 2
High impact 3 Very common 3

27. Examination structure
• Do the examination materials follow the exam
structure format as described in the paper
details?
• Are marks allocated to each question and
separate subpart?
• Are the marks allocated to each question and
each sub-part appropriate?
• Do the marks add up to the expected total?
• Is there sufficient time to answer all the
questions?

28. Pacing examinations
• Reading time: 40 words per minute (Klatt & Klatt,
2011).
• Writing time: 16-20 words per minute (Summers
& Catarro, 2003)
• Marking schemes help you check the pacing
• Set exam to be 1 mark = 1 minute
Klatt, E. C., & Klatt, C. A. (2011). How much is too much reading for medical students? Assigned reading and reading rates at one medical school. Academic
Medicine, 86(9), 1079-1083.
Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian
Occupational Therapy Journal 50(3): 148-157

29. Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian
Occupational Therapy Journal 50(3): 148-157

30. Question structure
• Is it clear for each question what is required in
terms of scope?
• Does each question contain an instructional
verb that explains the task required to the
student?

31. Don’t write questions; write tasks
What is your
diagnosis?
State the most likely diagnosis
State the most likely diagnosis
and explain your reasoning
Discuss the differential
diagnoses you would consider in
this case
or …..

32. Instructional verb examples
Compare: to find similarities between things, or to look for
characteristics and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject
matter, including all the main points, essential details, and pros and
cons of the problem, to show your complete understanding of the
subject.
Define: to provide a concise explanation of the meaning of a word
or phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions
or results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

33. Question structure
• Is it clear for each question what is required in
terms of scope?
• Does each question contain an instructional
verb that explains the task required to the
student? Questions should not really be
questions: they should be instructions instead.

34. Make short notes on THREE (3) of the following.
a) Pulmonary compliance.
b) Bicarbonate as a blood buffer.
c) The respiratory centre.
d) The respiratory functions of the nose.

35. Name two (2) diagnostic tests you would run next
to investigate the cause of this dog’s current
illness.

36. A dog is presented to your clinic after a road
traffic accident.
a) What clinical signs would be consistent
with urinary bladder rupture?

37. Outline and discuss a conceptual framework for
differentiating between ryegrass varieties
available in the New Zealand market place and
for defining possible strengths and weaknesses
of a particular cultivar. Your conceptual
framework should reflect the various options
currently employed by plant breeders in
developing new cultivars. (10 marks)

38. How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3

39. Question structure
• Is it clear for each question what is required in
terms of scope?
• Does each question contain an instructional
verb that explains the task required to the
student? Questions should not really be
questions: they should be instructions instead.
• Does each question contain ONLY relevant
content and no irrelevant content?

40. A client rings you up to arrange his quarterly visit as he is frustrated by ongoing
problems with his breeding herd. Some sows seem to take ages to cycle after weaning,
a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot
of stillborns. When you get to the farm, the farmer mentions that he has also had a
few sows go down at farrowing with a fever, they usually start panting and die, and
there’s one right now in the old farrowing room he wants you to take a look at. As you
walk through the farrowing rooms you notice that many of the sows, both expecting
and lactating, have swollen vulvas. The sow in question farrowed yesterday. She
clearly has a temperature and is panting. You also think she looks a bit anaemic and
note that she doesn’t appear to have any milk. You suspect what the problem is but
think it would be nice to confirm your diagnosis as, although it is suspected to occur in
Australia, the disease has never been definitively confirmed. You take a blood sample
and make a smear.
On your way home you drop the slide off at the lab and ask them if they will have a
look at it for you. Later that afternoon the pathologist rings up and (very) excitedly
tells you that after using Wright’s stain she spotted some cocci-like organisms attached
to the red blood cells. The pathologist has rung Biosecurity Australia and they are not
interested in pursuing this particular finding as they have always considered it to be
present.
You ring the farmer to tell him that you have confirmed your suspicion and that he
now needs to embark on the course of action you had discussed with him earlier.
Write the farmer a description of the disease and outline the short and long-term
course of action. Include in your report a brief discussion of what, if any, potential
there is for eradication. (25 marks)

41. Which of the following statements is TRUE?
a. A ventral hernia is classified as a false hernia
b. A diaphragmatic hernia is classified as a true hernia
c. Umbilical hernias are classified as false hernias
d. False hernias have a hernia sac
e. Umbilical hernias are relatively uncommon

42. Question structure
• Is it clear for each question what is required in
terms of scope?
• Does each question contain an instructional
verb that explains the task required to the
student? Questions should not really be
questions: they should be instructions instead.
• Does each question contain ONLY relevant
content and no irrelevant content?

43. Question structure continued
• Are any ancillary materials used (images,
radiographs, histopathology slides etc) fit for
purpose, clear, legible and of good quality?
• Is a marking scheme provided which indicates
the agreed standard by which students’
answers will be judged and marks awarded?
• Does the answer provided in the marking
scheme answer the question that is being
asked?
• Do you agree with the marking scheme?

44. Grammar and spelling
• Is the language used in each question clear
and grammatically correct?
• Are there any spelling mistakes?
• Are units given as SI units, or the units that
the student has been using in learning
activities?
• Are any numerical values used realistic and
correct?

45. http://www.slideshare.net/liznorman