Clarity and why its important Australian and New Zealand College of Veterinary Scientists Examiners Workshop 14-15 February, 2015

1. Clarity and why its important
Liz Norman
Massey University
Australian and New Zealand College of Veterinary Scientists
Examiner Workshop, 14-15 February, 2015

2. “Effective item writers are trained,
not born … “
Downing and Haladyna 2006, Handbook of test development ,p. 11

3. Validity
measuring what we
want to measure
and
not measuring what
we don’t want to
measure

4. What we don’t want to measure
• Ability to take tests
• Ability to write legibly and fast
• Ability to rote learn whole pages of textbooks or
review articles - prewriting
• Ability to write down a huge series of unconnected
facts in no particular order
• Ability to research examiner's fields of interest and
rote learn impressive aspects of that
• Ability to interpret what examiners are thinking

5. Why clarity is important
• We want the candidate to do the task we
envisaged, not something else
– Validity – measuring what we intend to measure
• We want to minimise irrelevant difficulty
– Validity – measuring what we intend to measure

6. Communication
The examination questions are the question setter’s
expression of the question setter’s task.
The candidate’s answer represents the candidate’s
expression of the candidate’s interpretation of the questions.
The marker evaluates the marker’s interpretation of the
candidate’s expression of the candidate’s answer.
The marker uses the marker’s interpretation of the setter’s
expression of the setter’s task to evaluate the candidate’s
answer.
Modified from Pollitt & Ahmed (1999) New Model of the Question Answering Process. IAEA. Bled, Slovenia

7. General issues with question
wording

8. Expectations and stereotypes
Examples:
• “treatment”
• all differential diagnoses vs those only applicable
in a particular case
• expectation that Qs will ask about what
something is rather than what it is not
• expectation of hard questions

9. Elephant Riding
Jan Kemp
Climbing up
the back of an elephant
you spring into
the toehold of its tail
held in place by the mahout
grab the ropes
strapped round its belly
& haul yourself up.
She rises
from buckled knees under you
moves like a ship
you’re high
under the hanging ashoka leaves
as you flow forward
her fly-bitten ears grey sails flap.
she flings the odd young-leaved branch
into her mouth
with her triumphant trunk.

10. Crisp et al. (2008). Tales of the expected: The influence of students' expectations on question validity
and implications for writing exam questions. Educational Research, 50(1), 95-115.
Year 11 Science exam Q

11. Discuss the diagnosis and management of a cat
with both chronic renal failure and hyperthyroidism.
(24 minutes)

12. Contextualising Qs
• Context is good because it brings relevance and
authenticity
• Allows assessment of concrete or specific examples
not abstract concepts or generalisations
• Allows assessment of applied learning (doing not
just knowing)
• All these carry with them a potential for bias.

13. Contextualising Qs
• Other disadvantages:
– More words used
– More intended/unintended demand
– Familiarity – schemas/sterotypes
– Focus may direct candidates to the wrong aspects
– Images can be particularly distracting

14. You are presented with a poorly controlled diabetic dog.
The dog is an 8 year old female spayed Labrador weighing
36 kg with a body condition score of 7/9. The dog has
been on porcine lente insulin (Caninsulin) twice daily for 3
months. Various doses have been tried during this time
and the dog is currently receiving 70 units twice daily. The
dog is still polyuric and polydipsic with 3+ to 4+ glucosuria.
Discuss the possible causes of poor control which you
should consider when evaluating this dog.

15. List the possible causes of consistently high blood
glucose values in a dog being treated with
subcutaneous insulin injections for the treatment of
diabetes mellitus. Outline your approach to
investigating these possible causes.

16. Question 1
A crate of 12 cans of cola costs $4.20. How much
do 7 crates of cola cost?
Question 2
A ski pass costs $4.20. How much would it cost for
7 days?
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus.
Assessment in Education: Principles, Policy & Practice, 14(2), 201-232.

17. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimental
investigationoffocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.

18. Guidelines for using context
• If you are going to use context use a natural real
one
• Contextualise purposefully for what it brings to
the task
• Only use images when the Q could not be asked
without them
• Don’t decorate!

19. Don’t write questions; write tasks
What is your
diagnosis?
State the most likely diagnosis
State the most likely diagnosis
and explain your reasoning
Discuss the differential
diagnoses you would consider
in this case
or …..

20. Instructional verb examples
Compare: to find similarities between things, or to look for characteristics
and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject matter,
including all the main points, essential details, and pros and cons of the
problem, to show your complete understanding of the subject.
Define: to provide a concise explanation of the meaning of a word or
phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions or
results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

21. Specify boundaries of the answer
Species
e.g. “in both dogs and cats…”
Quantities and amounts
e.g. “Provide 5 reasons why…”
With reference to
e.g. “ With reference to the published research from ..”
Time
eg: “in the first 24 hours”
Part of the question
Eg: “for one of your differentials….”

22. What are the
clinical signs of
hypothyroidism
in dogs and how
do they arise?
List the three most
common owner-
observed clinical signs
of hypothyroidism in
dogs and explain how
thyroid hormone
deficiency leads to each
of these signs.

23. Guidance
Question 1
Discuss the use of insulin for the treatment of diabetes
mellitus in cats (25 marks)
Question 2
a) Describe the advantages and disadvantages of insulin
therapy for diabetes mellitus in cats (10 marks)
b) Indicate the dose and frequency of administration of
insulin you would prescribe to a newly diagnosed cat
with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the
frequency and timing of feeding in relation to insulin
dosing in cats with diabetes mellitus (10 marks).

24. Guidance
You have been contacted by a farmer producing Pacific
oysters (Crassostrea gigas) intertidally, in a bay containing
a number of oyster farms. The farmer is concerned with the
amount of dead shell they are seeing during the current
grading. Explain how you would approach this scenario.
(20 marks)
Include in your answer how the information you could
gather might influence your assessment, what differential
diagnoses you consider and detail how you might further
investigate potential causes and what advice you would
provide

25. Wording questions – examples
of problems

26. Name two (2) diagnostic tests you would run next to
investigate the cause of this dog’s current illness.

27. Outline your approach to confirming the initial
clinical diagnosis and a management and
prevention plan for this problem. This discussion
should include an outline on further observations
taken about ….

28. A veterinarian asks your for assistance in
designing a protocol for the delivery of a vaccine
for cats in their practice. What factors would you
take into consideration in designing this protocol?

29. Are there any clinical features which can help you
determine this patient’s prognosis?

30. State what you believe is your most likely
diagnosis.

31. Discuss commonly found tumours and tumour-like
disorders associated with the oral cavity and
dental tissues of the horse.

32. How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3

33. Write notes on
a) considerations in the selection of stockpersons
and animal attendants

34. Describe and give reasons for your further clinical
examination (if any), and recommendations to the
owner/trainer for diagnostics and treatment.
Explain what you think is really important and why.
Explain if you think there is any controversy in
treatment plans.

35. Outline and discuss a conceptual framework for
differentiating between ryegrass varieties available
in the New Zealand market place and for defining
possible strengths and weaknesses of a particular
cultivar. Your conceptual framework should reflect
the various options currently employed by plant
breeders in developing new cultivars. (10 marks)

37. Briefly outline your interpretation of the laboratory
results. Do not just state the abnormalities.

38. Sentence-level clarity
• Simple sentence structures
• Grammatically correct
• Remove superfluous words

39. Question timing

40. How long it will take the candidate
• Unstructured tasks – open ended
• Need to control the demands of the Q carefully
• Question reading time
• Writing speed

41. Klatt & Klatt (2011) Acad Med. 86:1079–1083

42. Poppy is a 9 year old 12 kg female neutered fox terrier cross who has been on
treatment for diabetes mellitus with Caninsulin (porcine lente insulin) for 4 months.
She is currently receiving 5 units subcutaneously at 7:30 am and 5 units at 7:30 pm.
At the time of each injection she is fed a mixture of “Optimum” dog roll and Pedigree
Pal Meaty Bites. Her last visit was 2 weeks ago when her dose of insulin was
decreased by 2 units.
Since the last visit Poppy has been happy and active and is eating all her food at
mealtimes. Her body weight has not changed since the last visit and her body
condition score is 5/9. Her owner reports her to be drinking about 600mL of water
daily. Her owner is very happy with Poppy’s progress and comments that she is her
normal self again. There has been no signs of hypoglycaemia.
You detect no abnormalities on physical examination and admit Poppy to perform a
serial blood glucose curve. The owner has administered the morning insulin dose
and fed her as usual. The following results are obtained.
Based on your assessment of the clinical information and the glucose curve shown
what dose of insulin would you recommend Poppy be given now? (5 marks)

43. A client rings you up to arrange his quarterly visit as he is frustrated by ongoing
problems with his breeding herd. Some sows seem to take ages to cycle after
weaning, a lot are returning (and at funny times), there is a steady trickle of
abortions, and a lot of stillborns. When you get to the farm, the farmer mentions
that he has also had a few sows go down at farrowing with a fever, they usually
start panting and die, and there’s one right now in the old farrowing room he
wants you to take a look at. As you walk through the farrowing rooms you
notice that many of the sows, both expecting and lactating, have swollen
vulvas. The sow in question farrowed yesterday. She clearly has a temperature
and is panting. You also think she looks a bit anaemic and note that she doesn’t
appear to have any milk. You suspect what the problem is but think it would be
nice to confirm your diagnosis as, although it is suspected to occur in Australia,
the disease has never been definitively confirmed. You take a blood sample
and make a smear.
On your way home you drop the slide off at the lab and ask them if they will
have a look at it for you. Later that afternoon the pathologist rings up and (very)
excitedly tells you that after using Wright’s stain she spotted some cocci-like
organisms attached to the red blood cells. The pathologist has rung Biosecurity
Australia and they are not interested in pursuing this particular finding as they
have always considered it to be present.
You ring the farmer to tell him that you have confirmed your suspicion and that
he now needs to embark on the course of action you had discussed with him
earlier. Write the farmer a description of the disease and outline the short and
long-term course of action. Include in your report a brief discussion of what, if
any, potential there is for eradication. (25 marks)

44. Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148-157

45. Effect of time stress
• Time stress
– Increases the use of schemas (sterotyping)
– Decreases working memory processing capacity
– Decreases the ability to maintain relevant information
and suppress irrelevant information

46. Key points

47. Checking wording
• Check the question asks for what you are
rewarding in your marking scheme
• Ensure the wording gives an instruction
• Ensure the scope is clear – may need to be
specified
• Consider how the phrasing of the Q may distract
or misdirect candidates
• Simplify wording if necessary
• Check grammar and punctuation

48. Checking timing
• Allow reading time of 70 words per minute
(only necessary to check unusually wordy
examinations)
• Allow writing speed of no more than 20 words
per minute, preferably 15 words per minute

50. Next session’s task

51. Next session’s task:
• Finalise your Q wording
• Print a copy of your Qs and marking schemes
• Exchange your exam with someone from
another discipline
• Give constructive feedback

52. End of the day
• Discussion on progress and challenges
• Feedback loops between examiners and
chapters
• How best for BoE to support examiners
• How best to administratively support examiners