Presentation for examiner training at the Australian and NZ College of Veterinary Scientists Examiners Workshop 8-9 February 2020

1. Massey University | massey.ac.nz | 0800 MASSEY
Clarity and why it’s
important
Liz Norman
Massey University

2. Why clarity is important
• We want the candidate to do the task we envisaged, not
something else
– Validity – measuring what we intend to measure
• We want to minimise irrelevant difficulty
– Validity – measuring what we intend to measure

3. Thinking about thinking
https://www.youtube.com/watch?v=EchfO2pjOrM

4. Schemas and sterotypes
• All of us develop schemas or stereotypes
• Help us categorise and process complex information quickly
• Particular features of questions trigger certain schemas and
hence expectations.
• Triggered without us being aware of it and before the reading
of the text reaches consciousness
• Affect our interpretation of the question.

5. Thinking about thinking
• When candidates read a question, they must form a mental
model of the task required and begin to plan their response.
• They begin to form a model before they have even finished
reading the question.
• The model they form is influenced by their expectations and
their pre-existing mental schemas and sterotypes

6. Elephant Riding
Jan Kemp
Climbing up
the back of an elephant
you spring into
the toehold of its tail
held in place by the mahout
grab the ropes
strapped round its belly
and haul yourself up.
She rises
from buckled knees under you
moves like a ship
you’re high
under the hanging ashoka leaves
as you flow forward
her fly-bitten ears grey sails flap.
she flings the odd young-leaved
branch
into her mouth
with her triumphant trunk.

7. Effects of anxiety and time stress on thinking
• Decreased working memory
• Decreased processing capacity
• Use of type 1 processing increases
– Automatic thinking
– Use of schemas and stereotypes
• Lowered ability to concentrate on relevant information and
suppress irrelevant information
Can lead to misunderstanding of the intent of the
question and the task asked of them

8. Implications for question writing
• There may be problems with questions that you can’t see –
your schemas are different
• The text at the beginning of a question will have more
influence on the schema elicited than the text you place at the
end.
• Images have more influence on the expectations developed
than text

9. Implications for question writing
• Anxiety and time stress increase schema use
– We might be just measuring propensity to anxiety or writing speed
• Each candidate will have different schemas
– We might be just measuring cultural backgrounds or experience with
examinations
• Problems with questions may not affect all candidates equally
and therefore may be hard to detect.

10. Contextualising Qs
• Context is good because it brings relevance and authenticity
• Allows assessment of concrete or specific examples not
abstract concepts or generalisations
• Allows assessment of applied learning (doing not just knowing)
All these carry with them a potential for bias.

11. Contextualising Qs
Potential problems:
• More words used
• Use of colloquial expressions or
culturally-specific terms
• Differential familiarity
• Requirement for sifting
• May activate schemas that interfere with thinking

12. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation
offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.

13. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation
offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.

14. Focussed contextualisation
• Contextualisation works best when well focussed.
• The most salient aspects of the context are also the main
issues addressed in the question.
• Makes Qs more likely to elicit helpful schemas than unhelpful
ones, in candidates with relevant knowledge.
• Does not make Qs easier for candidates that lack relevant
knowledge.
Ahmed, A., & Pollitt, A. (2007). Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy and Practice, 14(2), 201-232.

15. Recommendations for maximising clarity
• Use clear language that is as simple as possible while still being
precise
• Contextualise purposefully for what it brings to the task
• Avoid contradicting expectations or switching schemas part
way through a question

16. Wording questions:
examples of problems

17. Name two (2) diagnostic tests you would run next to investigate
the cause of this dog’s current illness.

18. Outline your approach to confirming the initial clinical diagnosis
and a management and prevention plan for this problem. This
discussion should include an outline on further observations
taken about ….

19. A veterinarian asks your for assistance in designing a protocol for
the delivery of a vaccine for cats in their practice. What factors
would you take into consideration in designing this protocol?

20. Are there any clinical features which can help you determine this
patient’s prognosis?

21. State what you believe is your most likely diagnosis.

22. Discuss commonly found tumours and tumour-like disorders
associated with the oral cavity and dental tissues of the horse.

23. How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3

24. Write notes on
a) considerations in the selection of stockpersons and animal
attendants

25. Describe and give reasons for your further clinical examination (if
any), and recommendations to the owner/trainer for diagnostics
and treatment. Explain what you think is really important and
why. Explain if you think there is any controversy in treatment
plans.

26. Outline and discuss a conceptual framework for differentiating
between ryegrass varieties available in the New Zealand market
place and for defining possible strengths and weaknesses of a
particular cultivar. Your conceptual framework should reflect the
various options currently employed by plant breeders in
developing new cultivars. (10 marks)

28. Which of the following statements is TRUE?
a. A ventral hernia is classified as a false hernia
b. A diaphragmatic hernia is classified as a true hernia
c. Umbilical hernias are classified as false hernias
d. False hernias have a hernia sac
e. Umbilical hernias are relatively uncommon

30. Briefly outline your interpretation of the laboratory results. Do
not just state the abnormalities.

31. Discuss your approach to the management of large colon
volvulus.

32. A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with his
breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny
times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer
mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting
and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk
through the farrowing rooms you notice that many of the sows, both expecting and lactating, have
swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting.
You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect
what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to
occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make
a smear.
On your way home you drop the slide off at the lab and ask them if they will have a look at it for you.
Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain
she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung
Biosecurity Australia and they are not interested in pursuing this particular finding as they have always
considered it to be present.
You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark
on the course of action you had discussed with him earlier. Write the farmer a description of the disease
and outline the short and long-term course of action. Include in your report a brief discussion of what, if
any, potential there is for eradication. (25 marks)

33. Example from a job interview – manager position
Relationships with other managers and service leaders as well as
staff members from other units is vital to successful performance
in this role.
Tell us how you have developed an effective working relationship
with an influential person within one of your roles.

34. Steps for checking clarity
• Ensure the wording gives an instruction.
• Ensure that the question scope is clear. Specify it if necessary.
• Check the question asks for what you are rewarding in your marking scheme—all
of it.
• Check that understanding the task does not rely on information that might not
be common knowledge in different parts of the world.
• Consider how the phrasing of the question may distract or misdirect candidates
and rephrase if necessary.
• Use the simplest language possible while still being precise. This includes word
choice and sentence structure. Remove all unnecessary wording.
• Check that the discourse of the question mirrors the discourse of the response
you require.
• Check spelling, grammar, and punctuation. Explain all abbreviations unless
knowledge of them is considered part of the curriculum.

35. “Effective item writers are trained,
not born … “
Downing and Haladyna 2006, Handbook of test development ,p. 11