Examiners Workshop, 4-5 February, 2017, Gold Coast Australia

1. Clarity and why its important
Liz Norman
Massey University

2. “Effective item writers are trained,
not born … “
Downing and Haladyna 2006, Handbook of test development ,p. 11

3. Why clarity is important
• We want the candidate to do the task we
envisaged, not something else
– Validity – measuring what we intend to measure
• We want to minimise irrelevant difficulty
– Validity – measuring what we intend to measure

4. Thinking about thinking
https://www.youtube.com/watch?v=EchfO2pjOrM

5. Thinking about thinking
• When candidates read a question, they must
form a mental model of the task required and
begin to plan their response.
• They begin to form a model before they have
even finished reading the question.
• The model they form is influenced by their
expectations and their pre-existing mental
schemas and sterotypes

6. Elephant Riding
Jan Kemp
Climbing up
the back of an elephant
you spring into
the toehold of its tail
held in place by the mahout
grab the ropes
strapped round its belly
& haul yourself up.
She rises
from buckled knees under you
moves like a ship
you’re high
under the hanging ashoka leaves
as you flow forward
her fly-bitten ears grey sails flap.
she flings the odd young-leaved branch
into her mouth
with her triumphant trunk.

7. Effects of anxiety and time stress on
thinking
• Decreased working memory
• Decreased processing capacity
• Use of type 1 processing increases
– Automatic thinking
– Use of schemas and sterotypes
• Lowered ability to concentrate on relevant
information and suppress irrelevant information
Can lead to misunderstanding of the intent of the
question and the task asked of them

8. Schemas and sterotypes
• All of us develop schemas or stereotypes
• Help us categorise and process complex
information quickly
• Particular features of questions trigger certain
schemas and hence expectations.
• Triggered without us being aware of it and
before the reading of the text reaches
consciousness
• Affect our interpretation of the question.

9. Implications for question writing
• There may be problems with questions that
you cant see – your schemas are different
• The text at the beginning of a question will
have more influence on the schema elicited
than the text you place at the end.
• Images have more influence on the
expectations developed than text

10. Implications for question writing
• Anxiety and time stress increase schema use
– We might be just measuring propensity to
anxiety or writing speed
• Each candidate will have different schemas
– We might be just measuring cultural backgrounds
or experience with examinations
• Problems with questions may not affect all
candidates equally and therefore may be hard
to detect.

11. Contextualising Qs
• Context is good because it brings relevance
and authenticity
• Allows assessment of concrete or specific
examples not abstract concepts or
generalisations
• Allows assessment of applied learning (doing
not just knowing)
All these carry with them a potential for bias.

12. Contextualising Qs
Potential problems:
• More words used
• Use of colloquial expressions or
culturally-specific terms
• Differential familiarity
• Requirement for sifting
• May activate schemas that interfere with
thinking

13. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation
offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.

14. Focussed contextualisation
• Contextualisation works best when well
focussed.
• The most salient aspects of the context are
also the main issues addressed in the
question.
• Makes Qs more likely to elicit helpful schemas
than unhelpful ones, in candidates with
relevant knowledge.
• Does not make Qs easier for candidates that
lack relevant knowledge.
Ahmed, A., & Pollitt, A. (2007). Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy and Practice, 14(2), 201-
232.

15. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation
offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.

16. Recommendations for maximising
clarity
• Use clear language that is as simple as
possible while still being precise
• Contextualise purposefully for what it brings
to the task
• Avoid contradicting expectations or switching
schemas part way through a question

17. Wording questions – examples
of problems

18. Name two (2) diagnostic tests you would run next
to investigate the cause of this dog’s current
illness.

19. Outline your approach to confirming the initial
clinical diagnosis and a management and
prevention plan for this problem. This discussion
should include an outline on further
observations taken about ….

20. A veterinarian asks your for assistance in
designing a protocol for the delivery of a vaccine
for cats in their practice. What factors would you
take into consideration in designing this
protocol?

21. Are there any clinical features which can help
you determine this patient’s prognosis?

22. State what you believe is your most likely
diagnosis.

23. Discuss commonly found tumours and tumour-
like disorders associated with the oral cavity and
dental tissues of the horse.

24. How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3

25. Write notes on
a) considerations in the selection of
stockpersons and animal attendants

26. Describe and give reasons for your further
clinical examination (if any), and
recommendations to the owner/trainer for
diagnostics and treatment. Explain what you
think is really important and why. Explain if you
think there is any controversy in treatment
plans.

27. Outline and discuss a conceptual framework for
differentiating between ryegrass varieties
available in the New Zealand market place and
for defining possible strengths and weaknesses
of a particular cultivar. Your conceptual
framework should reflect the various options
currently employed by plant breeders in
developing new cultivars. (10 marks)

29. Which of the following statements is TRUE?
a. A ventral hernia is classified as a false hernia
b. A diaphragmatic hernia is classified as a true hernia
c. Umbilical hernias are classified as false hernias
d. False hernias have a hernia sac
e. Umbilical hernias are relatively uncommon

31. Briefly outline your interpretation of the
laboratory results. Do not just state the
abnormalities.

32. Discuss your approach to the management of
large colon volvulus.

33. A client rings you up to arrange his quarterly visit as he is frustrated by ongoing
problems with his breeding herd. Some sows seem to take ages to cycle after weaning,
a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot
of stillborns. When you get to the farm, the farmer mentions that he has also had a
few sows go down at farrowing with a fever, they usually start panting and die, and
there’s one right now in the old farrowing room he wants you to take a look at. As you
walk through the farrowing rooms you notice that many of the sows, both expecting
and lactating, have swollen vulvas. The sow in question farrowed yesterday. She
clearly has a temperature and is panting. You also think she looks a bit anaemic and
note that she doesn’t appear to have any milk. You suspect what the problem is but
think it would be nice to confirm your diagnosis as, although it is suspected to occur in
Australia, the disease has never been definitively confirmed. You take a blood sample
and make a smear.
On your way home you drop the slide off at the lab and ask them if they will have a
look at it for you. Later that afternoon the pathologist rings up and (very) excitedly
tells you that after using Wright’s stain she spotted some cocci-like organisms attached
to the red blood cells. The pathologist has rung Biosecurity Australia and they are not
interested in pursuing this particular finding as they have always considered it to be
present.
You ring the farmer to tell him that you have confirmed your suspicion and that he
now needs to embark on the course of action you had discussed with him earlier.
Write the farmer a description of the disease and outline the short and long-term
course of action. Include in your report a brief discussion of what, if any, potential
there is for eradication. (25 marks)

34. Steps for checking clarity
• Ensure the wording gives an instruction.
• Ensure that the question scope is clear. Specify it if necessary.
• Check the question asks for what you are rewarding in your marking
scheme—all of it.
• Check that understanding the task does not rely on information that
might not be common knowledge in different parts of the world.
• Consider how the phrasing of the question may distract or misdirect
candidates and rephrase if necessary.
• Use the simplest language possible while still being precise. This
includes word choice and sentence structure. Remove all
unnecessary wording.
• Check that the discourse of the question mirrors the discourse of
the response you require.
• Check spelling, grammar, and punctuation. Explain all abbreviations
unless knowledge of them is considered part of the curriculum.