3 clarity

SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Clarity
and why it’s important
Liz Norman
Massey University
http://www.slidehare.net/liznorman
Sharanne Raidal,
BVSc, PhD, FANZCVS, GradDipEd
Australian and New Zealand College of Veterinary Scientists
Examiner Workshop, 6 – 7th
February 2016

“Effective item writers are trained,
not born....”
Downing and Haladyna (2006) Handbook of Test Development, page 11

Validity
Measuring what we
want to measure
and
Not measuring what we
don’t want to measure
Content domain
Exam
2
Oral
Credentials
domain
Exam
1
Prac
Practice domain

What we don’t want to measure
• Ability to take tests
• Ability to write legibly and fast
• Ability to rote learn whole pages of textbooks or
review articles – prewriting
• Ability to write down a huge series of
unconnected facts in no particular order
• Ability to research examiner’s fields of interest
and rote learn impressive aspects of that
• Ability to interpret what examiners are thinking

Why clarity is important
• We want the candidate to do the task we
envisaged, not something else
• We want to minimise irrelevant difficulty
• Validity – measuring what we intend to
measure

Communication
The examination questions are the question setter’s
expression of the question setter’s task.
The candidate’s answer represents the candidate’s
expression of the candidate’s interpretation of the
questions.
The marker evaluates the marker’s interpretation of the
candidate’s expression of the candidate’s answer.
The marker uses the marker’s interpretation of the
setter’s expression of the setter’s task to evaluate the
candidate’s answer.
Modified from Pollitt & Ahmed (1999) New Model of the
Question Answering Process. IAEA. Bled. Slovenia

General issues with question wording

Expectations and stereotypes
Examples:
•“treatment”
•All differential diagnoses vs those only applicable
in a particular case
•Expectation that Qs will ask about what something
is rather than what it is not
•Expectation of hard questions

Crisp et al (2008) Tales of the expected: the influence of students‘ expectations on
question validity and implications for writing exam questions. Ed Res 50(1): 95-115.

Discuss the diagnosis and management of a cat
with both chronic renal failure and hyperthyroidism.
(24 marks)

Contextualising questions
• Context is good because it brings relevance and
authenticity
• Allows assessment of concrete or specific
examples not abstract concepts or
generalisations
• Allows assessment of applied learning (doing not
just knowing)
• All these carry with them a potential for bias

Contextualising questions
Other disadvantages:
•More words used
•More intended / unintended demand
•Familiarity – schemas / stereotypes
•Focus may direct candidates to the wrong aspects
•Images can be particularly distracting

You are presented with a poorly controlled diabetic
dog. The dog is an 8 year old female spayed
Labrador weighing 36 kg with a body condition
score of 7/9. The dog has been on porcine lente
insulin (Caninsulin) twice daily for 3 months.
Various doses have been tried during this time and
the dog is currently receiving 70 units twice daily.
The dog is still polyuric and polydipsic, with 3+ to
4+ glucosuria.
Discuss the possible causes of poor control which
you should consider when evaluating this dog.

List the possible causes of consistently high blood
glucose values in a dog being treated with
subcutaneous insulin injections for the treatment of
diabetes mellitus. Outline your approach to
investigating these possible causes.

Question 1
A crate of 12 cans of cola costs $4.20. How much
do 7 crates of cola cost?
Question 2
A ski pass costs $42.0. How much would it cost for
7 days.

Guidelines for using context
• If you are going to use context, use a natural,
realistic one
• Contextualise purposefully for what it brings to
the task
• Only use images when the question could not be
asked without them
• Don’t decorate!

a) After several weeks in a tropical
climate the body can adapt by
increasing maximum swat rate to 3.5
L/h. at what rate, in watts, is energy
being removed when sweating that
rapidly. Assume all of the sweat
evaporates. (3 marks)
b) If a 50 kg girl stopped seating because of severe
dehydration, how long would it take for her body to
rise to a dangerous level of 40o
C if originally she
was sweating at a rate of 3.5 L/h to maintain a
body temperature of 36o
C? (2 marks)

Don’t write questions, write tasks
What is your
diagnosis?
State the most likely diagnosis
State the most likely diagnosis
and explain your reasoning.
Discuss the differential
diagnoses you would consider
in this case.

Instructional verb examples
Compare: to find similarities between things, or to look for
characteristics and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject
matter, including all the main points, essential details, and pros
and cons of the problem, to show your complete understanding of
the subject.
Define: to provide a concise explanation of the meaning of a word
or phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of
opinions or results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

Specify boundaries of the answer
Species
eg. “in both dogs and cats....”
Quantities and amounts
eg. “Provide 5 reasons why.....”
With reference to
eg. “With reference to published research from...”
Time
eg. “in the first 24 hours”
Part of the question
eg. “for one of your differentials....”

What are the
clinical signs of
hypothyroidism in
dogs and how do
they arise?
List the three most
common owner-
observed clinical signs
of hypothyroidism in
dogs and explain how
thyroid hormone
deficiency leads to each
of these signs.

Guidance
Question 1
Discuss the use of insulin for the treatment of diabetes mellitus
in cats. (25 marks)
Question 2
a) Describe the advantages and disadvantages of insulin therapy for
diabetes mellitus in cats. (10 marks)
b) Indicate the dose and frequency of administration of insulin you
would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the frequency
and timing of feeding in relation to insulin dosing in cats with diabetes
mellitus. (10 marks)

Guidance
You have been contracted by a farmer producing Pacific oysters
(Crassostrea gigas) intertidally, in a bay containing a number of
oyster farms. The farmer is concerned with the amount of dead
shell they are seeing during the current grading. Explain how you
would approach this scenario. (20 marks)
Include in your answer how the information you could gather
might influence your assessment, what differential diagnoses you
consider, and detail how you might further investigate potential
causes, and what advice you would provide.

Purpose of marking schemes
•To help you during question writing
 What content is important
 Whether the Q asks what you intended it to ask
 Whether it is do-able in the time available
•To help you during question marking
 Helps you decide whether an answer is good enough to
be awarded a mark
 Facilitates reliable and fair marking

Wording of questions – examples of problems

Name two (2) diagnostic tests you would run next
to investigate the cause of this dog’s current
illness.

Outline your approach to confirming the initial
clinical diagnosis and a management and
prevention plan for this problem. This discussion
should include an outline on further observations
taken about .....

A veterinarian asks you for assistance in designing
a protocol for the delivery of a vaccine for cats in
their practice. What factors would you take into
consideration in designing this protocol?

Are there any clinical features which can help you
determine this patient’s prognosis?

State what you believe is your most likely
diagnosis.

Discuss commonly found tumours and tumour-like
disorders associated with the oral cavity and dental
tissues of the horse.

How would you localise the site of the lesion?
Answer provided in marking scheme:
Spinal lesion between T3 and L3

Write notes on
a) Considerations in the selection of
stockpersons and animal attendants

Briefly outline your interpretation of results. Do not
just state the abnormalities.

Sentence-level clarity
• Simple sentence structures
• Grammatically correct
• Remove superfluous words

Question timing

How long will it take the candidate?
• Unstructured tasks – open ended
• Need to control the demands of the question
carefully
• Question reading time
• Writing speed

Reading rates reported by 104
medical students at the Mercer University
School of Medicine, 2010
Number (%) of
students
Number of pages
read per hour
Reading rate
(WPM)
13 (13%) ≤ 6 ≤ 50
55 (53%) 7 – 12 51 - 100
18 (17%) 13 - 18 101 - 150
16 (15%) 19 - 24 151 - 200
2 (2%) ≥ 25 ≥ 201
Klatt and Klatt (2011) Acad Med 86: 1079 - 1083

Poppy is a 9 year old 12 kg female neutered fox terrier cross who has been on
treatment for diabetes mellitus with Caninsulin (porcine lente insulin) for 4
months. She is currently receiving 5 units subcutaneously at 7:30am and 5 units
at 7:30pm. At the time of each injection she is fed a mixture of ‘Optimum’ dog roll
and Pedigree Pal Meaty Bites. Her last visit was 2 weeks ago when her dose of
insulin was decreased by 2 units.
Since the last visit Poppy has been happy and active and is eating all her food at
mealtimes. Her body weight has not changed since the last visit and her body
condition score is 5/9. Her owner reports her to be drinking about 600 mL of
water daily. Her owner is very happy with Poppy’s progress and comments that
she is her normal self again. There have been no signs of hypoglycaemia.
You detect no abnormalities on physical examination and admit Poppy to
perform a serial blood glucose curve. Her owner has administered the morning
insulin dose and fed her as usual. You have been provided with the results
obtained.
Based on your assessment of the clinical information and the glucose curve
shown, what dose of insulin would you recommend Poppy be given now?

A client rings you up to arrange his quarterly visit as he is frustrated by ongoing problems with
his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and
at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the
farm, the farmer mentions that he has also ha d a few sows go down at farrowing with a fever,
they usually start panting and die, and there’s one right now in the old farrowing room he wants
you to take a look at. As you walk through the farrowing rooms you notice that many of the sows,
both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She
clearly has a temperature and is panting. You also thinks she looks a bit anaemic and note that
she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice
to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never
been definitively confirmed. You take a blood sample and make a smear.
On your way home you drop the slide off at the lab and ask them if they will have a look at it for
you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using
Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The
pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular
finding as they have always considered it to be present.
You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to
embark on the course of action you had discussed with him earlier. Write the farmer a
description of the disease and outline the short and long-term course of action. Include in your
report a brief discussion of what, if any, potential there is for eradication. (25 marks)

Handwriting speed for three writing tasks
Writing task
Handwriting speed (wpm)
Mean SD Range
Handwriting speed test
(3 min writing test) 37.83 4.71 27.0 – 51.7
Exam 1 (2 h exam with prior
knowledge of questions) 17.75 2.78 11.8 – 25.7
Exam 2 (2 h exam without
prior knowledge of
questions)
15.37 3.12 9.4 – 23.2
Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148 - 157

Effect of time stress
Time stress
•Increases the use of schemas (stereotyping)
•Decreases working memory processing capacity
•Decreases the ability to maintain relevant
information and suppress irrelevant information

Oral (viva voce) Examinations
Advantages
•A student’s [practitioner’s] ability to verbally explain /
communicate on a subject is a good measure of their
understanding
•Provide a measure of candidate’s verbal
communication skills
•Use of different (visual) cues
•Alternative assessment technique - interactive
•May provide a more authentic assessment of ‘real
world’ skills – thinking on feet, progressive
disclosure, no ‘cues’ in question
•Rapid marking turnaround

Oral (viva voce) Examinations
Disadvantages
•Scarey! [may be influenced by personality]
•Shorter time period – may limit the amount of material
that can be covered relative to a written examination
•Consistency may be hard to achieve
•More subjective?
•Harder to ‘re-grade’ / defend [College procedures]
•Delivered asynchronously (confidentiality / security
issues)

Oral Examinations – Design and Delivery
• Identify question set according to Subject
Guidelines (for each day examined)
• Situate oral exam content within your blueprint
• Determine an appropriate way to deliver questions
and cues (images, data, other)
• Decide appropriate questions – more colloquial
delivery than written papers, but aim for consistency
• Identify / predict need for additional cues
 Avoid prompting / coaching candidates during exam

Oral Examinations – Design and Delivery
•Prepare a marking scheme (as for written / prac)
and considering qualitative and quantitative
(content) aspects of potential answers
•Ensure images (or other props) are of adequate
quality
•Be able to respond to problems during exam
delivery - candidate nerves
- problems with delivery / marking
•Consider what you will say and do during the exam
(be prepared)

Conditions under which
Oral Examinations are valid, reliable and fair
• Ensure items focus on capabilities required for
professional practice (decision making, reasoning)
• Appropriate content (defined by peers, profession)
• Exam items are within the scope of professional
practice, at an appropriate level
• Adequate sampling of questions
• Examiner training (issues and methods)
• Inter-examiner variation monitored, addressed
• Consider potential for bias (pre- and post-exam)
Memon et al (2010) Oral Assessment and Postgraduate Medical
Examinations: Establishing Conditions for Validity, Reliability and Fairness.
Adv in Health Sci Educ 15(2): 277-289

Key points

Written papers - checking wording
• Check the question asks for what you are
rewarding in your marking scheme
• Ensure the wording gives an instruction
• Ensure the scope is clear – may need to be
specified
• Consider how the phrasing of the Q may distract
or misdirect candidates
• Simplify wording if necessary
• Check grammar and punctuation

Written papers - checking timing
• Allow reading time of 70 words per minute (only
necessary to check unusually wordy
examinations)
• Allow writing speed of no more than 20 words
per minute (pref 15 words per minute)

Oral Examinations
• Included within blueprinting process
• Check quality of images / prompts
• Ensure clarity of your questions and marking
scheme (as for writtens)
• Consider delivery (and responses) to
candidates
• Evaluate marking scheme
• CONSISTENCY

Next session’s task:
• Finalise your question wording
• Print a copy of your Qs and marking schemes
• Exchange your exam with someone from
another discipline
• Give constructive feedback

End of the day
• Discussion on progress and challenges
• Feedback loops between examiners and
chapters
• How best for the BoE to support examiners
• How best to administratively support examiners

Question / topic:
Criterion
Unsatisfactory Novice Proficient Weighting
0 1 2 3 4 5
Domain knowledge Little or no awareness of
relevant information;
frequent factual errors or
omission of key information
Able to recall sufficient factual
information relevant to topic;
some errors or key information
omitted
Well informed, excellent recall
of relevant information; few or
no errors, all/most key info
covered
X2
Application of info Little or no understanding of
topic; information presented
randomly and with limited
application to case scenario
Satisfactory understanding of
topic, limited or simplistic
application to case scenario
Good to excellent
understanding of topic;
information intelligently applied
to case scenario
X2
Use of veterinary
terminology
Unable to use, or limited use
of, correct terminology
Able to use correct
terminology, some errors in
pronunciation
Fluent use of appropriate
terminology
X1
Holistic impression
Examiner is uncomfortable
with student’s preparedness
for professional practice;
major deficiencies identified
on > 1 occasion
Examiner is satisfied that
student can operate safely in a
supportive environment
Examiner is impressed with
student’s abilities;
demonstrated knowledge,
critical thinking and
communication ability suggest
advanced readiness for
professional practice
Comments

Oral Examinations
•Mirrors oral communications typical of
professional practice
•Can test limits of knowledge and understanding
(weak and strong candidates)
•Good test of:
• clinical reasoning (the cognitive processes which
constitute professional thinking)
• Interpersonal competence
• Intra- personal qualities (confidence, self-
awareness)

Oral Examinations
BUT
•Resource intensive
•Judgements based on limited evidence
•May be difficult to defend on appeal
•Unfamiliar (potentially difficult) for both examiners
and candidate
•Can be difficult to distinguish between what the
candidate says and how they say it (discourse)
•Performance limiting anxiety

3 clarity

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 3 clarity

Similar to 3 clarity (20)

Recently uploaded

Recently uploaded (20)

3 clarity

Editor's Notes