Blueprinting and drafting questions Liz Norman ANZCVS 2019

Massey University | massey.ac.nz | 0800 MASSEY
Blueprinting and drafting
questions
Liz Norman
Massey University

What is a blueprint?
"blueprint, n.". Something which acts as a plan,
model, or template
OED Online http://www.oed.com
Railway and harbour report, Vancouver, B.C.
by City of Vancouver Archives, Attribution License

Why we need to blueprint
Its important that we
sample representatively
from the content
domain (all that it is
possible to examine)
This is so we can
generalise performance
to the whole content
domain

Constructing blueprints
• Table or grid with at least 2 dimensions
• Dimensions could include:
– Learning outcomes
– Diseases or presenting problem
– Body system
– Species
– Knowledge domain: anatomy, pathophysiology, diagnosis,
management, prognosis, epidemiology, investigations, ethics & law
– Skills: interpretation, reasoning, decision making, evaluation,
problem solving
– Cognitive level: factual recall, analysis and interpretation of data,
problem solving
– Degree of abstraction: principles (theoretical), applied (concrete)
– Location of question in the examination components

Constructing blueprints
• Begin by considering the curriculum
• What knowledge, skills, and attitudes are
examinable?
• Subject guidelines

Body system by knowledge domain
Written paper
1
Written paper
2
Oral
examination
Practical
examination
LO1 X X X
LO2 X X
LO3 X X X
LO4 X

Body system by knowledge domain by
exam component
Pathophysiology
Investigation and
diagnosis
Treatment and
management
Gastrointestinal P1Q1 P1Q1, P2Q4 OQ3
Cardiovascular P1Q4 P2Q2, OQ1 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 OQ2 P2Q3
Respiratory P2Q5

Proportion of exam by species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%

cognitive level
Pathophysiology Investigation and diagnosis Treatment and management
recall
analyse,
interpret
problem
solve
recall
analyse,
interpret
problem
solve
recall
analyse,
interpret
problem
solve
Gastrointes
tinal
P1Q1
P1Q1
P2Q4
OQ3
Cardiovasc
ular
P1Q4 P2Q2 OQ1 P2Q2
Nervous
P1Q2
P2Q1
Endocrine P1Q3 OQ2 P2Q3
Respiratory P2Q5

subject of question
Written paper 2 Pathophysiology Investigation and diagnosis
Treatment and
management
Gastrointestinal
Investigation of exocrine
pancreatic insufficiency
Cardiovascular
Investigation of atrial
fibrillation
Management of
cardiomyopathy
Nervous
Diagnosis of L2-3 spinal
neoplasia
Endocrine
Management of diabetic
ketoacidosis
Respiratory Diagnosis of feline asthma
Management of feline
asthma

Weighting questions
Impact Weight Frequency Weight
Less important 1 Rarely seen 1
Essential 2 Relatively common 2
High impact 3 Very common 3
McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine
clerkship evaluation. Med Teach, 27(6), 544-547

Creating appropriate tasks
• Questions are actually tasks
• We need the tasks to be things that Members
or Fellows need to be able to do.
• This ensures we can validly extrapolate from
performance on the exam, to performance in
the Membership or Fellowship domain.

Creating appropriate tasks
• Emphasis should be on assessing the
candidate’s
– understanding
– skills in analysing and interpreting information
– skills in problem solving
– judgement
• Not recall of information

Recall-higher order classification
Fact recall Questions capable of being answered by
reference to one paragraph in a text or notes
(or several paragraphs for questions requiring
recall of several facts)
Applied
(higher order)
Questions that require the use of facts or
concepts, the solution of a diagnostic or
physiologic problem, the perception of a
relationship, or other process beyond recalling
discrete fact
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical
students. Academic Medicine, 65(9), S59-60.

Webb’s depth of knowledge
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25. doi:10.1080/08957340709336728.
recall Recall information or perform a simple step, identify,
measure, describe, explain simple ideas
skill/concept Requires more than one step, comparing,
interpreting, estimating, making observations,
explaining, organising and displaying data
strategic
thinking
Requires planning and using evidence, explaining
reasoning, make conjectures, draw conclusions,
solving problems
extended
thinking
Complex reasoning, planning developing and thinking
over an extended period of time, making multiple
connections, synthesis of ideas into new concepts

Knowledge-understanding
classification
Knowledge knowing about; a body of coherent facts; can be
thought of as right or wrong.
Understanding knowing how and why; the meaning of facts; the
theory that links facts and provides meaning; how
sense is made of facts to enable them to be applied
to analysis, synthesis, evaluation; to be able to
explain why particular facts or skills are applicable
to a particular situation; to know which fact to
apply when; to be able to create new knowledge or
modify or adapt an idea to a new situation.
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.

SOLO taxonomy
Biggs, J. B., & Collis, K. F. (1982).
Evaluating the quality of learning:
The SOLO taxonomy (structure of
the observed learning outcome).
New York: Academic Press.

Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract

Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract
Quantitative change
Qualitative change

Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract
knowledge
understanding

Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract
surface
deep

Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract
recall
application

Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract
recall
skill/concept
strategic thinking
extended thinking

Lack of knowledge or common knowledge only.
Answer addresses a single aspect or a few aspects.
Oversimplified, reductionist.
Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract

Multi structural
Signs to look for in answers (not all need to be present)
• Multiple aspects considered and may be comprehensive
• No particular order to aspects presented, or serial structure
“he said, she said”
• Inclusion of irrelevant/less important material
• Lacks integration such as causal explanation or compare and
contrasting
• Replication of material from sources – rote learned or
reproduced without significant transformation

Relational
Signs to look for in answers (not all need to be present)
• Aspects explained relative to one another
• Logically organised answer
• Analysis and or synthesis
• Compares similarities and differences
• Integrates multiple levels (eg: molecular, biochemical, systemic)
• Expresses reasons, explains implications, or reaches a conclusion
• Expresses relative importance, value, significance of aspects.
• Selective answer that addresses the point of the question and may
be shorter than a multistructural answer
• Uses the language of the discipline - terminology and phrasing
• Relates answer to examples or experience
• Relates answer to organising principles of the discipline
• Evaluates inconsistencies

Lack of knowledge or common knowledge only.
Answer addresses a single aspect or a few aspects.
Oversimplified, reductionist.
Answer addresses multiple aspects and may well be
comprehensive. Connections simple or lacking.
Answer addresses multiple aspects and also how they
integrate and inter-relate. Makes connections between
aspects. Knows their relative
importance/value/significance.
Goes beyond a relational answer within and beyond a
domain, including to areas not experienced or only
imagined. Has more originality, creativity, meta-
connections, and utilisation of overarching principles.
Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract

Specifying tasks – give an instruction
Compare: to find similarities between things, or to look for
characteristics and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject
matter, including all the main points, essential details, and pros and
cons of the problem, to show your complete understanding of the
subject.
Define: to provide a concise explanation of the meaning of a word or
phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions
or results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

Specifying scope
• Do you want a general answer about this
condition that applies to all possibilities or one
that specifically applies to this case?
• Do you want an answer to include
– all possible options?
– those that are currently available in this region of
the world?
– those that are available under cost constraints?
• Writing marking scheme helps you get the
scope right

Specify boundaries of the answer
• Species
e.g. “in both dogs and cats…”
• Quantities and amounts
e.g. “Provide 5 reasons why…”
• With reference to
e.g. “ With reference to the published research from ..”
• Time
eg: “in the first 24 hours”
• Part of the question
Eg: “for one of your differentials….”

Construct irrelevance
When the skills, abilities or knowledge required
to answer examination questions is not part of
the domain we are trying to assess.
What are we actually assessing?

Important sources of construct
irrelevance
• Unclear tasks – because the candidate may do
something other than what we intended them
to do.
• Unanticipated and irrelevant difficulty and
demand
• Insufficient time to complete the task (fully)

Controlling task difficulty and demand
• Tasks need to be at an appropriate level of
difficulty
• They don’t need to be “tricky” in order to be
difficult

Some factors that affect demand and
difficulty
• Type of operation
• Degree of novelty
• Number of components or ideas involved
• Whether resources are provided or need to be
generated by the candidate
• The question wording and any images, diagrams,
or tables provided
• Degree of abstraction
• Response strategy – simple, stepwise, integrated
• Guidance provided

Some factors that affect demand and
difficulty
• Type of operation
– Simple steps vs synthesis, interpretation, evaluation

Question 1:
Explain the physiological actions of insulin.
Question 2:
Explain the physiological actions of ghrelin.

Question 1:
Compare and contrast the clinical signs of
hypoadrenocorticism with those of
hyperadrenocorticism in dogs.
Question 2:
Compare and contrast the clinical signs of
diabetes mellitus with those of
hyperadrenocorticism in dogs.

For EACH ECG, make a recommendation for
management of the horse and explain your
reasoning.
Discuss the recommended management for a
horse with atrial fibrillation and explain your
reasoning.

Discuss the concept of quality of life in terms of
biological function, “feelings” and natural
existence.
Describe how both classical conditioning and
operant conditioning are involved in cows
confidently entering the milking shed and letting
down.

Outline and discuss a conceptual framework for
differentiating between ryegrass varieties
available in the New Zealand market place and
for defining possible strengths and weaknesses
of a particular cultivar. Your conceptual
framework should reflect the various options
currently employed by plant breeders in
developing new cultivars. (10 marks)

Describe the skull. (30 marks)

Question 1
Discuss the use of insulin for the treatment of diabetes
mellitus in cats (25 marks)
Question 2
a) Describe the advantages and disadvantages of insulin
therapy for diabetes mellitus in cats (10 marks)
b) Indicate the dose and frequency of administration of
insulin you would prescribe to a newly diagnosed cat with
diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the
frequency and timing of feeding in relation to insulin
dosing in cats with diabetes mellitus (10 marks).

You have been contacted by a farmer producing Pacific
oysters (Crassostrea gigas) intertidally, in a bay containing
a number of oyster farms. The farmer is concerned with
the amount of dead shell they are seeing during the
current grading. Explain how you would approach this
scenario. (20 marks)
Include in your answer how the information you could
gather might influence your assessment, what differential
diagnoses you consider and detail how you might further
investigate potential causes and what advice you would
provide.

Pacing examinations
• Reading time: 40 words per minute
• Writing time: 20 words per minute
• Marking schemes help you check the pacing

Blueprinting
• Ensure the examination (as a whole)
representatively samples from the domain
specified for the examination
– Topics
– Level
– Species
– Other things eg:
• diagnosis vs management
• principles vs application

Question writing
• Choose a task that will allow candidates to
demonstrate higher order skills
• Phrase your “question” as an instruction
• Define the scope and boundaries of the
answer
• Remove all irrelevant difficulty
• Keep level of demand appropriate for the level
of examination (Fellowship vs Membership)
and the time available

What you have to watch out for
• Design assessment tasks that request an
integrated and coherently structured response.
Not: “write short notes on”, “use a table to
compare”
• Relational responses can be reproduced, so design
tasks that require an original application of related
knowledge.
• Decide on the nature of connections expected at
the level of expertise being assessed. Knowledge
connections can range from very basic to expert.

References and further reading
Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for
quality learning at university (4th ed.). Maidenhead
UK: McGraw-Hill
Biggs, J. B., & Collis, K. F. (1982). Evaluating the
quality of learning: The SOLO taxonomy (structure of
the observed learning outcome). New York:
Academic Press.
Biggs, J. (1992). A qualitative approach to grading
students. HERDSA News, 14(3), 3-6.

Blueprinting and drafting questions Liz Norman ANZCVS 2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Blueprinting and drafting questions Liz Norman ANZCVS 2019

Similar to Blueprinting and drafting questions Liz Norman ANZCVS 2019 (20)

More from Liz Norman

More from Liz Norman (19)

Recently uploaded

Recently uploaded (20)

Blueprinting and drafting questions Liz Norman ANZCVS 2019