This document provides an overview of blueprinting and drafting questions for exams. It discusses what a blueprint is and why they are important for ensuring content is sampled representatively. A blueprint example is provided showing distribution of questions by topic and species. Question difficulty is addressed, considering factors like familiarity, task complexity, structure, resources, and wording. The importance of clear instructional verbs and specifying the scope of answers is covered. Finally, question timing is discussed in terms of accounting for reading and writing times.

1. Blueprinting and drafting questions
Liz Norman
Massey University

2. Overview
1.
2.
3.
4.
Blueprinting
Question difficulty
Wording questions
Question timing

3. What is a blueprint?
"blueprint, n.". Something which acts as a plan,
model, or template
Specifies what’s going to be in the exam
– Content/topics – breadth of the exam
– Level - depth of the exam
OED Online http://www.oed.com

4. Why we need to blueprint
• Its important that we sample representatively
from the content domain (all that it is possible to
examine)
• This is so we can extrapolate performance to the
whole content domain

5. Breadth by topic
Pathophysiology
Investigation and
diagnosis
Gastrointestinal
P1Q1
P1Q1, P2Q4
Cardiovascular
P1Q4
P2Q2
Nervous
Endocrine
Musculoskeletal
Treatment and
management
P2Q2
P1Q3, P2Q1
P1Q3
P2Q3
P2Q5

6. Breadth across species
species
number of Qs
percent
small animal
15
52%
farm
5
17%
horse
4
14%
exotic
3
10%
lab
1
3%
all
1
3%

7. Level - depth
•
•
•
•
Surface – deep
Fact recall – applied
Blooms taxonomy
SOLO taxonomy

8. Fact recall vs applied
Fact recall:
Questions capable of being answered by reference to one
paragraph in a text or notes (or several paragraphs for
questions requiring recall of several facts)
Applied (higher order)
Questions that require the use of facts or concepts, the
solution of a diagnostic or physiologic problem, the
perception of a relationship, or other process beyond
recalling discrete fact
From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.

9. Blooms taxonomy
create
evaluate
Application
analyse
apply
understand
Recall
remember

10. Blooms instructional verbs
• Create: compose, plan, propose, design, formulate, arrange, assemble,
collect, construct, create, set-up, organise, manage, prepare.
• Evaluate: judge, appraise, evaluate, rate, compare, revise, assess,
estimate
• Analyse: distinguish, analyse, differentiate, appraise, calculate,
experiment, test, compare, contrast, criticise, diagram, inspect, debate,
question, relate, solve, examine, categorise.
• Apply: interpret, apply, employ, use, demonstrate, dramatise, practice,
illustrate, operate, schedule, sketch.
• Understand: translate, restate, discuss, describe, recognise, explain,
express, identify, locate, report, review, tell
• Remember: define, repeat, record, list, recall, name, relate, underline.

11. SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/

12. Prestructural
Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational
Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

13. Prestructural
Unistructural
Multistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Quantitative change
Relational
Qualitative change
Extended
abstract

14. Prestructural
Unistructural
Multistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
surface
Relational
deep
Extended
abstract

15. Prestructural
Unistructural
Multistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
recall
Relational
application
Extended
abstract

16. Prestructural
Unistructural
Paraphrase, define, identify, count, name, recite, follow simple
instructions, calculate, reproduce, arrange, recognise, find, note,
seek, sketch, pick
Multistructural
Combine, classify, structure, describe, enumerate, list, do
algorithm, apply method, account for execute, formulate, solve,
conduct, prove, complete, illustrate, express, characterise
Relational
Analyse, compare, contrast, integrate, relate, explain causes,
apply theory (to its own domain), argue, implement, plan,
summarize, construct, design, interpret (some senses), structure,
conclude, substantiate, exemplify, derive, adapt
Extended
abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory
(to new domain), assess, evaluate, interpret (some senses),
critically reflect, predict, criticise, reason
Potter & Kustra (2012). http://www1.uwindsor.ca/ctl/system/files/PRIMER-on-Learning-Outcomes.pdf

17. Topic 1
30%
Topic 2
30%
Topic 3
40%
total
Knowledge of terms
2
5
5
12
Comprehension of principles
4
3
4
11
Application of principles
3
3
3
9
Analysis of situations
3
2
5
10
Evaluation of solutions
3
2
3
8
Total questions
15
15
20
50
In this example
• Topic 1 represents 30% of the whole curriculum being assessed
so 15 of 50 questions are to address it.
• Four questions are to address comprehension of the principles
of Topic 1.
Scoring office, Michigan State University, Writing Test Items, http://scoring.msu.edu/writitem.html

18. Blueprints that include depth
Pathophysiology
recall
Gastrointestinal
P1Q4
higher
order
P1Q1,
P2Q4
Treatment and
management
recall
P2Q2
higher
order
P2Q2
P1Q2,
P2Q1
Nervous
Musculoskeletal
recall
P1Q1
Cardiovascular
Endocrine
higher
order
Investigation and
diagnosis
P1Q3
P2Q3
P2Q5

19. Blueprint
• Enables you to plan and check that an exam, or
set of exams, covers the content it is meant to
cover

20. Question difficulty

21. Should questions to be difficult?
• Norm-referenced vs standards-based grading

22. Question difficulty
• Familiarity - obscurity
eg:
– Name the two most significant muscles involved for
mastication.
– Explain the physiological actions of insulin
– Explain the physiological actions of ghrelin

23. Question difficulty
• Familiarity - obscurity
• Level of task
– Complexity
– Abstractedness

24. Question difficulty
• Familiarity - obscurity
• Level of task
– Complexity
– Abstractedness
• Structure

25. Structure
Discuss the use of insulin for the treatment of diabetes
mellitus in cats (25 marks)
a) Describe the advantages and disadvantages of insulin
therapy for diabetes mellitus in cats (10 marks)
b) Indicate the dose and frequency of administration of
insulin you would prescribe to a newly diagnosed cat
with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the
frequency and timing of feeding in relation to insulin
dosing in cats with diabetes mellitus (10 marks).

26. Strategy guidance
You have been contacted by a farmer producing Pacific
oysters (Crassostrea gigas) intertidally, in a bay containing
a number of oyster farms. The farmer is concerned with the
amount of dead shell they are seeing during the current
grading. Explain how you would approach this scenario.
(20 marks)
Include in your answer how the information you could
gather might influence your assessment, what differential
diagnoses you consider and detail how you might further
investigate potential causes and what advice you would
provide

27. Question difficulty
• Familiarity - obscurity
• Level of task
– Complexity
– Abstractedness
• Structure
• Resources

29. Question difficulty
• Familiarity - obscurity
• Level of task
– Complexity
– Abstractedness
• Structure
• Resources
• Wording

30. General issues with question
wording

31. Communication
The examination question is the question setter’s expression
of the question setter’s task.
The student’s answer represents the student’s expression of
the student’s interpretation of the question.
The marker evaluates the marker’s interpretation of the
student’s expression of the student’s answer.
The marker uses the marker’s interpretation of the setter’s
expression of the setter’s task to evaluate the student’s
answer.
Modified from Pollitt & Ahmed (1999) New Model of the Question Answering Process. IAEA. Bled, Slovenia

32. Expectations and stereotypes
Examples:
• male animal case
• all differential diagnoses vs those only applicable
in a particular case
• “treatment”
• expectation of hard questions
• expectation that Qs will ask about what
something is rather than what it is not

33. Contextualising Qs
• Context is good because it brings relevance and
authenticity
• Allows assessment of concrete or specific examples
not abstract concepts or generalisations
• Allows assessment of applied learning (doing not
just knowing)
• All these carry with them a potential for bias.

34. Real world context
• Advantages
– Assessing application to a real problem
– “Shows how” rather than “knows”
• Disadvantages
–
–
–
–
–
More words used
More intended/unintended demand
Familiarity – schemas/sterotypes
Focus may direct candidates to the wrong aspects
Images can be particularly distracting

35. Guidelines for context
• If you are going to use context use a natural real
one
• Contextualise purposefully for what it brings to
the task
• Only use images when the Q could not be asked
without them
• Don’t decorate!

36. Unpacking or sifting
• A crate of 12 cans of cola costs $4.20. How
much do 7 crates of cola cost?
• A ski pass costs $4.20. How much would it cost
for 7 days?
Ahmed & Pollitt (2007) Assessment in Education: Principles, Policy & Practice 14:201-232.

37. A client rings you up to arrange his quarterly visit as he is frustrated by ongoing
problems with his breeding herd. Some sows seem to take ages to cycle after
weaning, a lot are returning (and at funny times), there is a steady trickle of
abortions, and a lot of stillborns. When you get to the farm, the farmer mentions
that he has also had a few sows go down at farrowing with a fever, they usually
start panting and die, and there’s one right now in the old farrowing room he
wants you to take a look at. As you walk through the farrowing rooms you
notice that many of the sows, both expecting and lactating, have swollen
vulvas. The sow in question farrowed yesterday. She clearly has a temperature
and is panting. You also think she looks a bit anaemic and note that she doesn’t
appear to have any milk. You suspect what the problem is but think it would be
nice to confirm your diagnosis as, although it is suspected to occur in Australia,
the disease has never been definitively confirmed. You take a blood sample
and make a smear.
On your way home you drop the slide off at the lab and ask them if they will
have a look at it for you. Later that afternoon the pathologist rings up and (very)
excitedly tells you that after using Wright’s stain she spotted some cocci-like
organisms attached to the red blood cells. The pathologist has rung Biosecurity
Australia and they are not interested in pursuing this particular finding as they
have always considered it to be present.
You ring the farmer to tell him that you have confirmed your suspicion and that
he now needs to embark on the course of action you had discussed with him
earlier. Write the farmer a description of the disease and outline the short and
long-term course of action. Include in your report a brief discussion of what, if
any, potential there is for eradication. (25 marks)

38. Other aspects
•
•
•
•
Distractors
Everyday language
Highlighting of words and phrases
Showing the mark allocation

39. Wording questions - examples

40. Don’t write questions; write tasks
What is your
diagnosis?
State the most likely diagnosis
or
State the most likely diagnosis and
explain your reasoning
or
Discuss the differential diagnoses you
would consider in this case
or
…..

41. Instructional verb examples
Compare: to find similarities between things, or to look for characteristics
and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject matter,
including all the main points, essential details, and pros and cons of the
problem, to show your complete understanding of the subject.
Define: to provide a concise explanation of the meaning of a word or
phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions or
results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

42. Specify boundaries of the answer
Species
e.g. “in both dogs and cats…”
Quantities and amounts
e.g. “Provide 5 reasons why…”
With reference to
e.g. “ With reference to the published research from ..”

43. Q parts – the scope
List the clinical
signs of
hypothyroidism
in dogs.
List the three most common
owner-observed clinical signs of
hypothyroidism in dogs and
explain how thyroid hormone
deficiency leads to each of these
signs.

44. Examples of problems….
Name two (2) diagnostic tests you would run next to
investigate the cause of this dog’s current illness.

45. Examples of problems
Outline your approach to confirming the initial
clinical diagnosis and a management and
prevention plan for this problem. This discussion
should include an outline on further observations
taken about ….

46. Examples of problems
…list in dot point form: the gross pathological
features, the characteristic histopathological
changes, and the clinical pathology changes. In
your discussion, list one antemortem
test/procedure that can be used to aid in the
diagnosis …

47. Examples of problems
A veterinarian asks you for assistance in designing
a protocol for the delivery of a vaccine for cats in
their practice. What factors would you take into
consideration in designing this protocol?

48. Examples of question problems
Are there any clinical features which can help you
determine a patient’s prognosis?

49. Examples of question problems
Describe and discuss the following:
a) preparedness

50. Examples of question problems
State what you believe is your most likely
diagnosis.

51. Examples of question problems
Discuss commonly found tumours and tumour-like
disorders associated with the oral cavity and
dental tissues of the horse.

52. Examples of question problems
How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3

53. Examples of question problems
Write notes on
a) considerations in the selection of stockpersons
and animal attendants

54. Examples of question problems
• Describe and give reasons for your further
clinical examination (if any), and
recommendations to the owner/trainer for
diagnostics and treatment. Explain what you
think is really important and why. Explain if you
think there is any controversy in treatment plans.

55. Examples of question problems
Version 1
Using examples, describe the categories of ways that
the productivity and profitability of a livestock
enterprise may be compromised by disease.
Version 2
Using examples, describe at least five different ways
that the productivity and profitability of a livestock
enterprise may be compromised by disease.

56. Sentence-level clarity
•
•
•
•
Simple sentence structures
Grammatically correct
Subject verb object sequence
Remove superfluous words

57. Question timing

58. How long it will take the candidate
• Unstructured tasks – open ended
• Need to control the demands of the Q carefully
• Question reading time
• Writing speed

59. Klatt & Klatt (2011) Acad Med. 86:1079–1083

60. Summers & Catarro (2003) Australian Occupational Therapy Journal 50(3): 148-157

61. Effect of time stress
• Time stress
– Increases the use of schemas (sterotyping)
– Decreases working memory processing capacity
– Decreases the ability to maintain relevant information
and suppress irrelevant information

62. http://www.slideshare.net/liznorman