1. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting and drafting
questions
Liz Norman
Massey University
Sharanne Raidal,
BVSc, PhD, FANZCVS, GradDipEd
http://consultatutorblog.files.wordpress.com/2010/06/examstress.gif
Australian and New Zealand College of Veterinary Scientists
Examiner Workshop, 6 – 7th
February 2016
2. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Overview
• Blueprinting and drafting questions
• Grading criteria and marking schemes
• Clarity, timing; oral examinations
http://www.slidehare.net/liznorman
http://toonclips.com/600/4997.jpg
3. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What is a blueprint?
blueprint, n. something which acts as a
plan
model or
template
OED Online http://www.oed.com https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg
4. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What is a blueprint?
• Specifies what’ going to be in the exam
Content / topics (breadth)
Level (depth)
https://lnocc.org/wp-content/uploads/2013/01/blueprint1a.jpg
5. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Purpose of blueprinting
• To document
sampling
• To show that your
examination is
representative of all
that could be
examined and is
sufficient
Content domain
Exam
2
Oral
Essay
Credentials
domain
Exam
1
Prac
Practice domain
6. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Why blueprint?
• To show that we
have a
representative
sample
• To permit
extrapolation to
content (practice)
domain
• To plan how each
component assessed
Content domain
Exam
2
Oral
Essay
Credentials
domain
Exam
1
Prac
Practice domain
7. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Content of questions
• Scope of topics
• Other relevant considerations - eg. species
body systems
• Scope of skills and abilities
8. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Subject guidelines
• Represent the entire content domain
(ie. they define all that is examinable)
• Specify both scope (breadth) and level (depth) of
the knowledge, skills, attitudes and judgements
required
• For Fellowship, some aspects are assessed in
the credentials document (and process and
training)
9. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Subject guidelines
The candidate will have a detailed knowledge of:
The aetiology, pathogenesis and
pathophysiology of cardiac, renal, respiratory,
alimentary, musculoskeletal, endocrine,
ophthalmological and neurological organ
dysfunction in the cat and the dog.
10. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Level – depth
Levels of knowing and understanding
Thinking processes required by the discipline
•Surface vs deep
•Fact recall vs application
•Blooms taxonomy
•SOLO taxonomy
11. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Fact recall:
Questions capable of being answered by reference to one
paragraph in a text or notes (or several paragraphs for
questions requiring several facts).
Applied (higher order):
Questions that require the use of facts or concepts, the
solution of a diagnostic of physiologic problem, the
perception of a relationship or other process(es) beyond
recalling discrete fact.
From: Peitzman et al. (1990) Academic Medicine , 65(9): S59-60
12. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Bloom’s taxonomy
http://julietovar.edublogs.org/files/2011/05/blooms-taxonomy-1k4snjn.JPG
13. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Bloom’s taxonomy
http://www.psia-nw.org/wp-content/uploads/Blooms_Taxonomy.jpg
Recall
Application
15. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
http://www.pescholar.com/wp-content/uploads/2013/03/Solo-SS.png
16. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
SOLO taxonomy
http://litre.ncsu.edu/solowrittencomposite.gif
17. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions skills & abilities required
SOLO taxonomy
http://www.mbms.org.uk/faculties/science/assets/SOLO/solo-
taxonomy-explained-full.jpg
18. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely
misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response
(such as being able to describe aetiology, clinical features,
management) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect
relationships; able to compare similarities and differences
between apparently distinct phenomena; this level is taken as
suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena
and apply this information to a novel context – able to develop
novel hypotheses, theories and deduce principles; creative
thinking
19. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely
misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response
(such as being able to describe aetiology, clinical features,
management) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect
relationships; able to compare similarities and differences
between apparently distinct phenomena; this level is taken as
suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena
and apply this information to a novel context – able to develop
novel hypotheses, theories and deduce principles; creative
thinking
Quantitative change
Qualitative change
20. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely
misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response
(such as being able to describe aetiology, clinical features,
management) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect
relationships; able to compare similarities and differences
between apparently distinct phenomena; this level is taken as
suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena
and apply this information to a novel context – able to develop
novel hypotheses, theories and deduce principles; creative
thinking
Deep
Surface
21. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
SOLO taxonomy
Prestructural Question may be rephrased at the answer; almost completely
misses the point of the question
Unistructural Able to identify, list, name, enumerate, but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response
(such as being able to describe aetiology, clinical features,
management) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect
relationships; able to compare similarities and differences
between apparently distinct phenomena; this level is taken as
suggestion the learner has understood
Extended abstract Highly developed; able to explain mechanisms of phenomena
and apply this information to a novel context – able to develop
novel hypotheses, theories and deduce principles; creative
thinking
Application
Recall
23. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Level of questions; skills & abilities required
Skills and abilities
•What is contained in subject guidelines?
•Skills beyond cognitive?
•Recognition / interpretation?
•Communication?
- diagnostic images
- cytology / pathology
- reports
- statistics
24. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Mapping to scope and level
25. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by learning outcome
Written paper
1
Written paper
2
Practical
examination
Oral
examination
LO1 x x x
LO2 x x
LO3 x x x
LO4 x
26. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by topic
Patho-
physiology
Investigation &
diagnosis
Treatment &
management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
27. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by topic and depth
Patho-
physiology
Investigation &
diagnosis
Treatment &
management
recall higher recall higher recall higher
Gastrointestinal P1Q1 P1Q1 P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous
P1Q3,
P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
28. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Breadth by species
Species Number of Qs Percent
Small animal 15 52%
Farm animal 4 14%
Horse 5 17%
Exotic 3 10%
Lab 1 3%
All 1 3%
29. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting - process
• Content coverage x depth
Topic 1
30%
Topic 2
30%
Topic 3
40%
Total
Knowledge of terms 2 5 5 12
Comprehension of principles 4 3 4 11
Application of principles 3 3 3 9
Analysis of situations 3 2 5 10
Evaluation of situations 3 2 3 8
Total questions 15 15 20 50
http://scoring.msu.edu/written.html
30. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting
Enables you to plan and check that your exam, or
set of exams, covers the content (and skills) it is
meant to cover, at an appropriate level.
Without this you cannot extrapolate performance
on this exam to performance in the whole domain.
Validity
31. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Should questions be difficult?
Norm-referenced vs
standards-based grading
32. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Should questions be difficult?
Appropriate demand
vs irrelevant difficulty
Practice domain
Content
domain
33. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
What makes questions difficult?
•Familiarity / novelty
•Complexity – components, links
•Scale – components, links
•Resources utilised (provided, generated)
•Abstraction – concrete experience, concepts, principles
•Task strategy – simple, stepwise, integrated
•Guidance
Adapted from Hughes et al (1998) The development
of a tool for gauging demands of GCSE and A level
exam questions. Brit Ed Res Assoc Conference.
34. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Familiarity / novelty
Question 1
Explain the physiological actions of insulin.
Question 2
Explain the physiological actions of ghrelin.
35. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Familiarity / novelty
Question 1
List four (4) clinical signs of diabetes mellitus in dogs.
Question 2
List four (4) clinical signs of feline hypersomatotropism.
36. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Complexity, scale – components, links
Question 1
Compare and contrast the clinical signs of
hypoadrenocorticism with those of hyperadrenocorticism in
dogs.
Question 2
Compare and contrast the clinical signs of diabetes mellitus
with those of hyperadrenocorticism in dogs.
37. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Complexity, scale – components, links
Question 1
Compare and contrast the clinical signs of
diabetes mellitus with those of hyperadrenocorticism in dogs
and cats.
Question 2
Compare and contrast the clinical signs of diabetes mellitus
with those of hypoadrenocorticism in dogs and cats.
38. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Resources utilised – provided, generated
Describe the skull.
(30 marks)
39. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Abstraction – concrete experience, concepts,
principles
Discuss the concept of quality of life in terms of biological
function, “feelings” and natural existence.
Describe how both classical conditioning and operant
conditioning are involved in cows confidently entering the
milking shed and letting down.
40. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise,
integrated
Question 1
Discuss the use of insulin for the treatment of diabetes
mellitus in cats.
(25 marks)
41. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise,
integrated
Question 2
a)Describe the advantages and disadvantages of insulin therapy for
diabetes mellitus in cats. (10 marks)
b)Indicate the dose and frequency of administration of insulin you
would prescribe to a newly diagnosed cat with diabetes mellitus. (5
marks)
c)Describe the recommendations you would make for the frequency
and timing of feeding in relation to insulin dosing in cats with
diabetes mellitus. (10 marks)
42. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise,
integrated
Guidance
Describe the skull. (30 marks)
43. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise,
integrated
Guidance
You are presented with a 7 year old male Warmblood gelding with exercise
intolerance. You auscultate an arrhythmia with a variable pulse and order an
ECG. Define the type and give a specific name for the arrhythmia demonstrated
in the rhythm strip below. (2 marks)
44. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Controlling difficulty and demand
Task and response strategies – simple, stepwise,
integrated
Guidance
You have been contracted by a farmer producing Pacific oysters (Crassostrea
gigas) intertidally, in a bay containing a number of oyster farms. The farmer is
concerned with the amount of dead shell they are seeing during the current
grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence
your assessment, what differential diagnoses you consider, and detail how you
might further investigate potential causes, and what advice you would provide.
45. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
What’s different about oral questions?
• Thinking on your feet
• Can’t revisit or edit
• Tests verbal communication skills (as well as
knowledge)
• Interactive – can prompt and redirect, can ask
candidate to explain their answer
47. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting
• Ensure the examination (as a whole)
representatively samples from the domain
specified for the examination
• Membership – this is all the learning outcomes
• Fellowship – this is all the learning outcomes,
except those that can only be sampled in the
credentials document
48. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Blueprinting
• Ensure the examination (as a whole)
representatively samples from the domain
specified for the examination
• Consider - topics
- level
- species, body systems, modalities
- skills
- other things
(eg. Dx vs mgt, principles vs application)
49. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Structuring the exam
• Ensure you follow subject guidelines explicitly
• Shorter vs longer questions
• Scenario vs direct questions
• Difficulty, guidance
50. SCHOOL OF ANIMAL AND VETERINARY SCIENCES
Next session’s task:
• Check your blueprint - content
- level
• Classify your draft Qs according to SOLO
taxonomy
• Redraft Qs if necessary