Assessing Knowledge and Skills in Exams

Massey University | massey.ac.nz | 0800 MASSEY
Drafting questions and
marking schemes
Liz Norman
Massey University

Examining
What's going on? by hedera.baltica, Attribution-ShareAlike Licence
What is it to be a
Member or Fellow
in my discipline?
How will I know
a candidate has
got there?
I am observing the
right things?
How can I see
knowledge?
Have I observed
enough?

Outline of the 2 days
This morning
• What are we trying to assess?
• How will tell if a candidate has that knowledge and
skill?
• How will we design tasks that get candidates to
demonstrate that knowledge and skill?
This afternoon
• How will I tell if a candidate has met the standard?
• What’s different about oral exams?
Tomorrow morning
• How we can be sure the candidate does what we want
to observe?
In between activities:
Working on your own
questions and marking
schemes with help of myself
and BoE members.
Make use of other groups for
ideas.
Make use of other groups for
proof reading and
sense-checking.

Outline of this session
1. What are we trying to assess?
2. How will tell if a candidate has
that knowledge and skill?
Marking
schemes
3. How will we design tasks that get
candidates to demonstrate that
knowledge and skill?
Writing
questions

Exams and the domain of practice
Practice domain:
all that Members or Fellows know
and can do
Content domain:
all that we could examine
Examination domains:
all that a particular exam covers
Credentials domain:
all that the credentials document covers

Exams and the domain of practice
For validity, it must be
reasonable to:
• generalise from the
examination domain to the
content domain
• extrapolate from the content
domain to the practice
domain
alignment and representativeness

Ways to think about knowledge: Dictionary definitions
Knowledge:
The fact or state of having a correct idea or understanding of
something; the possession of information about something.
Understand:
To comprehend; to apprehend the meaning or import of; to grasp the
idea of. To be thoroughly acquainted or familiar with (an art,
profession, etc.); to be able to practise or deal with properly.
Oxford English Dictionary, 2012

Lack of knowledge or common knowledge only.
Answer addresses a single aspect or a few aspects. Oversimplified,
reductionist.
Answer addresses multiple aspects and may well be comprehensive.
Connections simple or lacking.
Answer addresses multiple aspects and also how they
integrate and inter-relate. Makes connections between aspects. Knows
their relative importance/value/significance.
Goes beyond a relational answer within and beyond a domain, including to
areas not experienced or only imagined. Has more originality, creativity,
meta-connections, and utilisation of overarching principles.
Pre-structural
Uni-structural
Multi-structural
Relational
Extended abstract
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press.

Uni-
structural
Multi-
structural
Relational
Extended
abstract
SOLO Taxonomy
QuantityQuality
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO
taxonomy (structure of the observed learning outcome). New York: Academic Press.

Bloom’s revised taxonomy
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4),
212-218. doi:10.1207/s15430421tip4104_2
Remember Retrieving relevant knowledge from long-term
memory.
Recognizing, Recalling
Understand Determining the meaning of instructional messages,
including oral, written, and graphic communication.
Interpreting, Exemplifying,
Classifying, Summarizing,
Inferring, Comparing, Explaining
Apply Carrying out or using a procedure in a given situation. Executing, Implementing
Analyze Breaking material into its constituent parts and
detecting how the parts relate to one another and to
an overall structure or purpose.
Differentiating, Organizing,
Attributing
Evaluate Making judgments based on criteria and standards. Checking, Critiquing
Create Putting elements together to form a novel, coherent
whole or make an original product.
Generating, Planning, Producing

Uni-
structural
Multi-
structural
Relational
Extended
abstract
SOLO Taxonomy Bloom’s Taxonomy
QuantityQuality
remember
understand
apply
analyse
evaluate
create
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview.
Theory Into Practice, 41(4), 212-218. doi:10.1207/s15430421tip4104_2

Recall-higher order classification
Fact recall Questions capable of being answered by reference to one
paragraph in a text or notes (or several paragraphs for questions
requiring recall of several facts)
Applied (higher
order)
Questions that require the use of facts or concepts, the solution of
a diagnostic or physiologic problem, the perception of a
relationship, or other process beyond recalling discrete fact
Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice
examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.

Uni-
structural
Multi-
structural
Relational
Extended
abstract
SOLO Taxonomy Bloom’s Taxonomy Recall application
classification
QuantityQuality
remember
understand
apply
analyse
evaluate
create
recall
application

Wiggins and McTighe
Knowledge-understanding classification
Knowledge knowing about; a body of coherent facts; can be thought of as right
or wrong.
Understanding knowing how and why; the meaning of facts; the theory that links
facts and provides meaning; how sense is made of facts to enable
them to be applied to analysis, synthesis, evaluation; to be able to
explain why particular facts or skills are applicable to a particular
situation; to know which fact to apply when; to be able to create
new knowledge or modify or adapt an idea to a new situation.
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and
Curriculum Development.

Uni-
structural
Multi-
structural
Relational
Extended
abstract
classification
Wiggins & McTighe
understanding by
design
QuantityQuality
remember
understand
apply
analyse
evaluate
create
recall
application
knowledge
understanding
Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria,
VA, USA: Association for Supervision and Curriculum Development.

Webb
Depth of knowledge classification
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25. doi:10.1080/08957340709336728.
recall Recall information or perform a simple step, identify, measure,
describe, explain simple ideas
skill/concept Requires more than one step, comparing, interpreting, estimating,
making observations, explaining, organising and displaying data
strategic
thinking
Requires planning and using evidence, explaining reasoning, make
conjectures, draw conclusions, solving problems
extended
thinking
Complex reasoning, planning developing and thinking over an
extended period of time, making multiple connections, synthesis of
ideas into new concepts

Uni-
structural
Multi-
structural
Relational
Extended
abstract
classification
Wiggins & McTighe
understanding by
design
Webb’s depth of
knowledge
QuantityQuality
remember
understand
apply
analyse
evaluate
create
recall
application
recall
skill or
concept
strategic
thinking
extended
thinking
knowledge
understanding
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments.
Applied Measurement in Education, 20(1), 7-25. doi:10.1080/08957340709336728.

What things are we trying to assess?
1. factual knowledge
– Can they recall it?
2. depth of knowledge/understanding
– Do they understand it?
– Can they use it?
3. skills (incl cognitive skills)
– Can they do it the right way?
4. product of performance
– Can they produce an appropriate outcome?
New
graduate
between Membership between Fellowship

How will we tell if a candidate
has that knowledge and skill?
Marking schemes

Validity again…
• Markers need to reward features we
want to evaluate and not reward
features we do not want to evaluate
• Different markers need to award
similar/scores for the same candidate
response
squirrel by Phil Fiddyment, Attribution License

Purpose of marking schemes
• To help you during Q writing
– What content is important
– Whether the Q asks what you intended it to ask
– Whether it is do-able in the time available
• To help you during Q marking
– helps you decide how to award marks to any particular candidate
response
– facilitates reliable and fair marking

Types of marking scheme
1. Model answer – “ideal” answer
2. Point-based schemes
3. Criteria- & level-based schemes
4. Schemes with incorporated principle(s) for discriminating
levels

Criteria- & level-based schemes
• Criteria – different dimensions of performance
• Level – different quality/standards of performance on a given
criteria
Criteria
Descriptions of levels of performance
High level performance Mid-level performance Low level performance
Correctness of
knowledge
All facts correct Many facts correct but some
errors
Errors of fact or content
Sufficiency of
coverage
Comprehensive coverage Some missing but most
important areas covered
Missing important
information
Rationale and
justification
Fully supported and justified
by argumentation and citing
of evidence.
Supported and justified by
argumentation but evidence
not cited.
Limited or no support for
facts presented

• Going to talk some more about this in the next session
• Difference between Membership and Fellowship
• Difference between passing level and failing level within
Membership and within Fellowship
New graduate between Membership between Fellowship
Clear fail level (0-
54%)
Marginal fail (55-
70%)
Marginal pass (70-
79%)
Clear pass level
(80-100%)
Levels of performance

Four main types of criteria
• Factual knowledge
– knowledge of and knowledge that
– knowledge that is typically recalled and
requires remembering
– usually thought of as being correct or
incorrect
• Understanding
– knowledge of how and why and the
meaning of things
– observed via explanations, rationale,
discussions, and justifications
– different aspects to understanding
• Planning or approach
– skill in performing a task during an
examination
– frequently revolve around interpretation,
analysis, problem solving, and planning
management of problems
– observed via candidate “workings” or by
observing them performing the task
– only important in some types of question
– different aspects to consider
• Product or output
– the candidate’s response: what we see to
examine
– only important in some types of question
– can indicate relevant skills, including
communicative and cognitive skills

Factual knowledge
Criterion High quality descriptor Low quality descriptor
factual knowledge Correct and sufficient facts or content (focus may
mean it is not comprehensive but includes all that
is important).
Errors of fact or content, missing important
information.
higher
order skills
factual
know-
ledge

Aspects of understanding
sense and extent of
explanation
Thorough, coherent, complete, systematic, deep and broad
and goes beyond the information provided.
Incomplete or superficial, does not extend beyond
what is given.
connection and
association, grasp of
subtleties and nuances
Explains or illustrates subtle connections, distinctions and
associations, and considers the whole picture and mitigating
circumstances. Draws inferences, conceptualises implications
and explains assumptions and exceptions. Explanation relates
to the wider theories and principles of the discipline or wider
disciplines.
Overly generalised, black and white account, little
discussion of associations, implications, assumptions,
exceptions. Parts treated additively and independently.
Explanation may lack connection to the wider theories
and principles of the discipline.
degree of support –
rationale and
justification,
understanding of why
Fully supported, justified, verified, by evidence or argument,
including explaining counter-arguments and counter-opinions.
Answer is qualified and takes account of context.
Knowledge telling with limited support, argument,
justification or verification.
understanding of
meaning
Thorough and insightful interpretation or analysis of the
significance, importance, meaning.
Decodes with little or simplistic and reductionist
interpretation, restates what is given, does not explain
significance, importance or meaning.
contextualisation Contextualises appropriately, applies meaning to specifics of
the situation. Answer is selective - addresses what is
important.
Theoretical and general response, not specific to
situation. Non selective answer - incorporates
irrelevant material.
insight Illuminates tacit or overlooked assumptions, implications,
conclusions.
Overlooks or glosses over tacit assumptions,
implications, conclusions.

Aspects of process or approach
efficacy of process or
approach
Effective application even in difficult contexts, performs
the task completely and correctly.
Does not perform the task or does so incorrectly, or can
only do so in simple contexts, outcome not produced or
incomplete.
planning and
approach
Methodical, logical, systematic and thorough plan for
approaching the problem. Approach fully accommodates
all requirements, purpose(s), and contextual matters.
Approach is unplanned or is disorganised, not thorough,
illogical or unsystematic. Does not fully accommodate all
requirements, purpose(s) or contextual matters.
customisation of
process
Performance adapted to suit the situation, performs
appropriately to context, constraints, purpose and
audience. May be innovative in application.
Scripted, procedural, algorithmic or predetermined
performance, follows steps irrespective of context or
situation.
critical stance Adopts a dispassionate, circumspect, and critical stance
that examines an issue from different perspectives,
considers other points of view, acknowledges other ways
to approach a problem.
Singular perspective adopted, dismisses or unaware of
alternative points of view or approaches.
awareness of
limitations
Acknowledges the boundaries of own and others’
understanding.
Unaware of the bounds of own understanding.
recognition of
personal prejudice and
bias
Acknowledges role of own prejudices and bias on
understanding.
Does not acknowledge role of projections and prejudice in
opinions presented and approaches taken.
responsive, self-
adjusting thinking or
approach
Employs approaches that account for or limit effects of
own limitations, prejudices and biases. Willing to change
thinking or approach when finds inconsistencies.
Employs approaches that do not account for or limit
effects of own limitations, prejudices and biases. Holds to
thinking or approach even in the face of inconsistencies.

Product or output
fluency, efficiency,
elegance, confidence of
product or output
Fluent, efficient, elegant, confident performance.
Writing is coherent, organised, engaging and
persuasive. Uses appropriate technical language
and ways of phrasing.
Mechanical or tentative performance, falters,
slow, inefficient, lacks confidence. Writing lacks
coherence, may be disordered, incomprehensible,
repetitive or contain irrelevancies.

Writing marking schemes
• Select and organise 1-4 criteria for each question
• Choose criteria based on their ability to differentiate
passing and failing levels
– do not have to be comprehensive
• Think about how important each criterion is – how they
should be weighted
• Remember that factual knowledge is encompassed in
higher order criteria so doesn’t need high weighting
• Develop clear descriptions for each level/standard
of each criteria
Attitude by Rberteig, Attribution License

Revising mark schemes in use
• Hopefully all types of answer are anticipated
• Sometimes though it is not – can indicate unanticipated
problems with the Q
• Marking schemes might need revising after first few candidates
marked
• HSE should moderate marking and should encourage team
members to report marking issues early in the piece.

How will we design tasks that
get candidates to demonstrate
their knowledge and skill?
Writing questions

Validity again…
Alignment and representativeness
The skills and knowledge required
to answer the questions must
• fall within the content domain
• be representative of the content
domain
But also….
The questions must elicit the
behaviour we want to evaluate

Specifying tasks – give an instruction
Compare: to find similarities between things, or to look for characteristics
and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject matter,
including all the main points, essential details, and pros and cons of the
problem, to show your complete understanding of the subject.
Define: to provide a concise explanation of the meaning of a word or phrase;
or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions or
results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

Specifying scope
• Do you want a general answer about this condition that applies
to all possibilities or one that specifically applies to this case?
• Do you want an answer to include
– all possible options?
– those that are currently available in this region of the world?
– those that are available under cost constraints?
• Writing marking scheme helps you get the scope right

Specify boundaries of the answer
• Species
e.g. “in both dogs and cats…”
• Quantities and amounts
e.g. “Provide 5 reasons why…”
• With reference to
e.g. “ With reference to the published research from ..”
• Time
eg: “in the first 24 hours”
• Part of the question
Eg: “for one of your differentials….”

Construct irrelevance
When the skills, abilities or knowledge required to answer
examination questions is not part of the domain we are trying to
assess.
What are we actually assessing?

Important sources of construct irrelevance
• Unclear tasks – because the candidate may do something
other than what we intended them to do.
• Unanticipated and irrelevant difficulty and demand
• Insufficient time to complete the task (fully)

Controlling task difficulty and demand
• Tasks need to be at an appropriate level of difficulty
• They don’t need to be “tricky” in order to be difficult

Some factors that affect demand and difficulty
• Type of operation
• Degree of novelty
• Number of components or ideas involved
• Whether resources are provided or need to be generated by the
candidate
• The question wording and any images, diagrams, or tables
provided
• Degree of abstraction
• Response strategy – simple, stepwise, integrated
• Guidance provided

Some factors that affect demand and difficulty
• Type of operation
simple steps
synthesis,
interpretation,
evaluation

Question 1:
Explain the physiological actions of insulin.
Question 2:
Explain the physiological actions of ghrelin.

Question 1:
Compare and contrast the clinical signs of
hypoadrenocorticism with those of
hyperadrenocorticism in dogs.
Question 2:
Compare and contrast the clinical signs of
diabetes mellitus with those of
hyperadrenocorticism in dogs.

For EACH ECG, make a recommendation for management of the
horse and explain your reasoning.
Discuss the recommended management for a horse with atrial
fibrillation and explain your reasoning.

Discuss the concept of quality of life in terms of
biological function, “feelings” and natural
existence.
Describe how both classical conditioning and
operant conditioning are involved in cows
confidently entering the milking shed and letting
down.

Outline and discuss a conceptual framework for differentiating
between ryegrass varieties available in the New Zealand market
place and for defining possible strengths and weaknesses of a
particular cultivar. Your conceptual framework should reflect the
various options currently employed by plant breeders in
developing new cultivars. (10 marks)

Describe the skull. (30 marks)

Question 1
Discuss the use of insulin for the treatment of diabetes mellitus in cats (25
marks)
Question 2
a) Describe the advantages and disadvantages of insulin therapy for diabetes
mellitus in cats (10 marks)
b) Indicate the dose and frequency of administration of insulin you would
prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the frequency and
timing of feeding in relation to insulin dosing in cats with diabetes
mellitus (10 marks).

You have been contacted by a farmer producing Pacific oysters
(Crassostrea gigas) intertidally, in a bay containing a number of
oyster farms. The farmer is concerned with the amount of dead
shell they are seeing during the current grading. Explain how you
would approach this scenario. (20 marks)
Include in your answer how the information you could gather
might influence your assessment, what differential diagnoses you
consider and detail how you might further investigate potential
causes and what advice you would provide.

Pacing examinations
• Reading time: 40 words per minute
• Writing time: 20 words per minute
• Model answers help you check the pacing

Key points: What are we trying to assess?
• Ensure the examination (as a whole) representatively samples
from the domain specified for the examination → Blueprint
• Level of difficulty (Qs and marking schemes) appropriate for
level of the examination (Membership vs Fellowship)
• We need the tasks to be
– things that Members or Fellows need to be able to do.
– things that will tell us something about the standard the candidate
has reached (pass-fail)

Key points: How will tell if a candidate has that
knowledge and skill?
Give clear descriptions of both criteria and levels in your marking
schemes
Give the greatest weight to the criteria that
– exemplify the most important skills of a fellow or member
– will facilitate differentiation of failing from passing performances
4 different types of criteria that your question may involve
1. Factual knowledge
2. Understanding
3. Planning or approach
4. Product or output

Key points: How will we design tasks that get
candidates to demonstrate that knowledge and skill?
• Choose a task that will allow candidates to demonstrate higher
order skills
• Phrase your “question” as an instruction
• Define the scope and boundaries of the answer
• Remove all irrelevant difficulty
• Keep level of demand appropriate for the level of examination
(Fellowship vs Membership) and the time available

Assessing Knowledge and Skills in Exams

Assessing Knowledge and Skills in Exams

Recommended

Recommended

More Related Content

Similar to Assessing Knowledge and Skills in Exams

Similar to Assessing Knowledge and Skills in Exams (20)

More from Liz Norman

More from Liz Norman (20)

Recently uploaded

Recently uploaded (20)

Assessing Knowledge and Skills in Exams