This document provides examination and moderation guidelines. It discusses principles for valid exam conclusions, including appropriate exam content, questions that assess intended skills and knowledge, avoiding irrelevant content, mark schemes that reward the proper level of understanding, question difficulty, and differentiation of performance between high and low achieving students. Specific guidelines are provided for each principle, such as ensuring content has been taught, using precise question wording, checking question difficulty after marking, and identifying issues with questions more students perform poorly on. The document emphasizes the importance of these principles and guidelines for drawing defensible conclusions from exam results.

1. Examination and moderation
guidelines
Liz Norman
Massey University

2. Examination validity
• Exams are not valid or invalid – it’s the
conclusions we draw from exam results that
are valid or invalid.
• The entire conclusion about the correct grade
for the student, can be called into question if
any step is questionable.
• Just like in research, defensible conclusions
require attention to methods and protocols.

3. 6 principles to follow
1. The content for examination must be appropriate
2. The questions must elicit skills and knowledge that we want
to assess
3. The questions must not elicit skills and knowledge that are
not part of the curriculum
4. The mark scheme must reward the appropriate level of skill
and knowledge
5. The questions must be the right level of difficulty
6. The highest performing students should answer questions
well and the lowest performing students should answer
them less well

4. 1. The content for examination must
be appropriate
• Content (as a whole) of all assessment for the paper
should:
– Reflect the weighting and importance in the curriculum
– Involve skills and knowledge the students have practiced and
received feedback on
• Do not reuse questions for more than 10-20% of the exam:
recall exams circulate even for total recovery exams
• If using old questions, check the content is still relevant and
the material has been taught
• Record procedures:
– blueprint
– LOs covered by questions
– where and when content was taught/practiced/fedback on

5. Breadth by learning outcome
Lit review
assignment
Group
presentation
Written
examination
Practical
examination
LO1 X X X
LO2 X X
LO3 X X X
LO4 X

6. Breadth across species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%

7. Breadth by topic
Pathophysiology
Investigation and
diagnosis
Treatment and
management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5

10. 2. The questions must elicit skills and
knowledge that we want to assess
• The wording of the questions needs to be
explicitly clear, precise, and unambiguous
• Give instructions rather than asking questions
• Use tasks that require higher order thinking
• Images and other ancillary materials must be high
quality and clearly show what is intended
• Moderators cannot check this without seeing the
mark scheme
• Moderators from different disciplines may see
issues better than moderators from the same
discipline

11. Don’t write questions; write tasks
What is your
diagnosis?
State the most likely diagnosis
State the most likely diagnosis
and explain your reasoning
Discuss the differential
diagnoses you would consider in
this case
or …..

12. Instructional verb examples
Compare: to find similarities between things, or to look for
characteristics and features that resemble each other.
Contrast: to find differences or to distinguish between things.
Discuss: to present a detailed argument or account of the subject
matter, including all the main points, essential details, and pros and cons
of the problem, to show your complete understanding of the subject.
Define: to provide a concise explanation of the meaning of a word or
phrase; or to describe the essential qualities of something.
Explain: to clarify, interpret, give reasons for differences of opinions or
results, or analyse causes.
Illustrate: to use a picture, diagram or example to clarify a point.

13. Make short notes on THREE (3) of the following.
a) Pulmonary compliance.
b) Bicarbonate as a blood buffer.
c) The respiratory centre.
d) The respiratory functions of the nose.

14. Discuss commonly found tumours and tumour-
like disorders associated with the oral cavity and
dental tissues of the horse.

15. Name two (2) diagnostic tests you would run next
to investigate the cause of this dog’s current
illness.

16. A dog is presented to your clinic after a road
traffic accident.
a) What clinical signs would be consistent
with urinary bladder rupture?

17. Outline and discuss a conceptual framework for
differentiating between ryegrass varieties
available in the New Zealand market place and
for defining possible strengths and weaknesses
of a particular cultivar. Your conceptual
framework should reflect the various options
currently employed by plant breeders in
developing new cultivars. (10 marks)

18. How would you localise the site of the lesion?
Answer provided in the marking scheme:
Spinal lesion between T3 and L3

19. 3. The questions must not elicit skills and
knowledge that are not part of the curriculum
• Common issues
– Timing issues (students can only write 16 wpm)
– Reading speed
– Vague questions (eg write short notes on…)
– Content at the wrong level
– Irrelevant content/images
– Trivial/unimportant content
• Also remember
– Colour blindness
– Don’t use choice in examinations
• Sources of construct irrelevant difficulty are multiple and
unpredictable: therefore
– Keep question wording concise and straightforward
– Only provide information required to answer the question and
relevant distractors if appropriate
– Use images only when required to answer the question

20. Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian
Occupational Therapy Journal 50(3): 148-157

21. Effect of time stress
• Time stress
– Increases the use of schemas (sterotyping)
– Decreases working memory processing capacity
– Decreases the ability to maintain relevant
information and suppress irrelevant information

22. Question 1
A crate of 12 cans of cola costs $4.20. How much
do 7 crates of cola cost?
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy
& Practice, 14(2), 201-232.

23. Question 1
A crate of 12 cans of cola costs $4.20. How much
do 7 crates of cola cost?
Question 2
A ski pass costs $4.20. How much would it cost
for 7 days?
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy
& Practice, 14(2), 201-232.

24. Question 1
A crate of 12 cans of cola costs $4.20. How much
do 7 crates of cola cost?
Pass rate 41.5%
Question 2
A ski pass costs $4.20. How much would it cost
for 7 days?
Pass rate 79%
Ahmed & Pollitt (2007) Improving the quality of contextualized questions: An experimental investigation of focus. Assessment in Education: Principles, Policy
& Practice, 14(2), 201-232.

25. Ahmed&Pollitt(2007)Improvingthequalityofcontextualizedquestions:Anexperimentalinvestigation
offocus.AssessmentinEducation:Principles,Policy&Practice,14(2),201-232.

30. 4. The mark scheme must reward the
appropriate level of skill and knowledge
• Avoid awarding marks for unprioritised volume of
facts rather than relevant and structured knowledge
• Think about the quality of thinking

31. Fact recall vs applied
Fact recall:
Questions capable of being answered by reference
to one paragraph in a text or notes (or several
paragraphs for questions requiring recall of several
facts)
Applied (higher order)
Questions that require the use of facts or concepts,
the solution of a diagnostic or physiologic problem,
the perception of a relationship, or other process
beyond recalling discrete fact
Peitzman et al. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical
performance of medical students. Academic Medicine, 65(9), S59-60.

32. Disconnected knowledge Connected knowledge
• No particular order to aspects
presented
• Inclusion of irrelevant material
• Inconsistencies or incorrect
aspects
• Superficial or oversimplified
• Replication of material from
sources – rote learned or
reproduced without significant
transformation
• Aspects explained relative to one another
• Logically organised answer
• Analysis and or synthesis
• Compares similarities and differences
• Integrates multiple levels (eg: molecular, biochemical,
systemic)
• Evaluates inconsistencies
• Expresses reasons
• Explains implications or reaches a conclusion
• Expresses relative importance, value, significance of
aspects.
• Selective answer that addresses the point of the question
and may be shorter than a multistructural answer
• Uses the language of the discipline - terminology and
phrasing
• Relates answer to examples or experience
• Relates answer to organising principles of the discipline
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press.

33. 4. The mark scheme must reward the
appropriate level of skill and knowledge
• Avoid awarding marks for unprioritised volume of
facts rather than relevant and structured knowledge
• Think about the quality of thinking
• Holistic marking schemes are fine, but must specify
criteria to be used
• Marking schemes may need refining during marking
• Aiming for the median mark to be above 70%

34. 5. The questions must be the right
level of difficulty
• After/during marking check for overly difficult
questions
– Was there a problem with the question wording that
has meant students misinterpreted the intended task?
– Was the answer keyed correctly (for MCQs)?
– Was the content actually taught/practiced and given
feedback on?
– Was there sufficient time to answer the question?
• May need to
– Adjust marking scheme
– Delete question – consult Liz or Jenny

35. 6. The highest performing students should
answer questions well; the lowest performing
students should answer them less well
• After/during marking check all items with low or
negative item to total score correlation.
– Problems with question wording and interpreted task?
– More than one right answer, exceptions that are also
correct?
– Run out of time?
– Even if you can’t see the problem, construct irrelevant
difficulty is likely
• May need to
– Adjust marking scheme
– Delete question – consult Liz or Jenny

36. http://www.slideshare.net/liznorman