This document discusses guidelines for developing effective grading criteria and marking schemes. It explains that marking schemes should help ensure markers award similar scores for the same response and reward the features being evaluated. Various types of marking schemes are described, including model answers, point-based, criteria-based, and those incorporating principles to discriminate performance levels. The document stresses that marking schemes need to consider a range of possible responses, not just ideal answers, and may need revision based on early student responses.

1. Grading criteria
and marking schemes
Liz Norman
Massey University

2. Validity
• Not a property of a test, but relates to the
interpretations or actions made on the basis of
test scores
• Eg:
The award of Bachelor of Veterinary Science implies that
the awardee has sufficient knowledge and experience to
be a competent veterinarian

3. Establishing validity
• Requires formation of a logical and plausible
argument based on evidence
(Shaw et al 2012, Kane 2004)
• The chain of inferences and
assumptions in the interpretations
of test scores is critically examined
for plausibility, threats to validity
and the supporting evidence for
validity

4. Inference 1

5. Inference 1
BVSc awardees have
sufficient knowledge,
skills, attitudes and
judgement to be
veterinarians

6. Inference 2
What we can measure
in examinations
(content domain)
are things that are
necessary to be a
BVSc (extrapolation)

7. Inference 3
A particular set of
examinations is a
measure of
achievement over
all that could be
examined
(extrapolation)

8. Inference 4
• The examination score is a measure of the level
of achievement on the examination
(generalisation)

9. Inference 5
• The passing score correctly separates
performances at or above the level of
achievement needed by a veterinarian from
performances below that level

10. Validity
The exam must elicit proper evidence of the
abilities of the candidate and we must evaluate the
evidence properly

11. Validity
• Different markers need to award similar/scores
for the same candidate response
• Markers need to reward features we want to
evaluate and not reward features we do not
want to evaluate

12. Purpose of marking schemes
• To help you during Q writing
– What content is important
– Whether the Q asks what you intended it to ask
– Whether it is do-able in the time available
• To help you during Q marking
– helps you decide whether an answer is good enough
to be awarded a mark
– facilitates reliable and fair marking

13. Types of marking scheme
1.
2.
3.
4.
Model answer – “ideal” answer
Point-based schemes
Criteria- & level-based schemes
Schemes with incorporated principle(s) for
discriminating levels

14. Model answers
• Not recommended as a sole component of a
marking scheme
– Usually more than would be expected to be given by
any candidate
– No guidance on how to assess alternative answers to
the model provided
– No guidance on how to award marks
• Can be a useful adjunct to a marking scheme

15. Point-based schemes
• Points for each objectively identifiable content
point
• Does not indicate the relative importance of the
points it awards
• Sum can be more or less than the whole
• Rewards quantity not quality
• Only useful for the quantitative SOLO levels

16. Prestructural
Unistructural
Multistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Quantitative change
Relational
Qualitative change
Extended
abstract

17. Criteria- & level-based schemes
• Criteria – different dimensions of performance
• Level – different quality/standards of
performance on a given criteria

18. Criteria- & level-based schemes
• Generic vs specific for the Q
• Explicit vs implicit weighting

19. Standards
Very poor
Analysis and 0-2 marks
interpretation Interpretation not
of results
provided or incorrect.
Total 8 marks
Criteria
Quality of
planning
Total 8 marks
0-2 marks
No plans provided or
plans not appropriate
or dangerous
Poor
2-4 marks
Lacks one or more key
elements.
Fair
4-6 marks
Adequate interpretation
that addresses key
elements. Misses nuances
of interpretation or
uncommon differentials
Excellent
6-8 marks
Thorough accurate
interpretation of results.
Well justified and
appropriately prioritised
list of differentials.
2-4 marks
Plans miss some key
aspects or overly
general
4-6 marks
Adequate plans that
address all key
differentials. Some
displaced in priority or not
pragmatic
6-8 marks
Thorough detailed and
well-prioritised and
pragmatic plan that
addresses all defined
differentials.
Knowledge of 0-2 marks
3-4 marks
current
Little or no literature referred to or incorrectly Answer refers to some of
literature
referred to.
the key literature
Total 5 marks
4-5 marks
Answer refers to current
literature including
controversies and
comparative work from
other species.
Logical
presentation
Total 4 marks
4 marks
The answer shows a high
degree of logical thought
and well-constructed
argument.
1 mark
Answer is disorganised
and includes a large
amount of irrelevant
material
2 marks
Answer is somewhat
disorganized and
includes some
irrelevant material
3 marks
The answer is relatively
well organized and
contains little irrelevant
material.

20. Southern Cross University (2013) http://scu.edu.au/teachinglearning/download.php?doc_id=12921&site_id=301&file_ext=.pdf

21. Analytical vs holistic schemes
• Both are valid
• Analytical (criteria scored separately)
–
–
–
–
Better agreement between examiners
Insufficient criteria
Overlapping criteria
Really just lots of smaller holistic decisions
• Holistic (scored as a whole)
– Don’t straightjacket examiners
– Challenging, especially for longer answers
– Less agreement between examiners

22. Whittem (2013) Guidelines for Oral Examiners, Faculty of Veterinary Science, University of Melbourne

23. Armstrong et al. (2008) University of Western Sydney http://www.uws.edu.au/__data/assets/pdf_file/0004/449860/Assessment_Guide.pdf

24. Armstrong et al. (2008) University of Western Sydney http://www.uws.edu.au/__data/assets/pdf_file/0004/449860/Assessment_Guide.pdf

25. Wood (2012). Macquarie University http://staff.mq.edu.au/public/download.jsp?id=56594

26. Prestructural
Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational
Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

27. SOLO levels in marking schemes
Prestructural
Unistructural
Multistructural
The task may be engaged, but the student is distracted or
misled by irrelevant aspects
The student focuses on the relevant domain and works with a
single aspect
The student provides correct material with discrete, separate
pieces of information that may be combined to provide a
composite picture
Relational
The student offers an integrated understanding of the
information. The whole has a coherent structure and meaning
Extended
abstract
Abstract general principles or hypotheses are provided
Scholten I, Keeves JP, Lawson MJ. High Educ 44:233–255, 2002.

28. Incorporating principles/rules
• Ideal is a specific criteria- & level-based
schemes with incorporated principle(s) for
discriminating levels

29. AQA GCSE Science A Physics 1 Foundation Tier Physics 1F Specimen Mark Scheme
http://www.aqa.org.uk/subjects/science/gcse/physics-4403/past-papers-and-mark-schemes

30. Incorporating principles
Calais has a warmer winter and a cooler summer
than Wroclaw. Explain why. (3 marks)
Marking scheme:
Looking for answers related to distance from the
sea therefore latitude is not credited.
– Land heats up quicker than sea (1 mark)
– A clear distinction between land and sea heating (2
marks)
Ahmed & Pollitt (2011) Improving marking quality through a taxonomy of mark schemes. Assessment in Education:
Principles, Policy & Practice 2011;18:259-278.

31. Incorporating principles
Marking scheme excerpt:
Discussion should focus on strategies adopted to
ensure sustainability and an evaluation of these with
regard to whether or not or to what extent the Sahel can
be sustainably managed. The discussion will depend on
the content and whether the overall view is optimistic or
pessimistic.
AQA (2013) General Certificate of Education (A-level) Geography Unit 1: Physical and Human Geography
http://www.aqa.org.uk/subjects/geography/a-level/geography-2030/past-papers-and-mark-schemes

32. Quality vs quantity
• More complex and unstructured the Q the more
assessing quality not quantity
• In very constrained tasks only judging how
correct the answer is
• In very open tasks, “correctness” is less
important and its quality that's judged instead

33. Prestructural
Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural
Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational
Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

34. Content vs connections
Lucander et al. (2010). European Journal of Dental Education, 14(3), 145-150.

35. Writing marking schemes
• Select and organise the criteria/dimensions
• Develop clear descriptions for each
level/standard of each criteria
• Need to think about poor answers as well as
good ones

36. Revising mark schemes in use
• Hopefully all types of answer are anticipated
• Sometimes though it is not – can indicate
unanticipated problems with the Q
• Marking schemes might need revising after first
few students marked

37. http://www.slideshare.net/liznorman