Certainty Based Marking - Maggie Tarling SHS Teaching and Learning Innovators - June 2017, City, University of London

1. Certainty Based Marking
Dr Maggie Tarling

2. The main criticisms of the value of MCQs as an assessment
method are that this format encourages students to
regurgitate facts and a tendency towards rote learning, and
poor reliability because of guessing.

3. Certainty Based Marking
Certainty based marking is used with Multiple Choice
Question exams and is a tool that encourages students to
reflect more deeply before answering a question. As well
as answering a question, students are required to reflect upon
how certain they are that their answer is correct.

5. CBM mark scheme
Degree of Certainty
C = 1
(Low)
C = 2
(Mid)
C = 3
(High) No reply
Mark if correct 1 2 3 0
Mark if incorrect 0 -2 -6 0

6. Feedback on correct answers
■The student is correct and is certain about their knowledge
■Knowledge base is sound
■The student is correct but is unsure about their knowledge
■The student lacks confidence about their knowledge or they have
guessed the answer

7. Feedback on incorrect answers
■The student is incorrect and is certain that they are correct
■Knowledge is incorrect and the student has erroneous levels of
knowledge
■The student is incorrect and is unsure about their knowledge
■Shows gaps in knowledge

8. 8
Certainty Based Marking was applied to two
practice MCQs papers for the module HS1023
Understanding Concepts of Biology for Health.

9. Module
■First year biology module
■Total of 216 students – Adult, Mental Health and Child
■Two practice MCQ with CBM papers (25 questions) were
available on Moodle for students to practice in preparation
for the summative
■The summative MCQ paper based exam

10. Evaluation
■All students were allowed to use the practice papers as often
was they wished.
■Students were invited to complete an on-line evaluation
questionnaire
■Permission was also sought from students to analyse their
answers in detail

11. Use of practice papers
■Total of 216 students enrolled
■A total of 170 students used the practice papers (79%)
■Paper one – 336 attempts
■Paper two – 282 attempts
■Median number of attempts –
■40 students completed the questionnaire (24%)

12. 83
93
83
70
75
93
75
83
70
68
50
7
5
12
17
18
2
15
5
15
10
38
10
2
5
13
7
5
10
12
15
22
12
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I found using certainty based marking helpful
CBM made me think about how certain I was that the…
CBM helped me to identify my knowledge gaps
CBM helped me to identify when I was guessing the…
CBM has helped me to focus my learning
CBM has made me think more carefully about my…
CBM has helped my revision and preparation for the…
CBM has helped me understand what I do and do…
CBM has helped me to focus my revision
CBM is easy to understand
CBM had increased my confidence in biology
% of respondents
Positive Neutral

13. 20
18
28
25
22
15
55
60
57
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
CBM is not a constructive use of time
CBM has limited my revision and study
CBM takes too long to answer questions
% Responses
Positive Neutral

14. Positive
It has encouraged my
learning
INCREASE THE NUMBER OF
QUESTIONS TO 50 FOR
EVERY ATTEMPT. EXCELLENT
WAY OF LEARNING.
I liked that you could check each
one as you went through and the
variety of questions. I would
definitely use it as a revision tool :)
It is surprising when you think you
know something only to find out
that you don’t
I liked that you had to choose how
certain or uncertain you were so I
thought more about what I chose
(although it took the first time to work
out how it worked and not rush into
answers like I would usually)
This module course has help me to
build and understands the concepts
of biology. certainty based marking
had help me to build my knowledge
in biology.

15. I did not understand the points
scheme or percentage at the
end 228/100? bonus points? All
a bit confusing.
These CBM papers have been very
useful. They have definitely identified
knowledge gaps, but, I am not sure if
this is due to the 'certainty' element of
it or rather just getting the questions
wrong
Some of the questions were
worded strangely found myself
guessing because I didn’t know
what was being asked
(pharmacokinetics question)
Please make CBM clear to
understand. The system gives marks
based on many things not just the
mark you get in total. It took me a
while to figure out the entire concept.
Although smart, it is a little bit
complicated
we would appreciate if there were
more topics in the CBM practice
papers. the current questions from
both papers do not cover enough
from each topic pf physiology and
anatomy
Problems

16. i dont understand the marking
skills
I'm a little confused with the combination of
certainty and marks. 224% seems to
suggest I should do less revision but as
there are only 25 questions I could have just
been lucky. It won't direct my revision but I
can understand why it could be beneficial to
examine a medical student in this way
Didn't see the use of CBM, I have
found gaps in my knowledge through
revision and classes.
Problems

17. Conclusions
■The majority of students used the CBM practice papers
■The comments were generally positive and the students
found them useful
■There was some confusion regarding the marking summary
Moodle produces
■Some students weren’t clear about CBM usefulness
■It is unclear why 21% of the students didn’t use the practice
papers

18. City, University of London
Northampton Square
London
EC1V 0HB
United Kingdom
T: +44 (0)20 7040 5060
E: department@city.ac.uk
www.city.ac.uk/department